Search Results - RepositoryStats

803

17.5k

apache-2.0

108

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

npm apify nodejs crawler scraper crawling headless scraping puppeteer automation javascript playwright typescript web-crawler web-crawling web-scraping headless-chrome

Created 2016-08-26

4,896 commits to master branch, last one 4 hours ago

crawlee-python apify

377

5.6k

apache-2.0

36

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works...

pip apify python crawler scraper crawling headless scraping automation playwright web-crawler web-crawling web-scraping beautifulsoup hacktoberfest headless-chrome

Created 2024-01-10

870 commits to master branch, last one 5 hours ago

botasaurus omkarcloud

161

1.8k

mit

18

The All in One Framework to Build Undefeatable Scrapers

Created 2023-05-09

422 commits to master branch, last one 10 days ago

Craw4LLM cxcscmu

56

612

mit

4

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

llm crawler crawling pretraining web-crawler pre-training web-crawling large-language-models

Created 2025-02-17

8 commits to main branch, last one about a month ago

amazon-scraper scrapehero-code

158

389

unknown

9

A simple web scraper to extract Product Data and Pricing from Amazon

page-scraper web-crawling web-scraping amazon-scraper scrape-products web-scraping-tutorials

Created 2020-04-20

9 commits to master branch, last one 4 years ago

crawler crwlrsoft

13

360

mit

4

Library for Rapid (Web) Crawler and Scraper Development

php crawler scraper crawling scraping web-crawler web-scraper web-crawling web-scraping hacktoberfest scraping-websites

Created 2022-01-12

434 commits to main branch, last one 23 hours ago

selenium-twitter-scraper godkingjay

64

255

apache-2.0

3

This is a Twitter Scraper which uses Selenium for scraping tweets. It is capable of scraping tweets from home, user profile, hashtag, query or search, and advanced searches.

scraper twitter selenium collaborate web-crawling hacktoberfest twitter-scraper selenium-scraper hacktoberfest-accepted

Created 2023-09-08

62 commits to master branch, last one 12 days ago

InfinityCrawler TurnerSoftware

36

251

mit

10

A simple but powerful web crawler library for .NET

spider crawler robots-txt web-crawler web-crawling

Created 2018-12-28

355 commits to main branch, last one about a year ago

omnisci3nt spyboy-productions

38

242

mit

3

Omnisci3nt – See What They’ve Tried to Hide Extract deep intelligence from any domain. From subdomains to SSL certs, archived secrets to exposed ports — Omnisci3nt gives you the full picture in second...

Created 2023-08-16

97 commits to main branch, last one 8 days ago

ayakashi ayakashi-io

8

213

other

5

:zap: Ayakashi.io - The next generation web scraping framework

automation data-mining web-crawling web-scraping headless-chrome

Created 2019-04-12

175 commits to master branch, last one about a year ago

clauneck serpapi

13

177

mit

6

A tool for scraping emails, social media accounts, and much more information from websites using Google Search Results.

ruby serp email rubygem automation open-source webscraping command-line web-crawling email-scraper data-extractor data-extraction email-extractor email-marketing email-extraction command-line-tool social-media-scraper email-extract-with-proxy

Created 2023-07-06

16 commits to master branch, last one about a year ago

krawler brianmadden

16

128

mit

6

A web crawling framework written in Kotlin

kotlin crawler4j framework webcrawler web-crawler link-checker web-crawling

Created 2016-10-24

211 commits to master branch, last one 4 years ago

bancocentralbrasil fintech-hub

34

124

mit

11

💵 💰 :brazil: Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil

money brasil brazil web-crawling web-scraping banco-central

Created 2016-07-07

99 commits to main branch, last one 3 years ago

Terpene-Profile-Parser-for-Cannabis-Strains MaxValue

18

118

unknown

17

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Created 2018-02-05

239 commits to master branch, last one about a year ago

scout-lang maxmindlin

6

112

apache-2.0

2

A web crawling programming language

dsl scraper scraping web-crawling web-scraping scraping-websites programming-language

Created 2024-05-20

138 commits to main branch, last one 8 months ago

JAW SoheilKhodayari

17

105

agpl-3.0

3

JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

csrf neo4j javascript client-side web-crawling property-graph static-analysis vulnerability-detection

Created 2020-12-11

110 commits to master branch, last one 4 months ago

robots.txt jonasjacek

38

86

other

6

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization

Created 2016-01-08

304 commits to master branch, last one 2 months ago

amazon_scraper ScrapingAnt

19

81

unknown

5

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

js amazon node-js scraper scraping data-mining web-crawler scraping-api scraping-web web-crawlers web-crawling price-scraper scraping-data amazon-scraper price-scraping scrape-products scraping-python scraping-websites amazon-scraping-library

Created 2020-05-05

32 commits to master branch, last one 2 years ago

PhantomCrawler spyboy-productions

9

64

agpl-3.0

1

Boost website hits by generating requests from multiple proxy IPs.

proxy web-crawling website-hits web-scrapping proxy-rotation ddos-attack-tools website-analytics proxy-configuration

Created 2023-12-03

29 commits to main branch, last one about a year ago

Amazon-Flipkart-Price-Comparison-Engine sushantPatrikar

35

64

mit

4

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart :moneybag: :bar_chart:

amazon python python3 tkinter flipkart python-3 web-crawling web-crawler-python corresponding-prices ecommerce-sites-amazon

Created 2018-07-06

72 commits to master branch, last one 3 years ago

python-frameworks jgujerry

4

61

mit

2

Another curated list of Python frameworks

api cms devops python webapp pipeline messaging frameworks task-queue web-crawling data-workflow deep-learning machine-learning parallel-computing distributed-computing artificial-intelligence enterprise-integrations

Created 2023-10-29

190 commits to main branch, last one 4 days ago

tweetsOLAPing MohamedHmini

7

57

unknown

3

implementing an end-to-end tweets ETL/Analysis pipeline.

ssis tweets analysis api-client twitter-api etl-pipeline web-crawling cube-analysis datawarehouse multithreading powerbi-report tweets-scraper datawarehousing google-api-client ssas-multidimensional tweets-classification multi-dimensional-analysis

Created 2020-06-09

34 commits to master branch, last one 4 years ago

webtranspose mike-gee

2

41

other

1

Web scraping API for building AI applications.

python chatbots crawling scraping web-crawling web-scraping crawling-python scraping-python web-scraping-python

Created 2023-10-21

41 commits to master branch, last one about a year ago