Statistics for topic crawler

RepositoryStats tracks 595,856 Github repositories, of these 573 are tagged with the crawler topic. The most common primary language for repositories using this topic is Python (273). Other languages include: Go (70),  JavaScript (52),  Java (28),  PHP (28),  TypeScript (27),  C# (16),  HTML (16),  Rust (11)

Stargazers over time for topic crawler

Most starred repositories for topic crawler (view more)

scrapy
scrapy scrapy
10.6k
53.5k
bsd-3-clause
1.8k
Scrapy, a fast high-level web crawling & scraping framework for Python.
python crawler crawling scraping framework web-scraping hacktoberfest web-scraping-python
Created 2010-02-22
10,648 commits to master branch, last one 2 days ago
NaiboWang
EasySpider NaiboWang
4.5k
36.6k
other
231
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
gui rpa web www html layman spider visual crawler scraper frontend robotics code-free parameters batch-script visualization data-collection batch-processing input-parameters visualprogramming
Created 2020-07-18
592 commits to master branch, last one 10 days ago
iawia002
lux iawia002
3.0k
28.0k
mit
381
👾 Fast and simple video download library and CLI tool written in Go
go qq iqiyi video youku golang tumblr crawler scraper youtube bilibili download downloader
Created 2018-02-24
857 commits to master branch, last one 2 months ago
gocolly
colly gocolly
1.8k
23.5k
apache-2.0
334
Elegant Scraper and Crawler Framework for Golang
go golang spider crawler scraper crawling scraping framework
Created 2017-09-29
666 commits to master branch, last one 6 months ago
jhao104
proxy_pool jhao104
5.2k
21.8k
mit
446
Python ProxyPool for web spider
http proxy redis spider crawler
Created 2016-11-25
652 commits to master branch, last one 3 months ago
mendableai
firecrawl mendableai
1.6k
20.3k
agpl-3.0
111
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
ai llm rag data crawler scraper markdown scraping ai-scraping web-crawler webscraping html-to-markdown
Created 2024-04-15
2,473 commits to main branch, last one 19 hours ago

Trending repositories for topic crawler (view more)

Last 3 days (new repositories)
no newly created repositories trending in the last 3 days
Last 3 days (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

20,336 (+343)
agpl-3.0
NaiboWang
NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

36,586 (+61)
Evil0ctal
Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

9,826 (+57)
apache-2.0
Evil0ctal
Evil0ctal/Fast-Powerful-Whisper-AI-Services-API

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计。还内置了包括TikTok、抖音等社交媒体平台的爬虫，可实现来自多个社交平台的无缝媒体处理，为媒体内容数据自动化处理提供了强大且可扩展的解决方案。

257 (+52)
apache-2.0
apify
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

16,237 (+52)
apache-2.0
Last 3 days (relative gain)
xiaoxiunique
xiaoxiunique/x-kit

一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。

39 (+105%)
mit
Evil0ctal
Evil0ctal/Fast-Powerful-Whisper-AI-Services-API

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计。还内置了包括TikTok、抖音等社交媒体平台的爬虫，可实现来自多个社交平台的无缝媒体处理，为媒体内容数据自动化处理提供了强大且可扩展的解决方案。

257 (+25%)
apache-2.0
janreges
janreges/siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Wi...

352 (+15%)
mit
eredotpkfr
eredotpkfr/subscan

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

27 (+13%)
mit
rebrowser
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on dema...

416 (+5%)
Last week (new repositories)
xiaoxiunique
xiaoxiunique/x-kit

一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。

39
mit
Last week (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

20,336 (+683)
agpl-3.0
apify
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

16,237 (+133)
apache-2.0
Evil0ctal
Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

9,826 (+119)
apache-2.0
Evil0ctal
Evil0ctal/Fast-Powerful-Whisper-AI-Services-API

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计。还内置了包括TikTok、抖音等社交媒体平台的爬虫，可实现来自多个社交平台的无缝媒体处理，为媒体内容数据自动化处理提供了强大且可扩展的解决方案。

257 (+106)
apache-2.0
NaiboWang
NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

36,586 (+101)
Last week (relative gain)
XiaomingX
XiaomingX/proxy-pool

Python ProxyPool for web spider

52 (+940%)
apache-2.0
xiaoxiunique
xiaoxiunique/x-kit

一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。

39 (+105%)
mit
Evil0ctal
Evil0ctal/Fast-Powerful-Whisper-AI-Services-API

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计。还内置了包括TikTok、抖音等社交媒体平台的爬虫，可实现来自多个社交平台的无缝媒体处理，为媒体内容数据自动化处理提供了强大且可扩展的解决方案。

257 (+70%)
apache-2.0
janreges
janreges/siteone-crawler-gui

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Support...

128 (+39%)
mit
eredotpkfr
eredotpkfr/subscan

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

27 (+35%)
mit
Last month (new repositories)
XiaomingX
XiaomingX/proxy-pool

Python ProxyPool for web spider

52
apache-2.0
XiaomingX
XiaomingX/pornhub-spider

A web crawler for downloading WEBM and MP4 video formats from Pornhub. This project is designed to scrape and download available video content for educational or research purposes. Note that usage mus...

41
apache-2.0
xiaoxiunique
xiaoxiunique/x-kit

一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。

39
mit
Last month (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

20,336 (+1,476)
agpl-3.0
projectdiscovery
projectdiscovery/katana

A next-generation crawling and spidering framework.

12,697 (+564)
mit
apify
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

16,237 (+547)
apache-2.0
NaiboWang
NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

36,586 (+486)
Evil0ctal
Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

9,826 (+436)
apache-2.0
Last month (relative gain)
eredotpkfr
eredotpkfr/subscan

⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations

27 (+440%)
mit
janreges
janreges/siteone-crawler-gui

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Support...

128 (+198%)
mit
XiaomingX
XiaomingX/awesome-chinese-law

一个网络安全法律法规、安全政策、国家标准、行业标准知识库。A knowledge base of cybersecurity laws and regulations, security policies, national standards, and industry standards.

84 (+180%)
apache-2.0
Evil0ctal
Evil0ctal/Fast-Powerful-Whisper-AI-Services-API

⚡ 一款用于自动语音识别 (ASR)、翻译的高性能异步 API。不需要购买Whisper API，使用本地运行的Whisper模型进行推理，并支持多GPU并发，针对分布式部署进行设计。还内置了包括TikTok、抖音等社交媒体平台的爬虫，可实现来自多个社交平台的无缝媒体处理，为媒体内容数据自动化处理提供了强大且可扩展的解决方案。

257 (+140%)
apache-2.0
XiaomingX
XiaomingX/pornhub-spider

A web crawler for downloading WEBM and MP4 video formats from Pornhub. This project is designed to scrape and download available video content for educational or research purposes. Note that usage mus...

41 (+116%)
apache-2.0
Last 12-months (new repositories)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

20,336
agpl-3.0
apify
apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works...

4,869
apache-2.0
D4Vinci
D4Vinci/Scrapling

Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python

1,727
bsd-3-clause
Autumn-27
Autumn-27/ScopeSentry

ScopeSentry-网络空间测绘、子域名枚举、端口扫描、敏感信息发现、漏洞扫描、分布式节点

877
agpl-3.0
rebrowser
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on dema...

416
Last 12-months (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

20,336 (+20,335)
agpl-3.0
NaiboWang
NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

36,586 (+18,083)
apify
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

16,237 (+5,275)
apache-2.0
iawia002
iawia002/lux

👾 Fast and simple video download library and CLI tool written in Go

27,975 (+5,213)
mit
projectdiscovery
projectdiscovery/katana

A next-generation crawling and spidering framework.

12,697 (+4,888)
mit
Last 12-months (relative gain)
rebrowser
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on dema...

416 (+5,843%)
flairNLP
flairNLP/fundus

A very simple news crawler with a funny name

304 (+1,927%)
mit
AndyTheFactory
AndyTheFactory/newspaper4k

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

529 (+1,789%)
mit
karthikuj
karthikuj/sasori

Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.

132 (+1,786%)
mit
janreges
janreges/siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Wi...

352 (+1,576%)
mit