Statistics for topic webscraping

RepositoryStats tracks 579,129 Github repositories, of these 178 are tagged with the webscraping topic. The most common primary language for repositories using this topic is Python (99). Other languages include: JavaScript (12),  Go (11)

Stargazers over time for topic webscraping

Most starred repositories for topic webscraping (view more)

huginn
huginn huginn
3.8k
43.4k
mit
734
Create agents that monitor and act on your behalf. Your agents are standing by!
rss feed agent huginn scraper twitter automation monitoring webscraping feedgenerator notifications twitter-streaming
Created 2013-03-10
3,777 commits to master branch, last one 3 days ago
mendableai
firecrawl mendableai
1.4k
18.3k
agpl-3.0
96
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
ai llm rag data crawler scraper markdown scraping ai-scraping web-crawler webscraping html-to-markdown
Created 2024-04-15
2,154 commits to main branch, last one a day ago
ScrapeGraphAI
Scrapegraph-ai ScrapeGraphAI
1.3k
15.6k
mit
103
Python scraper based on AI
ai sc llm gpt-3 gpt-4 llama3 scraping scrapingweb webscraping scraping-python machine-learning automated-scraper
Created 2024-01-27
2,297 commits to main branch, last one 21 hours ago
assafelovic
gpt-researcher assafelovic
2.0k
14.8k
apache-2.0
122
LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.
ai llms agent openai python search research automation webscraping
Created 2023-05-12
1,942 commits to master branch, last one 17 hours ago
pystardust
ani-cli pystardust
553
8.0k
gpl-3.0
60
A cli tool to browse and play anime
cli fzf mac rofi anime linux posix shell termux windows syncplay terminal steamdeck webscraping
Created 2021-06-09
764 commits to master branch, last one 12 days ago
lorien
awesome-web-scraping lorien
787
6.7k
other
230
List of libraries, tools and APIs for web scraping and data processing.
spider crawler crawling scraping webscraping web-scraping crawling-tool scraping-tool captcha-bypass crawling-python scraping-python captcha-recaptcha crawling-framework scraping-framework
Created 2015-08-12
534 commits to master branch, last one 10 days ago

Trending repositories for topic webscraping (view more)

Last 3 days (new repositories)
no newly created repositories trending in the last 3 days
Last 3 days (absolute gain)
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584 (+143)
mit
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

18,259 (+104)
agpl-3.0
huginn
huginn/huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

43,439 (+25)
mit
assafelovic
assafelovic/gpt-researcher

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.

14,778 (+25)
apache-2.0
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81 (+17)
Last 3 days (relative gain)
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81 (+27%)
daijro
daijro/camoufox

🦊 Anti-detect browser

160 (+4%)
mpl-2.0
hitarth-gg
hitarth-gg/zenshin

🔖 Web & Electron based Anime Torrent Streamer for 🐈s

169 (+3%)
mit
OSINT-TECHNOLOGIES
OSINT-TECHNOLOGIES/dpulse

DPULSE - Tool for complex approach to domain OSINT

79 (+1%)
mit
0xPrateek
0xPrateek/Stardox

Github stargazers information gathering tool

169 (+0.6%)
gpl-3.0
Last week (new repositories)
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81
Last week (absolute gain)
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584 (+143)
mit
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

18,259 (+104)
agpl-3.0
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81 (+56)
huginn
huginn/huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

43,439 (+25)
mit
assafelovic
assafelovic/gpt-researcher

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.

14,778 (+25)
apache-2.0
Last week (relative gain)
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81 (+224%)
ayushagarwalk
ayushagarwalk/Email-Scraping

Email Scraping with Python

51 (+4%)
phoenixthrush
phoenixthrush/AniWorld-Downloader

AniWorld Downloader is a command-line tool for downloading and streaming content from aniworld.to, compatible with Windows, macOS, and Linux.

26 (+4%)
mit
daijro
daijro/camoufox

🦊 Anti-detect browser

160 (+4%)
mpl-2.0
OSINT-TECHNOLOGIES
OSINT-TECHNOLOGIES/dpulse

DPULSE - Tool for complex approach to domain OSINT

79 (+1%)
mit
Last month (new repositories)
D4Vinci
D4Vinci/Scrapling

Lightning-Fast, Adaptive Web Scraping for Python

120
bsd-3-clause
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81
Last month (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

18,259 (+1,855)
agpl-3.0
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584 (+833)
mit
assafelovic
assafelovic/gpt-researcher

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.

14,778 (+454)
apache-2.0
huginn
huginn/huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

43,439 (+396)
mit
alirezamika
alirezamika/autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

6,431 (+223)
mit
Last month (relative gain)
intergalacticalvariable
intergalacticalvariable/reader

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefix http://127.0.0.1:3000/https://website-to-scrape.com/

43 (+258%)
apache-2.0
tonywangcn
tonywangcn/ten-million-domains

27.6% of the Top 10 Million Sites are Dead

81 (+224%)
hitarth-gg
hitarth-gg/zenshin

🔖 Web & Electron based Anime Torrent Streamer for 🐈s

169 (+141%)
mit
phoenixthrush
phoenixthrush/AniWorld-Downloader

AniWorld Downloader is a command-line tool for downloading and streaming content from aniworld.to, compatible with Windows, macOS, and Linux.

26 (+86%)
mit
daijro
daijro/camoufox

🦊 Anti-detect browser

160 (+78%)
mpl-2.0
Last 12-months (new repositories)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

18,259
agpl-3.0
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584
mit
reworkd
reworkd/tarsier

Vision utilities for web interaction agents 👀

1,431
mit
itsOwen
itsOwen/CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

1,266
mit
raznem
raznem/parsera

Lightweight library for scraping web-sites with LLMs

859
gpl-2.0
Last 12-months (absolute gain)
mendableai
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

18,259 (+18,258)
agpl-3.0
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584 (+15,571)
mit
assafelovic
assafelovic/gpt-researcher

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.

14,778 (+9,494)
apache-2.0
huginn
huginn/huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

43,439 (+3,759)
mit
pystardust
pystardust/ani-cli

A cli tool to browse and play anime

8,004 (+2,794)
gpl-3.0
Last 12-months (relative gain)
ScrapeGraphAI
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

15,584 (+119,777%)
mit
itsOwen
itsOwen/CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

1,266 (+8,943%)
mit
reworkd
reworkd/tarsier

Vision utilities for web interaction agents 👀

1,431 (+3,666%)
mit
hitarth-gg
hitarth-gg/zenshin

🔖 Web & Electron based Anime Torrent Streamer for 🐈s

169 (+2,717%)
mit
raznem
raznem/parsera

Lightweight library for scraping web-sites with LLMs

859 (+2,161%)
gpl-2.0