Statistics for topic crawling

RepositoryStats tracks 518,991 Github repositories, of these 80 are tagged with the crawling topic. The most common primary language for repositories using this topic is Python (33). Other languages include: Go (12)

Stargazers over time for topic crawling

Most starred repositories for topic crawling (view more)

scrapy scrapy

10.4k

51.1k

bsd-3-clause

1.8k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler crawling scraping framework web-scraping hacktoberfest web-scraping-python

Created 2010-02-22

10,431 commits to master branch, last one 14 hours ago

colly gocolly

1.7k

22.3k

apache-2.0

329

Elegant Scraper and Crawler Framework for Golang

go golang spider crawler scraper crawling scraping framework

Created 2017-09-29

665 commits to master branch, last one about a month ago

newspaper codelucas

2.1k

13.8k

mit

386

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

news python crawler scraper crawling news-aggregator

Created 2013-11-25

651 commits to master branch, last one 3 years ago

crawlee apify

530

12.3k

apache-2.0

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

npm apify nodejs crawler scraper crawling headless scraping puppeteer automation javascript playwright typescript web-crawler web-crawling web-scraping headless-chrome

Created 2016-08-26

4,353 commits to master branch, last one 15 hours ago

awesome-web-scraping lorien

772

6.4k

other

230

List of libraries, tools and APIs for web scraping and data processing.

spider crawler crawling scraping webscraping web-scraping crawling-tool scraping-tool captcha-bypass crawling-python scraping-python captcha-recaptcha crawling-framework scraping-framework

Created 2015-08-12

501 commits to master branch, last one 7 days ago