brendonboshell / supercrawler

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

Date Created 2016-07-17 (8 years ago)

Commits 103 (last one 5 years ago)

Stargazers 380 (0 this week)

Watchers 11 (0 this week)

Forks 62

License apache-2.0

Ranking

RepositoryStats indexes 589,134 repositories, of these brendonboshell/supercrawler is ranked #111,571 (81st percentile) for total stargazers, and #191,662 for total watchers. Github reports the primary language for this repository as JavaScript, for repositories using this language it is ranked #14,366/65,568.

brendonboshell/supercrawler is also tagged with popular topics, for these it's ranked: crawler (#185/567), robot (#69/301)

Other Information

brendonboshell/supercrawler has 16 open pull requests on Github, 15 pull requests have been merged over the lifetime of the repository.

Github issues are enabled, there are 7 open issues and 21 closed issues.

All Topics

robot crawler sitemap web-crawler distributed-crawler

Star History

Github stargazers over time

Watcher History

Github watchers over time, collection started in '23

Recent Commit History

0 commits on the default branch (master) since jan '22

No recent commits to this repository

Yearly Commits

Commits to the default branch (master) per year

Issue History

Languages

The only known language in this repository is JavaScript

updated: 2024-11-27 @ 01:29pm, id: 63552531 / R_kgDOA8m8Ew