brendonboshell / supercrawler

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

brendonboshell
brendonboshell
Date Created 2016-07-17 (8 years ago)
Commits 103 (last one 5 years ago)
Stargazers 380 (0 this week)
Watchers 11 (0 this week)
Forks 61
License apache-2.0
Ranking

RepositoryStats indexes 620,786 repositories, of these brendonboshell/supercrawler is ranked #115,418 (81st percentile) for total stargazers, and #194,286 for total watchers. Github reports the primary language for this repository as JavaScript, for repositories using this language it is ranked #14,556/67,568.

brendonboshell/supercrawler is also tagged with popular topics, for these it's ranked: crawler (#196/583),  robot (#74/309)

Other Information

brendonboshell/supercrawler has 16 open pull requests on Github, 15 pull requests have been merged over the lifetime of the repository.

Github issues are enabled, there are 7 open issues and 21 closed issues.

Star History

Github stargazers over time

400400350350300300250250200200150150100100505000201720172018201820192019202020202021202120222022202320232024202420252025

Watcher History

Github watchers over time, collection started in '23

11111111111110.510.510101010101020232023Feb '23Feb '23Apr '23Apr '23Jun '23Jun '23Aug '23Aug '23Oct '23Oct '23Dec '23Dec '23Feb '24Feb '24Apr '24Apr '24Jun '24Jun '24Aug '24Aug '24Oct '24Oct '24Dec '24Dec '24Feb '25Feb '25

Recent Commit History

0 commits on the default branch (master) since jan '22

Inactive

No recent commits to this repository

Yearly Commits

Commits to the default branch (master) per year

606050504040303020201010002016201620172017201820182019201920202020202120212022202220242024

Issue History

Total Issues
Open Issues
Closed Issues
3030252520201515101055002018201820192019202020202021202120222022202320232024202420252025

Languages

The only known language in this repository is JavaScript

JavaScriptJavaScript

updated: 2025-02-25 @ 07:55pm, id: 63552531 / R_kgDOA8m8Ew