2 results found Sort:

samclarke
robots-parser samclarke
19
150
mit
6
NodeJS robots.txt parser with support for wildcard (*) matching.
nodejs rfc-9309 javascript robots-txt user-agent robots-parser robots-exclusion-standard
Created 2014-09-27
90 commits to master branch, last one about a month ago
scrapy
protego scrapy
28
56
bsd-3-clause
9
A pure-Python robots.txt parser with support for modern conventions.
python robots-txt hacktoberfest robots-parser
Created 2019-06-19
134 commits to master branch, last one 19 days ago