2 results found Sort:

samclarke
robots-parser samclarke
19
150
mit
6
NodeJS robots.txt parser with support for wildcard (*) matching.
nodejs rfc-9309 javascript robots-txt user-agent robots-parser robots-exclusion-standard
Created 2014-09-27
90 commits to master branch, last one 10 days ago
scrapy
protego scrapy
28
55
bsd-3-clause
9
A pure-Python robots.txt parser with support for modern conventions.
python robots-txt hacktoberfest robots-parser
Created 2019-06-19
131 commits to master branch, last one 22 days ago