2 results found Sort:

samclarke
robots-parser samclarke
18
143
mit
6
NodeJS robots.txt parser with support for wildcard (*) matching.
nodejs javascript robots-txt user-agent robots-parser robots-exclusion-standard
Created 2014-09-27
86 commits to master branch, last one 11 days ago
scrapy
protego scrapy
26
52
bsd-3-clause
9
A pure-Python robots.txt parser with support for modern conventions.
python robots-txt hacktoberfest robots-parser
Created 2019-06-19
126 commits to master branch, last one about a month ago