2 results found Sort:

samclarke
robots-parser samclarke
19
151
mit
6
NodeJS robots.txt parser with support for wildcard (*) matching.
nodejs rfc-9309 javascript robots-txt user-agent robots-parser robots-exclusion-standard
Created 2014-09-27
90 commits to master branch, last one about a month ago
jonasjacek
robots.txt jonasjacek
38
87
other
7
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization
Created 2016-01-08
303 commits to master branch, last one 10 months ago