2 results found Sort:

samclarke
robots-parser samclarke
18
143
mit
6
NodeJS robots.txt parser with support for wildcard (*) matching.
nodejs javascript robots-txt user-agent robots-parser robots-exclusion-standard
Created 2014-09-27
86 commits to master branch, last one 10 days ago
jonasjacek
robots.txt jonasjacek
37
83
other
7
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization
Created 2016-01-08
303 commits to master branch, last one 4 months ago