1 result found Sort:

jonasjacek
robots.txt jonasjacek
38
87
other
7
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization
Created 2016-01-08
303 commits to master branch, last one 10 months ago