5 results found Sort:

dmi3kno
polite dmi3kno
14
326
other
6
Be nice on the web
r rvest rstats crawler memoise scraper r-package robotstxt webscraping rate-limiter
Created 2018-07-23
84 commits to develop branch, last one about a year ago
alextim
astro-lib alextim
6
106
mit
1
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
seo astro sitemap robotstxt robots-txt sitemap-xml webmanifest
Created 2022-04-26
377 commits to main branch, last one about a year ago
Folyd
robotstxt Folyd
13
85
apache-2.0
6
A native Rust port of Google's robots.txt parser and matcher C++ library.
rust robotstxt google-robots-parser
Created 2020-04-02
61 commits to master branch, last one 3 years ago
ropensci
robotstxt ropensci
10
69
other
8
R 📦 for parsing and checking robots.txt files 🤖
r rstats spider crawler scraper r-package robotstxt webscraping peer-reviewed
Created 2015-12-01
310 commits to main branch, last one 27 days ago
itgalaxy
generate-robotstxt itgalaxy
8
65
mit
6
Generator robots.txt for node js
cli robot robots robotstxt robots-txt generator-robots robots-generator
Created 2014-11-07
173 commits to master branch, last one 4 years ago