Search Results - RepositoryStats

ai.robots.txt ai-robots-txt

58

1.7k

mit

22

A list of AI agents and robots to block.

ai privacy crawlers crawling

Created 2024-03-27

336 commits to main branch, last one 12 hours ago

isbot omrilotan

78

978

unlicense

12

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

crawlers user-agent web-crawlers user-agent-parser user-agent-analysis

Created 2015-07-24

329 commits to main branch, last one 4 days ago

Rcrawler salimk

92

352

other

40

An R web crawler and scraper

r crawler scraper crawlers rpackage webcrawler webscraper webscraping webscrapping

Created 2016-11-08

201 commits to master branch, last one 4 years ago

seonaut StJudeWasHere

48

293

mit

5

Open source SEO auditing tool.

go seo web audit docker golang crawler crawlers crawling seotools crawlergo multiuser seo-audit docker-compose search-engine-optimization

Created 2022-03-02

785 commits to main branch, last one 16 days ago

crawlers Norconex

67

186

apache-2.0

33

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

java crawler crawlers flexible web-crawler collector-fs search-engine collector-http filesystem-crawler

Created 2013-02-20

1,245 commits to main branch, last one 16 days ago

astray SiENcE

10

148

zlib

7

Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games

lua maze room love2d dungeon crawlers rougelike video-game dungeon-crawler procedural-generation

Created 2014-01-08

20 commits to master branch, last one about a month ago

wget-lua ArchiveTeam

15

109

gpl-3.0

20

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

ftp lua warc wget zstd crawl spider crawler scraper crawlers crawling scraping wget-lua archiving downloader archiveteam webarchiving

Created 2019-09-07

4,429 commits to v1.21.3-at branch, last one 2 months ago

Proxy-List-Scrapper narkhedesam

20

100

mit

3

Proxy List Scrapper

Created 2020-05-15

54 commits to master branch, last one 3 years ago

robots.txt jonasjacek

38

87

other

7

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization

Created 2016-01-08

303 commits to master branch, last one 11 months ago

social-scraper behitek

45

74

unknown

5

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

crawler scraper youtube crawlers requests instagram selenium-python scraping-websites crawling-framework

This repository has been archived (exclude archived)

Created 2020-02-28

14 commits to master branch, last one 2 years ago

hproxy howie6879

14

66

mit

5

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

proxy sanic hproxy asyncio crawler crawlers schedule proxy-pool proxy-spider

Created 2018-04-06

69 commits to master branch, last one 6 years ago

Raven Symbolexe

7

40

mit

1

Raven is a powerful and customizable web crawler written in Go.

golang crawler crawlers crawling bugbounty pentesting

Created 2024-05-08

9 commits to main branch, last one 5 months ago

sneakpeek flulemon

0

37

bsd-3-clause

1

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant ba...

vue python crawler python3 scraper crawlers crawling scrapers scraping scraper-api crawler-python scraper-engine crawling-engine website-crawler crawling-framework scraping-framework

Created 2021-01-17

190 commits to main branch, last one about a year ago