13 results found Sort:
- Filter by Primary Language:
- Python (5)
- Go (2)
- C (1)
- R (1)
- TypeScript (1)
- Lua (1)
- Java (1)
- +
A list of AI agents and robots to block.
Created
2024-03-27
299 commits to main branch, last one 18 days ago
🤖/👨🦰 Detect bots/crawlers/spiders using the user agent string
Created
2015-07-24
325 commits to main branch, last one 5 days ago
An R web crawler and scraper
Created
2016-11-08
201 commits to master branch, last one 4 years ago
Open source SEO auditing tool.
Created
2022-03-02
783 commits to main branch, last one 25 days ago
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
Created
2013-02-20
1,042 commits to main branch, last one 3 months ago
Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games
Created
2014-01-08
20 commits to master branch, last one 6 days ago
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Created
2019-09-07
4,429 commits to v1.21.3-at branch, last one about a month ago
Proxy List Scrapper
Created
2020-05-15
54 commits to master branch, last one 3 years ago
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
Created
2016-01-08
303 commits to master branch, last one 10 months ago
Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)
This repository has been archived
(exclude archived)
Created
2020-02-28
14 commits to master branch, last one 2 years ago
hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)
Created
2018-04-06
69 commits to master branch, last one 6 years ago
Raven is a powerful and customizable web crawler written in Go.
Created
2024-05-08
9 commits to main branch, last one 3 months ago
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant ba...
Created
2021-01-17
190 commits to main branch, last one about a year ago