Search Results - RepositoryStats

DrissionPage g1879

565

5.6k

bsd-3-clause

147

基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器自动化的便利性和requests的高效率。功能强大，内置无数人性化设计和便捷功能。语法简洁而优雅，代码量少。

crawlers requests selenium-python automation-framework

Created 2020-05-21

297 commits to master branch, last one a day ago

isbot omrilotan

73

849

unlicense

11

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

crawlers user-agent web-crawlers user-agent-parser user-agent-analysis

Created 2015-07-24

299 commits to main branch, last one 2 days ago

Rcrawler salimk

94

349

other

40

An R web crawler and scraper

r crawler scraper crawlers rpackage webcrawler webscraper webscraping webscrapping

Created 2016-11-08

201 commits to master branch, last one 4 years ago

ai.robots.txt ai-robots-txt

9

314

mit

13

A list of AI agents and robots to block.

ai privacy crawlers crawling

Created 2024-03-27

55 commits to main branch, last one 2 days ago

crawlers Norconex

67

176

apache-2.0

33

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

java crawler crawlers flexible web-crawler collector-fs search-engine collector-http filesystem-crawler

Created 2013-02-20

811 commits to main branch, last one 3 days ago

seonaut StJudeWasHere

26

165

mit

4

Open source SEO auditing tool.

go seo web audit docker golang crawler crawlers crawling seotools crawlergo multiuser seo-audit docker-compose search-engine-optimization

Created 2022-03-02

710 commits to main branch, last one 23 days ago

Proxy-List-Scrapper narkhedesam

19

95

mit

3

Proxy List Scrapper

Created 2020-05-15

54 commits to master branch, last one 3 years ago

robots.txt jonasjacek

37

83

other

7

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

seo bingbot crawlers crawling googlebot whitelist robots-txt twitterbot user-agent web-robots baiduspider web-crawling blocking-bots search-engine robots-exclusion-standard search-engine-optimization

Created 2016-01-08

303 commits to master branch, last one 3 months ago

wget-lua ArchiveTeam

14

83

gpl-3.0

20

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

ftp lua warc wget zstd crawl spider crawler scraper crawlers crawling scraping wget-lua archiving downloader archiveteam webarchiving

Created 2019-09-07

4,428 commits to v1.21.3-at branch, last one 5 months ago

social-scraper behitek

40

69

unknown

6

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

crawler scraper youtube crawlers requests instagram selenium-python scraping-websites crawling-framework

Created 2020-02-28

14 commits to master branch, last one about a year ago

hproxy howie6879

13

66

mit

5

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

proxy sanic hproxy asyncio crawler crawlers schedule proxy-pool proxy-spider

Created 2018-04-06

69 commits to master branch, last one 5 years ago

sneakpeek flulemon

0

37

bsd-3-clause

2

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant ba...

vue python crawler python3 scraper crawlers crawling scrapers scraping scraper-api crawler-python scraper-engine crawling-engine website-crawler crawling-framework scraping-framework

Created 2021-01-17

190 commits to main branch, last one 9 months ago