Search Results - RepositoryStats

firecrawl mendableai

1.6k

20.4k

agpl-3.0

111

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

ai llm rag data crawler scraper markdown scraping ai-scraping web-crawler webscraping html-to-markdown

Created 2024-04-15

2,473 commits to main branch, last one 20 hours ago

Scrapling D4Vinci

91

1.7k

bsd-3-clause

15

Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python

Created 2024-10-13

299 commits to main branch, last one 2 days ago

CyberScraper-2077 itsOwen

126

1.5k

mit

11

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

llm openai scraper gemini-api ai-scraping llm-scraper web-scraper webscraping

Created 2024-08-17

70 commits to main branch, last one about a month ago

spider spider-rs

109

1.3k

mit

14

A web crawler and scraper for Rust

rust spider crawler indexer ai-scraping llm-crawler web-crawler web-scraper web-scraping headless-chrome

Created 2018-01-07

974 commits to main branch, last one 3 days ago

firecrawl-app-examples mendableai

11

52

unknown

1

🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

ai llm rag data examples markdown scrapers templates ai-scraping web-crawler html-to-markdown

Created 2024-06-22

6 commits to main branch, last one 8 days ago

abx-dl ArchiveBox

2

49

mit

4

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/s...

cli curl wget chrome yt-dlp cli-tool crawling headless scraping puppeteer archivebox downloader gallery-dl playwright youtube-dl ai-scraping http-client internet-archiving

Created 2024-10-21

91 commits to main branch, last one 4 days ago