26 results found Sort:
- Filter by Primary Language:
- Python (12)
- TypeScript (3)
- Java (3)
- JavaScript (2)
- Cython (1)
- C (1)
- Rust (1)
- Scala (1)
- Go (1)
- Roff (1)
- +
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Created
2017-05-05
3,807 commits to dev branch, last one 9 days ago
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Created
2011-10-21
2,494 commits to master branch, last one 16 days ago
Collect and revisit web pages.
Created
2015-05-13
1,889 commits to master branch, last one 2 years ago
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Created
2015-02-05
1,165 commits to master branch, last one 4 months ago
Serverless replay of web archives directly in the browser
Created
2019-12-09
445 commits to main branch, last one about a month ago
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
Created
2016-03-04
1,563 commits to master branch, last one 7 days ago
Run a high-fidelity browser-based crawler in a single Docker container
Created
2020-11-02
359 commits to main branch, last one 7 days ago
WarcDB: Web crawl data as SQLite databases.
Created
2022-05-29
72 commits to main branch, last one 3 months ago
Streaming WARC/ARC library for fast web archive IO
Created
2017-03-06
141 commits to master branch, last one 4 days ago
:whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation
Created
2013-03-20
821 commits to main branch, last one 6 months ago
Bitextor generates translation memories from multilingual websites
Created
2018-04-16
3,994 commits to master branch, last one 8 months ago
News crawling with StormCrawler - stores content as WARC
Created
2016-07-18
159 commits to master branch, last one 5 months ago
Chrome extension to "Create WARC files from any webpage"
Created
2013-03-20
181 commits to main branch, last one 6 months ago
CoCrawler is a versatile web crawler built using modern tools and concurrency.
Created
2016-07-15
1,142 commits to main branch, last one 2 years ago
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
Created
2018-03-03
243 commits to main branch, last one 2 years ago
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Created
2015-08-06
141 commits to master branch, last one about a month ago
Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
Created
2021-06-28
1,179 commits to main branch, last one a day ago
Drill into WARC web archives
Created
2023-12-07
69 commits to main branch, last one 4 months ago
🗄️ A simple CLI for converting WARC to Parquet.
Created
2022-06-20
69 commits to main branch, last one 14 days ago
Parse And Create Web ARChive (WARC) files with node.js
Created
2017-05-21
114 commits to master branch, last one 5 years ago
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Created
2019-09-07
4,428 commits to v1.21.3-at branch, last one 5 months ago
Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC
Created
2023-02-05
420 commits to develop branch, last one 8 months ago
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
Created
2015-04-22
285 commits to master branch, last one about a month ago
WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.
Created
2023-10-23
67 commits to main branch, last one 9 days ago
Summarize web archive capture index (CDX) files.
Created
2021-11-01
120 commits to main branch, last one about a year ago
A robust web archive analytics toolkit
Created
2021-06-22
879 commits to develop branch, last one about a month ago