30 results found Sort:
- Filter by Primary Language:
- Python (13)
- TypeScript (4)
- Java (3)
- JavaScript (3)
- Go (2)
- Rust (1)
- Scala (1)
- C (1)
- Cython (1)
- Roff (1)
- +
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Created
2017-05-05
4,606 commits to dev branch, last one 4 days ago
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Created
2011-10-21
2,624 commits to master branch, last one 6 days ago
Collect and revisit web pages.
Created
2015-05-13
1,889 commits to master branch, last one 3 years ago
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Created
2015-02-05
1,172 commits to master branch, last one 5 months ago
A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!
Created
2020-02-10
245 commits to main branch, last one about a month ago
Serverless replay of web archives directly in the browser
Created
2019-12-09
486 commits to main branch, last one 7 days ago
Run a high-fidelity browser-based web archiving crawler in a single Docker container
Created
2020-11-02
458 commits to main branch, last one about a month ago
InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS
Created
2016-03-04
1,618 commits to master branch, last one 2 months ago
WarcDB: Web crawl data as SQLite databases.
Created
2022-05-29
73 commits to main branch, last one 5 months ago
Streaming WARC/ARC library for fast web archive IO
Created
2017-03-06
157 commits to master branch, last one 20 days ago
:whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation
Created
2013-03-20
836 commits to main branch, last one 2 months ago
News crawling with StormCrawler - stores content as WARC
Created
2016-07-18
159 commits to master branch, last one about a year ago
Bitextor generates translation memories from multilingual websites
Created
2018-04-16
4,001 commits to master branch, last one about a month ago
WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.
Created
2023-10-23
211 commits to main branch, last one about a month ago
Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
Created
2021-06-28
1,437 commits to main branch, last one 7 days ago
Chrome extension to "Create WARC files from any webpage"
Created
2013-03-20
181 commits to main branch, last one about a year ago
CoCrawler is a versatile web crawler built using modern tools and concurrency.
Created
2016-07-15
1,142 commits to main branch, last one 2 years ago
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
Created
2018-03-03
259 commits to main branch, last one 3 months ago
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Created
2015-08-06
144 commits to master branch, last one 3 months ago
Drill into WARC web archives
Created
2023-12-07
78 commits to main branch, last one 2 months ago
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
Created
2019-09-07
4,429 commits to v1.21.3-at branch, last one about a month ago
🗄️ A simple CLI for converting WARC to Parquet.
Created
2022-06-20
72 commits to main branch, last one about a month ago
Parse And Create Web ARChive (WARC) files with node.js
Created
2017-05-21
114 commits to master branch, last one 5 years ago
A robust web archive analytics toolkit
Created
2021-06-22
930 commits to develop branch, last one 25 days ago
Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC
Created
2023-02-05
420 commits to develop branch, last one about a year ago
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
Created
2015-04-22
296 commits to master branch, last one 10 days ago
Summarize web archive capture index (CDX) files.
Created
2021-11-01
120 commits to main branch, last one 2 years ago
🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.
Created
2020-02-08
17 commits to master branch, last one 4 months ago
Command line tool to convert a file in the WARC format to a file in the ZIM format
Created
2020-07-10
630 commits to main branch, last one about a month ago
Command line tool for digging into WARC files
Created
2021-09-23
395 commits to main branch, last one 2 months ago