Awesome Web Archiving

Easy-to-use Web archiver

A dockerized, queued high fidelity web archiver based on Squidwarc

:gear: A Rust library for reading and writing WARC files

Java library for reading and writing WARC files with a typed API

Converts WARC files to static HTML

Chrome extension that uses Memento to indicate that a page a user is viewing on the live web has an archived copy and to give the user access to the copy

Convert HTTP Archive (HAR) -> Web Archive (WARC) format

NPM package and CLI tool for saving web page as single HTML file

A Rails engine supporting the discovery of web archives.

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

A whirlwind tour of Common Crawl's data using Python

Web archiving using Google Chrome

Web archive index server based on RocksDB

Prototype SOLR-powered web archive exploration UI.

Download and attach provenance to public datasets

Command-line tool and Rust library for handling Web ARChive (WARC) files

Converts HTTrack crawls to WARC files

No description.

Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.

An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

Wget with Lua extension

simple script to convert web resources to a single warc file

Web Archiving Course

A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.

Create Robust Links from within Zotero

DuckDB extension to fetch pages from Wayback Machine & Common Crawl

Web application for distributed compute analysis of Archive-It web archive collections.

golang readers for ARC and WARC webarchive formats

A tool for detecting viruses and NSFW material in WARC files

No description.

Internet Archive's Sparkling Data Processing Library

A data retrieval & exploration protocol designed to investigate science and policy processes

A client for the Archive-It And Webrecorder WASAPI Data Transfer API

Object Resource Stream and CDXJ Drafts

No description.

Playback webpages from Wayback Machine

A Tool to Summarize Web Archive Holdings

Tika based link (URL) extractor for httpreserve

The UKWA Heritrix3 custom modules and Docker builder.

Web archive deduplication tools

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

CLI implementation of httpreserve that can test links and retrieve internet archive replacements

No description.

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

Java application to download WARCs from WASAPI

DuckDB extension for parsing WARC files

JWAT Tools

A jupyter notebook illistrating the basics of Common Crawl's datasets.

Java Web Archive Toolkit

A whirlwind tour of Common Crawl's data using Java