Highlighted
Open highlighted repo slot
Put your repository first
Promote a GitHub repo at the top of Awesome repository list views for 7 days.
Awesome
GitHub projects from awesome lists
Search awesome repositories
Search names, descriptions, topics, tags, and stacks, then tune results by ecosystem, freshness, health, and cross-list signal.
- Repos indexed
- 9,945
- Awesome lists tracked
- 76
- Current results
- 103
103 repos shown
commoncrawl/whirlwind-python
A whirlwind tour of Common Crawl's data using Python
PromyLOPh/crocoite
Web archiving using Google Chrome
Python
#archiving#chrome-browser#devtools#warc
archived
1 awesome list
282 commits
first commit 2017-11-17
2 history points
updated 2019-12-30
nla/outbackcdx
Web archive index server based on RocksDB
Java
#wayback#web-archiving
1 awesome list
588 commits
first commit 2015-01-15
2 history points
updated 2026-05-01
ukwa/shine
Prototype SOLR-powered web archive exploration UI.
JavaScript
archived
1 awesome list
748 commits
first commit 2013-07-03
2 history points
updated 2020-06-03
harvard-lil/bag-nabit
Download and attach provenance to public datasets
Python
1 awesome list
39 commits
first commit 2024-11-28
2 history points
updated 2025-03-31
nla/httrack2warc
Converts HTTrack crawls to WARC files
Java
#web-archiving
1 awesome list
117 commits
first commit 2017-10-23
2 history points
updated 2024-08-06
chfoo/warcat-rs
Command-line tool and Rust library for handling Web ARChive (WARC) files
Rust
1 awesome list
81 commits
first commit 2024-09-15
2 history points
updated 2025-06-02
webis-de/wasp
No description.
Java
1 awesome list
57 commits
first commit 2018-03-25
2 history points
updated 2022-10-14
archivesunleashed/notebooks
Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.
Jupyter Notebook
#juypter-notebook#notebooks#pyspark-notebook#python3#spark
1 awesome list
185 commits
first commit 2019-11-06
2 history points
updated 2022-12-05
helgeho/Web2Warc
An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
Scala
1 awesome list
11 commits
first commit 2016-01-29
2 history points
updated 2017-10-09
alard/wget-lua
Wget with Lua extension
C
1 awesome list
3507 commits
first commit 1999-12-02
2 history points
updated 2015-12-17
steffenfritz/html2warc
simple script to convert web resources to a single warc file
Python
1 awesome list
16 commits
first commit 2015-01-19
2 history points
updated 2023-05-11
vphill/web-archiving-course
Web Archiving Course
1 awesome list
103 commits
first commit 2022-02-22
2 history points
updated 2024-03-04
lanl/Zotero-Robust-Links-Extension
Create Robust Links from within Zotero
JavaScript
#link-rot#memento#reference-rot#references#web-archives
1 awesome list
94 commits
first commit 2019-08-29
2 history points
updated 2022-05-10
internetarchive/arch
Web application for distributed compute analysis of Archive-It web archive collections.
Scala
1 awesome list
569 commits
first commit 2020-11-30
2 history points
updated 2026-03-24
midwork-finds-jobs/duckdb-web-archive
DuckDB extension to fetch pages from Wayback Machine & Common Crawl
C++
#archived#cdx-api#common-crawl#duckdb#duckdb-extension
AI dev signals
1 awesome list
68 commits
first commit 2025-11-21
2 history points
updated 2026-02-03
richardlehane/webarchive
golang readers for ARC and WARC webarchive formats
Go
1 awesome list
57 commits
first commit 2015-09-21
2 history points
updated 2023-04-03
natliblux/warc-safe
A tool for detecting viruses and NSFW material in WARC files
Python
#antivirus#nsfw-classifier#warc#warc-safe#webarchiving
1 awesome list
30 commits
first commit 2024-05-03
2 history points
updated 2026-05-20
nlnwa/gowarcserver
No description.
Go
1 awesome list
372 commits
first commit 2021-02-02
2 history points
updated 2025-03-31
Guillaume-Levrier/PANDORAE
A data retrieval & exploration protocol designed to investigate science and policy processes
JavaScript
1 awesome list
1514 commits
first commit 2018-12-10
2 history points
updated 2026-05-28
internetarchive/Sparkling
Internet Archive's Sparkling Data Processing Library
Scala
1 awesome list
47 commits
first commit 2022-04-28
2 history points
updated 2026-05-04
unt-libraries/py-wasapi-client
A client for the Archive-It And Webrecorder WASAPI Data Transfer API
Python
1 awesome list
118 commits
first commit 2017-08-10
2 history points
updated 2019-10-18
oduwsdl/ORS
Object Resource Stream and CDXJ Drafts
1 awesome list
20 commits
first commit 2015-10-06
2 history points
updated 2018-11-28
harvard-lil/warcbench
A tool for exploring, analyzing, transforming, recombining, and extracting data from WARC (Web ARChive) files.
Python
1 awesome list
348 commits
first commit 2025-02-24
2 history points
updated 2025-07-30
emmadickson/unwarcit
No description.
Python
1 awesome list
8 commits
first commit 2021-12-13
2 history points
updated 2022-01-07
wabarc/playback
Playback webpages from Wayback Machine
Go
#go#playback#wayback-machine#webpage
1 awesome list
21 commits
first commit 2021-04-17
2 history points
updated 2026-04-25
httpreserve/tikalinkextract
Tika based link (URL) extractor for httpreserve
HTML
#archives#code4lib#digitalpreservation#httpreserve#iipc
1 awesome list
51 commits
first commit 2017-04-03
2 history points
updated 2025-04-26
oduwsdl/MementoMap
A Tool to Summarize Web Archive Holdings
Python
#memento#mementomap#profiling#python#ukvs
1 awesome list
63 commits
first commit 2019-01-20
2 history points
updated 2021-06-15
ukwa/ukwa-heritrix
The UKWA Heritrix3 custom modules and Docker builder.
Java
archived
1 awesome list
575 commits
first commit 2013-06-06
2 history points
updated 2024-12-02
arcalex/warcrefs
Web archive deduplication tools
Java
1 awesome list
39 commits
first commit 2014-04-22
2 history points
updated 2018-10-18