Highlighted
Open highlighted repo slot
Put your repository first
Promote a GitHub repo at the top of Awesome repository list views for 7 days.
Awesome
GitHub projects from awesome lists
Search awesome repositories
Search names, descriptions, topics, tags, and stacks, then tune results by ecosystem, freshness, health, and cross-list signal.
- Repos indexed
- 17,373
- Awesome lists tracked
- 125
- Current results
- 22
22 repos shown
firecrawl/firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
AI dev
Stack
GitHub topics
- Updated
- 2026-07-17
- Lists
- 1 list mention
- First commit
- 2024-04-15
- History
- 69 history points
- License
- AGPL-3.0
- Issues
- 408 open
152,085
stars
- Forks
- 8,688
- Commits
- 5,959 commits
- Star growth, last 7 days
- +3,110 +2.1%
- Commit velocity, last 7 days
- +62 +1.1%
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
GitHub topics
- Updated
- 2026-07-15
- Lists
- 1 list mention
- First commit
- 2024-10-13
- History
- 42 history points
- License
- BSD-3-Clause
- Issues
- 4 open
69,810
stars
- Forks
- 6,915
- Commits
- 1,532 commits
- Star growth, last 7 days
- +750 +1.1%
- Commit velocity, last 7 days
- +17 +1.1%
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub topics
- Updated
- 2026-07-13
- Lists
- 2 list mentions
- First commit
- 2009-05-04
- History
- 28 history points
- License
- BSD-3-Clause
- Issues
- 587 open
63,189
stars
- Forks
- 11,789
- Commits
- 11,204 commits
- Star growth, last 7 days
- +116 +0.2%
- Commit velocity, last 7 days
- +2 0.0%
iawia002/lux
👾 Fast and simple video download library and CLI tool written in Go
Stack
- Updated
- 2026-03-29
- Lists
- 1 list mention
- First commit
- 2018-02-06
- History
- 2 history points
- License
- MIT
- Issues
- 545 open
31,451
stars
- Forks
- 3,307
- Commits
- 869 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
- Updated
- 2026-07-15
- Lists
- 0 list mentions
- First commit
- 2024-01-27
- History
- 24 history points
- License
- MIT
- Issues
- 7 open
28,414
stars
- Forks
- 2,759
- Commits
- 2,889 commits
- Star growth, last 7 days
- +157 +0.6%
- Commit velocity, last 7 days
- +1 0.0%
s0md3v/Photon
Incredibly fast crawler designed for OSINT.
- Updated
- 2026-02-10
- Lists
- 3 list mentions
- First commit
- 2018-03-30
- History
- 3 history points
- License
- GPL-3.0
- Issues
- 58 open
13,015
stars
- Forks
- 1,677
- Commits
- 331 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
AI dev
GitHub topics
- Updated
- 2026-07-16
- Lists
- 1 list mention
- First commit
- 2024-01-10
- History
- 25 history points
- License
- Apache-2.0
- Issues
- 80 open
9,330
stars
- Forks
- 777
- Commits
- 1,707 commits
- Star growth, last 7 days
- +33 +0.4%
- Commit velocity, last 7 days
- +26 +1.5%
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- Updated
- 2026-07-09
- Lists
- 1 list mention
- First commit
- 2019-04-04
- History
- 8 history points
- License
- Apache-2.0
- Issues
- 73 open
6,290
stars
- Forks
- 395
- Commits
- 1,628 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
myreader-io/myGPTReader
A community-driven way to read and chat with AI bots - powered by chatGPT.
GitHub topics
- Updated
- 2026-02-05
- Lists
- 1 list mention
- First commit
- 2023-03-08
- History
- 4 history points
- License
- MIT
- Issues
- 15 open
4,419
stars
- Forks
- 441
- Commits
- 305 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
Nekmo/dirhunt
Find web directories without bruteforce
GitHub topics
- Updated
- 2023-10-29
- Lists
- 1 list mention
- First commit
- 2018-01-05
- History
- 2 history points
- License
- MIT
- Issues
- 15 open
2,007
stars
- Forks
- 272
- Commits
- 329 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
AI dev
GitHub topics
- Updated
- 2026-07-02
- Lists
- 2 list mentions
- First commit
- 2026-03-23
- History
- 8 history points
- License
- AGPL-3.0
- Issues
- 0 open
1,725
stars
- Forks
- 180
- Commits
- 228 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
ArchiveTeam/grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
- Updated
- 2025-05-23
- Lists
- 1 list mention
- First commit
- 2015-02-05
- History
- 5 history points
- License
- NOASSERTION
- Issues
- 103 open
1,598
stars
- Forks
- 157
- Commits
- 1,174 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
felipecsl/wombat
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
- Updated
- 2026-04-07
- Lists
- 1 list mention
- First commit
- 2011-11-14
- History
- 3 history points
- License
- MIT
- Issues
- 24 open
1,363
stars
- Forks
- 128
- Commits
- 300 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
orangecoding/fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
AI dev
GitHub topics
- Updated
- 2026-07-09
- Lists
- 1 list mention
- First commit
- 2018-01-20
- History
- 5 history points
- License
- Apache-2.0
- Issues
- 8 open
1,260
stars
- Forks
- 186
- Commits
- 718 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
codelibs/fess
Fess is very powerful and easily deployable Enterprise Search Server.
AI dev
- Updated
- 2026-07-05
- Lists
- 1 list mention
- First commit
- 2013-09-13
- History
- 5 history points
- License
- Apache-2.0
- Issues
- 15 open
1,118
stars
- Forks
- 174
- Commits
- 4,845 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
Stack
GitHub topics
- Updated
- 2026-07-11
- Lists
- 1 list mention
- First commit
- 2020-10-31
- History
- 5 history points
- License
- AGPL-3.0
- Issues
- 137 open
1,079
stars
- Forks
- 147
- Commits
- 696 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
0x676e67/wreq
An ergonomic, censorship-resistant Rust HTTP Client
GitHub topics
- Updated
- 2026-07-13
- Lists
- 1 list mention
- First commit
- 2016-07-01
- History
- 5 history points
- License
- Apache-2.0
- Issues
- 14 open
921
stars
- Forks
- 110
- Commits
- 2,857 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
- Updated
- 2026-01-12
- Lists
- 1 list mention
- First commit
- 2008-05-19
- History
- 3 history points
- License
- MIT
- Issues
- 16 open
836
stars
- Forks
- 107
- Commits
- 898 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
s0rg/crawley
The unix-way web crawler
Stack
- Updated
- 2026-06-19
- Lists
- 2 list mentions
- First commit
- 2021-10-27
- History
- 5 history points
- License
- MIT
- Issues
- 4 open
339
stars
- Forks
- 18
- Commits
- 158 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
karust/gogetcrawl
Extract web archive data using Wayback Machine and Common Crawl
Stack
GitHub topics
- Updated
- 2024-11-04
- Lists
- 1 list mention
- First commit
- 2019-06-14
- History
- 5 history points
- License
- MIT
- Issues
- 0 open
183
stars
- Forks
- 17
- Commits
- 31 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
N0taN3rd/Squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Stack
GitHub topics
- Updated
- 2020-05-19
- Lists
- 1 list mention
- First commit
- 2017-07-20
- History
- 5 history points
- License
- Apache-2.0
- Issues
- 11 open
178
stars
- Forks
- 25
- Commits
- 119 commits
- Star growth, last 7 days
- 0 0.0%
- Commit velocity, last 7 days
- 0 0.0%
JakePartusch/lumberjack
An automated website accessibility scanner and cli
Stack
GitHub topics
- Updated
- 2023-03-03
- Lists
- 1 list mention
- First commit
- 2020-02-05
- History
- 1 history point
- License
- MIT
- Issues
- 4 open
127
stars
- Forks
- 7
- Commits
- 34 commits
- Star growth, last 7 days
- No 7-day history
- Commit velocity, last 7 days
- No 7-day history
