Highlighted
Open highlighted repo slot
Put your repository first
Promote a GitHub repo at the top of Awesome repository list views for 7 days.
Awesome
Awesome-list intelligence for GitHub
Search every repository hiding inside awesome lists.
Discover projects curated by awesome-list maintainers, then narrow them by stars, age, freshness, archive status, language, topics, generated tags, detected stacks, package managers, and source list.
- Repos indexed
- 9,926
- Awesome lists tracked
- 76
- Current results
- 9
9 repos shown
Topic: web-crawler
firecrawl/firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
TypeScript
AstroExpresspytestReact
BundlerCargoComposer
#ai#ai-agents#ai-crawler#ai-scraping#ai-search
AI dev signals
1 awesome list
5571 commits
first commit 2024-04-15
3 history points
updated 2026-06-01
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Python
LangChain
uv
#ai-crawler#ai-scraping#ai-search#crawler#data-extraction
0 awesome lists
2877 commits
first commit 2024-01-27
1 history point
updated 2026-06-01
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Python
pytestReact
npmPEP 517pip
#apify#automation#beautifulsoup#crawler#crawling
AI dev signals
1 awesome list
1558 commits
first commit 2024-01-10
2 history points
updated 2026-06-02
adithya-s-k/omniparse
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks
Python
#ingestion-api#ocr#omniparser#parse-server#parser-library
1 awesome list
122 commits
1 history point
updated 2025-12-12
MarginaliaSearch/MarginaliaSearch
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
Java
Gradlepip
#alt-search#indexer#internet-search#java#language-processing
0 awesome lists
4205 commits
first commit 2022-05-19
1 history point
updated 2026-05-23
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Rust
#ai#ai-agents#ai-scraping#cli#crawler
AI dev signals
2 awesome lists
160 commits
first commit 2026-03-23
5 history points
updated 2026-05-31
webrecorder/browsertrix-crawler
Run a high-fidelity browser-based web archiving crawler in a single Docker container
TypeScript
#crawler#crawling#wacz#warc#web-archiving
1 awesome list
649 commits
first commit 2020-10-31
2 history points
updated 2026-05-26
graphlit/graphlit-mcp-server
Model Context Protocol (MCP) Server for Graphlit Platform
TypeScript
#claude#content-extraction#content-ingestion#data-collection#llm-tools
1 awesome list
261 commits
2 history points
updated 2026-01-12
s0rg/crawley
The unix-way web crawler
Go
#cli#crawler#go#golang#golang-application
2 awesome lists
157 commits
first commit 2021-10-27
2 history points
updated 2026-05-08