AIMLPM/markcrawl
Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.
2 stars
Python
1 awesome list
unclecode/crawl4ai
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
FastAPIFlask
- Stars
- 67,581
- Forks
- 6,905
- Commits
- 1528
- Language
- Python
- Awesome lists
- 2
Similar repositories
LearningCircuit/local-deep-research
~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
8254 stars
Python
1 awesome list
firecrawl/firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
127429 stars
TypeScript
1 awesome list
amantus-ai/llm-codes
Transform developer documentation to clean Markdown
327 stars
TypeScript
0 awesome lists
Pyx-Corp/spectrawl
The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One npm install, self-hosted.
24 stars
JavaScript
1 awesome list
jina-ai/reader
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
10925 stars
TypeScript
1 awesome list
Tracked growth
5 captures since 2026-05-22
Latest capture 2026-06-02 07:22
Stars history
Total starsCommits history
Default branch commitsDetected stack
Frameworks and tools
- FastAPI · web framework · high confidence
- Flask · web framework · high confidence
pip
uv
Dependency files
-
pyproject.toml· python · 43 dependencies -
requirements.txt· python · 33 dependencies -
setup.cfg· python · 0 dependencies -
setup.py· python · 0 dependencies -
uv.lock· python · 0 dependencies -
deploy/docker/requirements.txt· python · 17 dependencies -
tests/memory/requirements.txt· python · 4 dependencies -
deploy/docker/tests/requirements.txt· python · 2 dependencies -
docs/examples/website-to-api/requirements.txt· python · 5 dependencies -
docs/examples/c4a_script/tutorial/requirements.txt· python · 2 dependencies
Metadata
- Created: 2024-05-09
- First commit: 2024-05-09
- Last pushed: 2026-06-01
- Website: https://crawl4ai.com
- Archived: no
- Stack detected: 2026-06-02 07:22
- License: Apache-2.0
AI development signals
AI agent config detected
4 config paths
2 files
2 directories
Claude Code
4
Key config paths
-
dir
.claude
Review config paths
-
Claude Code
.claude -
Claude Code
.claude/commands -
Claude Code
.claude/commands/c4ai-check.md -
Claude Code
.claude/settings.local.json