Sign in
← Back to search
github Active

Repository profile

simplecto/sitemap_grabber

A python library to recursively crawl every sitemap.xml for a website. Also handles robots.txt and other well-knowns.

Python MIT master Stack scanned README.md
Sign in to like simplecto/sitemap_grabber
Open website Open GitHub
Stars
1
Forks
0
Watchers
1
Issues
5
Commits
77
Awesome lists
0

Repository updates

Follow repository updates

Get generated simplecto/sitemap_grabber development summaries by email, or follow the weekly and monthly RSS feeds.

Weekly RSS Monthly RSS

Sign in to subscribe by email. RSS feeds are public.

Sign in to subscribe

Activity and growth

Tracked growth, recent movement, and commit velocity from stored repository snapshots.

Latest capture 2026-07-17 04:31

Star growth, last 7 days
0 0.0%
Commit velocity, last 7 days
0 0.0%
Stars since baseline
0
Snapshot coverage
28

Tracked growth

28 captures since 2026-06-02

Stars from baseline 0

Time horizon

All tracked data

Stars history

Total stars

Commits history

Default branch commits

Detected stack

Frameworks, package managers, ecosystems, and dependency manifests found during catalog scans.

Scanned 2026-07-17 04:31

Stack signals
0
Package managers
2
Manifest files
2
Dependencies
0

Frameworks and tools

  • No framework dependencies detected.
PEP 517 pip python

Dependency files

2 manifests
  • pyproject.toml python ecosystem, 0 dependencies
  • requirements.txt python ecosystem, 0 dependencies

Classification

Searchable topics, generated tags, and stack labels that explain where this repository fits.

Topics
4
Tags
0
Stacks
0

Topics

#robots-txt #security-txt #sitemap-generator #well-known

Generated tags

No generated tags yet.

Stack labels

No stack labels yet.

AI development signals

Agent instructions and tool configuration paths found in the repository tree.

0 paths
No AI development config files detected.

Similar repositories

Nearest indexed repositories by embedding similarity.

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

9,330 stars
Python 1 awesome list

s0rg/crawley

The unix-way web crawler

339 stars
Go 2 awesome lists

firecrawl/firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

152,085 stars
TypeScript 1 awesome list

scanapi/scanapi

Automated Integration Testing and Live Documentation for your API

1,569 stars
Python 1 awesome list

Metadata

Language
Python
License
MIT
Default branch
master
Created
2024-06-05
First commit
2024-06-05
Last pushed
2024-10-23
GitHub updated
2025-05-07
Last synced
2026-07-17 04:31
Stack detected
2026-07-17 04:31
Archived
no

Links and files

GitHub Website

https://pypi.org/project/sitemap_grabber/

README

403 Forbidden | https://api.github.com/repos/simplecto/sitemap_grabber/readme | message=API rate limit exceeded for user ID 8257474. If you reach out to GitHub Support for help, please include the request ID D9EC:1E605F:BB0F061:B09F516:6A59B036 and timestamp 2026-07-17 04:31:50 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https | rate_limit_remaining=0 | rate_limit_reset=1784264421

Appears in

0
  • No awesome list links recorded.