Repository profile
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Repository updates
Get generated postmodern/spidr development summaries by email, or follow the weekly and monthly RSS feeds.
Sign in to subscribe by email. RSS feeds are public.
Sign in to subscribeTracked growth, recent movement, and commit velocity from stored repository snapshots.
Latest capture 2026-07-24 03:05
4 captures since 2026-06-09
Stars from baseline -1
All tracked data
Frameworks, package managers, ecosystems, and dependency manifests found during catalog scans.
Scanned 2026-07-24 03:05
Gemfile
ruby ecosystem,
12 dependencies
spidr.gemspec
ruby ecosystem,
0 dependencies
Searchable topics, generated tags, and stack labels that explain where this repository fits.
Agent instructions and tool configuration paths found in the repository tree.
Nearest indexed repositories by embedding similarity.
Interact with REST services in an ActiveRecord-like manner
The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One npm install, self-hosted.
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
Log and Analyze Outgoing HTTP Requests
The unix-way web crawler
Self-hosted webscraper.
