Sign in
← Back to search
github Active

Repository profile

s0rg/crawley

The unix-way web crawler

Go MIT master Stack scanned README.md
Sign in to like s0rg/crawley
Open GitHub
Stars
339
Forks
18
Watchers
1
Issues
4
Commits
158
Awesome lists
2

Repository updates

Follow repository updates

Get generated s0rg/crawley development summaries by email, or follow the weekly and monthly RSS feeds.

Weekly RSS Monthly RSS

Sign in to subscribe by email. RSS feeds are public.

Sign in to subscribe

Activity and growth

Tracked growth, recent movement, and commit velocity from stored repository snapshots.

Latest capture 2026-07-06 03:08

Star growth, last 7 days
No 7-day history
Commit velocity, last 7 days
No 7-day history
Stars since baseline
-1
Snapshot coverage
5

Tracked growth

5 captures since 2026-05-22

Stars from baseline -1

Time horizon

All tracked data

Stars history

Total stars

Commits history

Default branch commits

Detected stack

Frameworks, package managers, ecosystems, and dependency manifests found during catalog scans.

Scanned 2026-07-06 03:08

Stack signals
0
Package managers
1
Manifest files
2
Dependencies
9

Frameworks and tools

  • No framework dependencies detected.
Go modules go

Dependency files

2 manifests
  • go.mod go ecosystem, 4 dependencies
  • go.sum go ecosystem, 5 dependencies

Classification

Searchable topics, generated tags, and stack labels that explain where this repository fits.

Topics
12
Tags
0
Stacks
0

Topics

#cli #crawler #go #golang #golang-application #pentest #pentest-tool #pentesting #unix-way #web-crawler #web-scraping #web-spider

Generated tags

No generated tags yet.

Stack labels

No stack labels yet.

AI development signals

Agent instructions and tool configuration paths found in the repository tree.

0 paths
No AI development config files detected.

Similar repositories

Nearest indexed repositories by embedding similarity.

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

9,339 stars
Python 1 awesome list

karust/gogetcrawl

Extract web archive data using Wayback Machine and Common Crawl

183 stars
Go 1 awesome list

AIMLPM/markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

3 stars
Python 1 awesome list

Pyx-Corp/spectrawl

The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One npm install, self-hosted.

26 stars
JavaScript 1 awesome list

N0taN3rd/Squidwarc

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

178 stars
JavaScript 1 awesome list

ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

1,598 stars
Python 1 awesome list

Metadata

Language
Go
License
MIT
Default branch
master
Created
2021-10-27
First commit
2021-10-27
Last pushed
2026-06-19
GitHub updated
2026-06-30
Last synced
2026-07-06 03:08
Stack detected
2026-07-06 03:08
Archived
no

Links and files

GitHub README

Appears in

2