Rock (Website Crawler)
Documentation crawler that indexes websites into SQLite with full-text search, enabling AI agents to search, read, and expand on crawled content on demand.
About
Crawls documentation websites and indexes their content into a local SQLite database with full-text search capabilities. Provides nine tools for web search via DuckDuckGo, async URL crawling, full-text doc search, page retrieval with section jumping, topic expansion, source management, and recrawling. Configurable crawl depth (2 hops default), max pages (50 per source), and concurrent request limits with redirect deduplication and HTML-only indexing.
Is this your product? Claim this listing with an email at your domain to post updates.
Is this your project?
Claim this listing to manage your page, access analytics, and unlock upgrades. Verification takes 60 seconds.
Share This Project
Embed Badge
Add this badge to your README:
[](https://hifriendbot.com/ai-list/rock-website-crawler/)
