Web Crawler
Configurable web crawler that extracts structured content from websites while respecting robots.txt rules and offering customizable settings for depth, delay, and concurrency.
About
This web crawler MCP server, implemented in TypeScript, provides a flexible and configurable tool for crawling websites and extracting content. It integrates with the Model Context Protocol SDK and uses libraries like Axios and Cheerio for efficient web scraping. The crawler respects robots.txt rules, handles concurrent requests, and offers customizable depth, delay, and timeout settings. It stands out by providing a simple JSON interface for initiating crawls and retrieving structured results, making it ideal for AI systems or applications that need to gather web content programmatically. Use cases include data collection for training models, content aggregation, or web monitoring tasks.
Is this your product? Claim this listing with an email at your domain to post updates.
Is this your project?
Claim this listing to manage your page, access analytics, and unlock upgrades. Verification takes 60 seconds.
Share This Project
Embed Badge
Add this badge to your README:
[](https://hifriendbot.com/ai-list/web-crawler/)
