ShapBot is a web crawler by Parallel that collects and structures web content to power its search, extraction, and deep research APIs, providing AI agents with high-accur…
Official documentation ↗
Add this to your site’s /robots.txt:
/robots.txt
User-agent: ShapBot Disallow: /
← All AI bots