That’s 0.345% of the 48,534,257 live websites we index — the #14 most-blocked AI crawler.
Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can provide more detail, please contact us.
Add this to your site’s /robots.txt:
User-agent: Claude-Web Disallow: /