Category
Primary
web-development · embed-ft 0.73
Tree path
Technology & Computing › Computing › Internet › Web Development
Group (tier-1)
Tech stack
AI readiness
AI training policy
reserved
AI-bot protection
Blocked AI bots
AI files
Evidence
robots:ClaudeBot+Google-Extended+GPTBot+Meta-ExternalAgent+Amazonbot+Applebot-Extended+Bytespider+CCBot
Compliance (GEO / GDPR)
TLD
Overview
Title
CrawlStop.com - Anti-Crawling Test Site
Description
CrawlStop.com is designed to test web crawlers' compliance with anti-scraping measures and crawling restrictions. Features various deterrents and access controls.
Final URL
Language
en (html)
Scanned at
2026-06-20 18:09:40