Category
Primary
web-development · embed-ft 0.68
Tree path
Technology & Computing › Computing › Internet › Web Development
Group (tier-1)
Tech stack
AI readiness
AI training policy
reserved
AI-bot protection
Blocked AI bots
AI files
Evidence
robots:ClaudeBot+Google-Extended+GPTBot+Meta-ExternalAgent+Amazonbot+Applebot-Extended+Bytespider+CCBot
Compliance (GEO / GDPR)
TLD
Overview
Title
CrawlTest.com - AI Crawler Testing Ground
Description
CrawlTest.com provides a structured environment for testing AI crawlers, web scrapers, and indexing tools. Features semantic HTML, JSON-LD data, and best practices for content extraction.
Final URL
Language
en (html)
Scanned at
2026-06-20 18:09:40