Category & trade
Primary
artificial-intelligence · embed-ft 0.39
Tree path
Technology & Computing › Artificial Intelligence
Group (tier-1)
Tech stack
AI readiness
AI-Readiness score
34 / 100 · partial · see similar
AI training policy
prohibited
ToS prohibits AI
yes
Terms of Service
Scraping ban
yes
AI-bot protection
Monetization signal
opportunity This site reserves AI-training rights — Cloudflare Pay-Per-Crawl could charge AI bots here with one setting.
AI files
Evidence
tos:no-ai-trainingscraping-ban
Compliance (GEO / GDPR)
TLD
Overview
Title
Common Crawl - Open Repository of Web Crawl Data
Description
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
Final URL
Language
en (html)
Scanned at
2026-07-18 15:06:25