How AI crawling and AI-blocking change over time — the share of the web that blocks AI bots (updated weekly from our 48M-domain policy index) and live AI-bot activity on our own network (daily). ← back to the Observatory
Share of the web blocking AI crawlers
% of live websites that disallow ≥1 AI bot in robots.txt · now 5.207%
📈 Collecting — a new data point is added automatically; the trend line fills in as history builds.
Training crawlers vs answer engines blocked
Websites blocking a training bot (orange) vs an answer engine (blue)
📈 Collecting weekly.
AI bot crawl activity on our network
Total AI-bot requests we log per day (live signal)