Common Crawl · AI crawler
2,204,671 websites block CCBot
That’s 4.543% of the 48,534,257 live websites we index — the #2 most-blocked AI crawler.
Model trainingRespects robots.txtAI Data Scrapers
Web archive going back to 2008. [Cited in thousands of research papers per year](https://commoncrawl.org/research-papers).
Official documentation ↗
🛰️ We haven’t observed CCBot crawling piperic.com yet. This page updates automatically when it does.
Block CCBot in robots.txt
Add this to your site’s /robots.txt:
User-agent: CCBot
Disallow: /
Who blocks CCBot
Top countries blocking it
Top categories blocking it
← All AI bots