AI crawler comparison
CCBot vs Amazonbot
Common Crawl
Model trainingRespects robots.txt
2,330,028
websites block it · 4.672% of the web · #2
VS
Amazon
Model trainingRespects robots.txt
2,228,896
websites block it · 4.469% of the web · #3
The verdict
CCBot is blocked by more websites than Amazonbot (2,330,028 vs 2,228,896).
| CCBot | Amazonbot |
|---|
| Operator | Common Crawl | Amazon |
| Purpose | Model training | Model training |
| Respects robots.txt | Respects robots.txt | Respects robots.txt |
| Category | AI Data Scrapers | AI Data Scrapers |
| Websites blocking it | 2,330,028 | 2,228,896 |
| Share of the web | 4.672% | 4.469% |
| Rank | #2 most-blocked | #3 most-blocked |
| Verify method | reverse-dns | ip-range |
⚖️ Compare any two AI crawlers
vs
Block both in robots.txt
Add this to your site’s /robots.txt:
User-agent: CCBot
User-agent: Amazonbot
Disallow: /
Other comparisons
← AI Bot Observatory · Full database →