AI crawler comparison
ClaudeBot vs Bytespider
Anthropic
Model trainingRespects robots.txt
2,840,278
websites block it · 4.636% of the web · #4
VS
ByteDance
Model trainingIgnores robots.txt
2,838,269
websites block it · 4.633% of the web · #5
The verdict
ClaudeBot is blocked by more websites than Bytespider (2,840,278 vs 2,838,269). ClaudeBot honours robots.txt; Bytespider does not, so a Disallow rule may not stop it.
| ClaudeBot | Bytespider | |
|---|---|---|
| Operator | Anthropic | ByteDance |
| Purpose | Model training | Model training |
| Respects robots.txt | Respects robots.txt | Ignores robots.txt |
| Category | AI Data Scrapers | AI Data Scrapers |
| Websites blocking it | 2,840,278 | 2,838,269 |
| Share of the web | 4.636% | 4.633% |
| Rank | #4 most-blocked | #5 most-blocked |
| Verify method | ip-range | none |
⚖️ Compare any two AI crawlers
vs
Block both in robots.txt
Add this to your site’s /robots.txt:
User-agent: ClaudeBot User-agent: Bytespider Disallow: /