AI crawler comparison
CCBot vs Bytespider Common Crawl
Model training Respects robots.txt
2,330,028
websites block it · 4.672% of the web · #2
VS
ByteDance
Model training Ignores robots.txt
2,222,300
websites block it · 4.456% of the web · #4
The verdict
CCBot is blocked by more websites than Bytespider (2,330,028 vs 2,222,300). CCBot honours robots.txt; Bytespider does not , so a Disallow rule may not stop it.
CCBot Bytespider Operator Common Crawl ByteDance Purpose Model training Model training Respects robots.txt Respects robots.txt Ignores robots.txt Category AI Data Scrapers AI Data Scrapers Websites blocking it 2,330,028 2,222,300 Share of the web 4.672% 4.456% Rank #2 most-blocked #4 most-blocked Verify method reverse-dns none
⚖️ Compare any two AI crawlers
Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot vs Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot Compare →
Block both in robots.txt Add this to your site’s /robots.txt:
User-agent: CCBot
User-agent: Bytespider
Disallow: / Other comparisons ← AI Bot Observatory · Full database →