AI crawler comparison
Amazonbot vs Bytespider Amazon
Model training Respects robots.txt
2,228,896
websites block it · 4.469% of the web · #3
VS
ByteDance
Model training Ignores robots.txt
2,222,300
websites block it · 4.456% of the web · #4
The verdict
Amazonbot is blocked by more websites than Bytespider (2,228,896 vs 2,222,300). Amazonbot honours robots.txt; Bytespider does not , so a Disallow rule may not stop it.
Amazonbot Bytespider Operator Amazon ByteDance Purpose Model training Model training Respects robots.txt Respects robots.txt Ignores robots.txt Category AI Data Scrapers AI Data Scrapers Websites blocking it 2,228,896 2,222,300 Share of the web 4.469% 4.456% Rank #3 most-blocked #4 most-blocked Verify method ip-range none
⚖️ Compare any two AI crawlers
Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot vs Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot Compare →
Block both in robots.txt Add this to your site’s /robots.txt:
User-agent: Amazonbot
User-agent: Bytespider
Disallow: / Other comparisons ← AI Bot Observatory · Full database →