AI crawler comparison
PerplexityBot vs Google-Extended Perplexity
Answer engine Respects robots.txt
229,184
websites block it · 0.454% of the web · #12
VS
Google
Model training Respects robots.txt
2,269,449
websites block it · 4.499% of the web · #6
The verdict
Google-Extended is blocked by more websites than PerplexityBot (2,269,449 vs 229,184 — 9.9×). They serve different jobs: PerplexityBot is a answer engine , Google-Extended a model training .
PerplexityBot Google-Extended Operator Perplexity Google Purpose Answer engine Model training Respects robots.txt Respects robots.txt Respects robots.txt Category AI Search Crawlers AI Data Scrapers Websites blocking it 229,184 2,269,449 Share of the web 0.454% 4.499% Rank #12 most-blocked #6 most-blocked Verify method ip-range reverse-dns
⚖️ Compare any two AI crawlers
Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot vs Amazonbot anthropic-ai Applebot Applebot-Extended Bytespider CCBot ChatGPT Agent ChatGPT-User Claude-SearchBot Claude-User ClaudeBot Gemini-Deep-Research Google-Extended GoogleOther GPTBot meta-externalagent MistralAI-User OAI-SearchBot Operator Perplexity-User PerplexityBot Compare →
Block both in robots.txt Add this to your site’s /robots.txt:
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: / Other comparisons ← AI Bot Observatory · Full database →