The AI Bot Observatory
A self-growing directory of 161 AI crawlers & agents — who operates them, what they do, and how the web responds. It expands automatically: every unrecognised bot we catch fetching our llms.txt / ai.txt is added. Backed by robots.txt policy across 48,534,257 live websites.
2,465,571
websites block ≥1 AI bot (5.08% of the web)
2,446,584
block a training bot (GPTBot, CCBot…)
414,936
block an answer engine (ChatGPT, Perplexity…)
💡 The web blocks AI training far more than AI answers. 5.9× more sites block a training crawler than an answer engine — most sites stay AI-visible (citable in ChatGPT/Perplexity) while opting out of model training.
Most-blocked AI crawlers
Share of the indexed web that disallows each bot in robots.txt.
1GPTBotOpenAI · Model training2,232,1064.599%2CCBotCommon Crawl · Model training2,204,6714.543%3AmazonbotAmazon · Model training2,106,0974.339%4ClaudeBotAnthropic · Model training2,102,7424.332%5BytespiderByteDance · Model training2,099,3884.326%6Google-ExtendedGoogle · Model training2,077,9854.281%7Meta-ExternalAgentMeta · Model training2,014,9394.152%8Applebot-ExtendedApple · Model training1,979,7734.079%9anthropic-aiAnthropic · Model training278,1390.573%10ChatGPT-UserOpenAI · User assistant245,3740.506%11PerplexityBotPerplexity · Answer engine216,7140.447%12FacebookBotMeta · Model training213,8160.441%13cohere-aiCohere · User assistant169,2700.349%14Claude-WebAnthropic · User assistant167,5050.345%15OmgilibotWebz.io · Search index164,1790.338%16OAI-SearchBotOpenAI · Answer engine144,9730.299%17DiffbotDiffbot · Data reseller141,9430.292%18YouBotYou.com · Answer engine130,9510.27%19ImagesiftBotImageSift · Search index125,8930.259%20TimpibotTimpi · Model training104,7020.216% Full AI bot registry · 161 agents
Ai2, a non-profit AI research institute
Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers
Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services
Baidu that fetches web content for the yiyan
Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi…
Butterfly Effect, a company based in China
Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products
CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services
Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications
GeistHaus, a company developing AI systems for therapy and psychological assessment
Google that retrieves web content on behalf of Gemini API users
https://brightdata.com/brightbot
Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…
Large-scale Artificial Intelligence Open Network
Lyrenth that builds an AI-readable index of web content for AI systems
Moonshot AI that fetches web content on behalf of users interacting with Kimi
Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts)
Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan…
Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg…
Querit, a company providing a search API for large language model integration
the Chinese company Huawei
Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…