AI Crawler Database — the complete list of AI bot user agents

Q: How do I block AI crawlers?

Add the crawlers you want to exclude to your site's /robots.txt with a User-agent line each and a Disallow: /. Use the generator on this page to copy-paste (or download) a ready-made robots.txt that blocks every known AI bot, kept up to date automatically.

Q: What is the GPTBot user agent, and how do I block it?

GPTBot is OpenAI's training crawler. Block it with User-agent: GPTBot then Disallow: / in robots.txt. See the GPTBot profile for its full user-agent string and how many sites block it.

The most complete, searchable list of 930 AI crawlers, bots & agents — 228 AI + 702 adjacent (search, SEO, feed) — with their user-agent strings, operator, purpose and whether they respect robots.txt. Aggregated from every open source and cross-checked against robots.txt policy on 48M+ live websites. Updated 2026-07-17. ← back to the Observatory

930

agents in the database

228

AI crawlers

702

adjacent bots (search, SEO, feed)

Block all AI crawlers — robots.txt

Copy-paste this into your site’s /robots.txt to ask every known AI crawler not to use your content. Updated automatically as new bots appear.

# Block AI crawlers in robots.txt — generated by Piperic
# The complete, updated list: https://piperic.com/tools/ai-bots

User-agent: AddSearchBot
User-agent: AgentTimes
User-agent: AGI Agent
User-agent: AI Search
User-agent: AI Search External
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
User-agent: aiHitBot
User-agent: AIWebIndex
User-agent: Alphalens Bot
User-agent: Amazon Bedrock AgentCore Browser (AP Northeast)
User-agent: Amazon Bedrock AgentCore Browser (AP South)
User-agent: Amazon Bedrock AgentCore Browser (AP Southeast 1)
User-agent: Amazon Bedrock AgentCore Browser (AP Southeast)
User-agent: Amazon Bedrock AgentCore Browser (EU Central 1)
User-agent: Amazon Bedrock AgentCore Browser (EU West 1)
User-agent: Amazon Bedrock AgentCore Browser (US East 1)
User-agent: Amazon Bedrock AgentCore Browser (US East 2)
User-agent: Amazon Bedrock AgentCore Browser (US West 2)
User-agent: amazon-kendra
User-agent: amazon-QBusiness
User-agent: Amazonbot
User-agent: Amazonbot-Video
User-agent: AmazonBuyForMe
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Anchor Browser
User-agent: Andibot
User-agent: Anomura
User-agent: anthropic-ai
User-agent: Apify Website Content Crawler
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Aranet-SearchBot
User-agent: atlassian-bot
User-agent: Awario
User-agent: AwarioSmartBot
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: Big Sur AI
User-agent: bigsur.ai
User-agent: BorderxBot
User-agent: Brandwatch
User-agent: Bravebot
User-agent: Brightbot
User-agent: Brightbot 1.0
User-agent: Browserbase
User-agent: BuddyBot
User-agent: Bytespider
User-agent: CCBot
User-agent: ccBot crawler
User-agent: Channel3Bot
User-agent: ChatGLM-Spider
User-agent: ChatGPT Agent
User-agent: ChatGPT-Browser
User-agent: ChatGPT-User
User-agent: Chathive crawler
User-agent: CitibotSiteCrawler
User-agent: Claude
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: Cledara SaaS Management Agent
User-agent: Cloudflare Browser Run
User-agent: Cloudflare Crawler
User-agent: Cloudflare-AutoRAG
User-agent: CloudVertexBot
User-agent: Code
User-agent: cohere-ai
User-agent: Cohere-Command
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: CragCrawler
User-agent: Crawl4AI
User-agent: Crawlspace
User-agent: Cursor
User-agent: Daric2
User-agent: Daric3
User-agent: Daric4
User-agent: Datenbank Crawler
User-agent: DeepSeekBot
User-agent: Devin
User-agent: Diffbot
User-agent: Direqt Anomura
User-agent: Doximity-Diffbot
User-agent: DuckAssistBot
User-agent: EasyScan
User-agent: Echobot Bot
User-agent: EchoboxBot
User-agent: Element451Bot
User-agent: ExaBot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FirmlyAI Bot
User-agent: FishBot
User-agent: FriendlyCrawler
User-agent: GeistHaus-PageFetcher
User-agent: Gemini-AI
User-agent: Gemini-Deep-Research
User-agent: Generic Bot
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: HenkBot
User-agent: Henry Shopping Agent
User-agent: HIFIBot
User-agent: iAskBot
User-agent: iaskspider
User-agent: iaskspider/2.0
User-agent: IbouBot
User-agent: ICC-Crawler
User-agent: ImageSift
User-agent: ImagesiftBot
User-agent: imageSpider
User-agent: img2dataset
User-agent: Instapaper
User-agent: ISSCyberRiskCrawler
User-agent: kagi-fetcher
User-agent: Kagibot
User-agent: Kangaroo Bot
User-agent: Kernel
User-agent: Kimi-User
User-agent: KimiBot
User-agent: KlaviyoAIBot
User-agent: KunatoCrawler
User-agent: laion-huggingface-processor
User-agent: LAIONDownloader
User-agent: LCC
User-agent: LINER Bot
User-agent: LinerBot
User-agent: Linguee Bot
User-agent: LinkupBot
User-agent: Make.com
User-agent: Manus Bot
User-agent: Manus-User
User-agent: Meta-ExternalAds
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
User-agent: Mozilla-Tabstack
User-agent: MyCentralAIScraperBot
User-agent: NagetBot
User-agent: Nava Labs ASP (Dev)
User-agent: Navu
User-agent: netEstate Imprint Crawler
User-agent: newsai
User-agent: NotebookLM
User-agent: NovaAct
User-agent: Novellum AI Crawl
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
User-agent: OpenAI
User-agent: opencode
User-agent: Operator
User-agent: PanguBot
User-agent: Panscient
User-agent: panscient.com
User-agent: Payhawk Invoice Fetching Agent
User-agent: payroll-bot
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poggio-Citations
User-agent: Poseidon Research Crawler
User-agent: QAtechBot
User-agent: QualifiedBot
User-agent: Querit-SearchBot
User-agent: QueritBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: Retool
User-agent: RyeBot
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: Selectika AI
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: SemrushBotSwa
User-agent: Shap-User
User-agent: ShapBot
User-agent: Sidetrade indexer bot
User-agent: Spider
User-agent: Strivve Automation
User-agent: TavilyBot
User-agent: Terra Cotta
User-agent: TerraCotta
User-agent: Thinkbot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: TongyiBot
User-agent: Trae
User-agent: TwinAgent
User-agent: UseAI
User-agent: VelenPublicWebCrawler
User-agent: Visually.io Shopify Editor
User-agent: WARDBot
User-agent: webzio
User-agent: Webzio-Extended
User-agent: wpbot
User-agent: WRTNBot
User-agent: Yahoo Gemini
User-agent: YaK
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: YGS Group Falconer Scraper
User-agent: YiyanBot
User-agent: YouBot
User-agent: ZanistaBot
Disallow: /

Download (AI only)Download (AI + adjacent)

Identify a user-agent

The full database · 930 crawlers

Bot	Operator	Type	User-agent	robots.txt	Blocked by
Amazonbot	Amazon	Model training	`amazonbot`	Respects robots.txt	2,228,896
anthropic-ai	Anthropic	Model training	`anthropic ai`	Unclear	287,136
Applebot	Apple	Search index	`applebot`	Respects robots.txt	—
Applebot-Extended	Apple	Model training	`applebot-extended`	Respects robots.txt	2,093,234
Bytespider	ByteDance	Model training	`bytespider`	Ignores robots.txt	2,222,300
CCBot	Common Crawl	Model training	`ccbot`	Respects robots.txt	2,330,028
ChatGPT Agent	OpenAI	Browsing agent	`chatgpt agent`	Respects robots.txt	—
ChatGPT-User	OpenAI	User assistant	`chatgpt-user`	Respects robots.txt	253,965
Claude-SearchBot	Anthropic	Answer engine	`claude-searchbot`	Respects robots.txt	—
Claude-User	Anthropic	User assistant	`claude-user`	Respects robots.txt	—
ClaudeBot	Anthropic	Model training	`claudebot`	Respects robots.txt	2,220,022
Gemini-Deep-Research	Google	Browsing agent	`gemini-deep-research`	Unclear	—
Google-Extended	Google	Model training	`google-extended`	Respects robots.txt	2,200,685
GoogleOther	Google	Data reseller	`googleother`	Respects robots.txt	—
GPTBot	OpenAI	Model training	`gptbot`	Respects robots.txt	2,358,506
meta-externalagent	Meta	Model training	`meta-externalagent`	Respects robots.txt	2,127,714
MistralAI-User	Mistral	User assistant	`mistralai-user`	Unclear	—
OAI-SearchBot	OpenAI	Answer engine	`oai-searchbot`	Respects robots.txt	148,213
Operator	OpenAI	Browsing agent	`operator`	Unclear	—
Perplexity-User	Perplexity	User assistant	`perplexity-user`	Ignores robots.txt	—
PerplexityBot	Perplexity	Answer engine	`perplexitybot`	Respects robots.txt	225,076
AddSearchBot	Addsearch	Search index	`addsearchbot`	Unclear	—
AgentTimes	The Agent Times	Data reseller	`agenttimes`	Unclear	—
AGI Agent	AGI, Inc	User assistant	`agi agent`	Unclear	—
AI Search	Cloudflare	Answer engine	`cloudflare-ai-search`	Respects robots.txt	—
AI Search External	Cloudflare	Answer engine	`cloudflare-ai-search-external`	Unclear	—
AI2Bot	Ai2	Model training	`ai2bot`	Respects robots.txt	—
AI2Bot-DeepResearchEval	Ai2, a non-profit AI research institute	User assistant	`ai2bot-deepresearcheval`	Unclear	—
Ai2Bot-Dolma	Ai2	Model training	`ai2bot-dolma`	Respects robots.txt	—
aiHitBot	aiHit	Model training	`aihitbot`	Respects robots.txt	—
AIWebIndex	Lyrenth that builds an AI-readable index of web content for AI systems	Data reseller	`aiwebindex`	Unclear	—
Alphalens Bot	Alphalens	Answer engine	`alphalens-bot`	Respects robots.txt	—
Amazon Bedrock AgentCore Browser (AP Northeast)	Amazon	User assistant	`amazon bedrock agentcore browser (ap northeast)`	Unclear	—
Amazon Bedrock AgentCore Browser (AP South)	Amazon	User assistant	`amazon bedrock agentcore browser (ap south)`	Unclear	—
Amazon Bedrock AgentCore Browser (AP Southeast 1)	Amazon	User assistant	`amazon bedrock agentcore browser (ap southeast 1`	Unclear	—
Amazon Bedrock AgentCore Browser (AP Southeast)	Amazon	User assistant	`amazon bedrock agentcore browser (ap southeast)`	Unclear	—
Amazon Bedrock AgentCore Browser (EU Central 1)	Amazon	User assistant	`amazon bedrock agentcore browser (eu central 1)`	Unclear	—
Amazon Bedrock AgentCore Browser (EU West 1)	Amazon	User assistant	`amazon bedrock agentcore browser (eu west 1)`	Unclear	—
Amazon Bedrock AgentCore Browser (US East 1)	Amazon	User assistant	`amazon bedrock agentcore browser (us east 1)`	Unclear	—
Amazon Bedrock AgentCore Browser (US East 2)	Amazon	User assistant	`amazon bedrock agentcore browser (us east 2)`	Unclear	—
Amazon Bedrock AgentCore Browser (US West 2)	Amazon	User assistant	`amazon bedrock agentcore browser (us west 2)`	Unclear	—
amazon-kendra	Amazon	Search index	`amazon kendra`	Respects robots.txt	—
amazon-QBusiness	Unknown	User assistant	`amazon-qbusiness`	Unclear	—
Amazonbot-Video	Unknown	Data reseller	`amazonbot-video`	Unclear	—
AmazonBuyForMe	Amazon	Browsing agent	`amazonbuyforme`	Unclear	—
Amzn-SearchBot	Amazon	Search index	`amzn-searchbot`	Unclear	—
Amzn-User	Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services	User assistant	`amzn-user`	Unclear	—
Anchor Browser	Anchor	Model training	`anchor browser`	Unclear	—
Andibot	Andi	Search index	`andibot`	Unclear	—
Anomura	Direqt	Search index	`anomura`	Respects robots.txt	—
Apify Website Content Crawler	Apify	User assistant	`apifywebsitecontentcrawler`	Unclear	—
ApifyBot	Unknown	Data reseller	`apifybot`	Unclear	—
ApifyWebsiteContentCrawler	Unknown	Data reseller	`apifywebsitecontentcrawler`	Unclear	—
Aranet-SearchBot	Unknown	Undocumented	`aranet-searchbot`	Unclear	—
atlassian-bot	Atlassian	Search index	`atlassian-bot`	Respects robots.txt	—
Awario	Awario	Data reseller	`awario`	Unclear	—
AwarioSmartBot	Awario	Model training	`awario`	Unclear	—
AzureAI-SearchBot	Unknown	Search index	`azureai-searchbot`	Unclear	—
bedrockbot	Amazon	User assistant	`bedrockbot`	Respects robots.txt	—
Big Sur AI	Big Sur AI	Model training	`bigsur.ai`	Unclear	—
bigsur.ai	Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi…	User assistant	`bigsur.ai`	Unclear	—
BorderxBot	Borderxlab (Nubestore AI)	Model training	`borderxbot`	Unclear	—
Brandwatch	Brandwatch	Model training	`magpie-crawler`	Unclear	—
Bravebot	Brave	Search index	`bravebot`	Respects robots.txt	—
Brightbot	Unknown	Data reseller	`brightbot`	Unclear	—
Brightbot 1.0	https://brightdata.com/brightbot	Model training	`brightbot 1.0`	Unclear	—
Browserbase	Browserbase	User assistant	`browserbase`	Unclear	—
BuddyBot	BuddyBotLearning	Undocumented	`buddybot`	Unclear	—
ccBot crawler	Unknown	Data reseller	`ccbot crawler`	Unclear	—
Channel3Bot	Channel3	Search index	`channel3bot`	Unclear	—
ChatGLM-Spider	Unknown	Data reseller	`chatglm-spider`	Unclear	—
ChatGPT-Browser	Unknown	Data reseller	`chatgpt-browser`	Unclear	—
Chathive crawler	Inteso Group BV	User assistant	`chathivecrawler`	Unclear	—
CitibotSiteCrawler	Citibot	Model training	`citibotsitecrawler`	Unclear	—
Claude	Anthropic	Model training	`claudebot`	Unclear	—
Claude-Code	Anthropic	Browsing agent	`claude-code`	Unclear	—
Claude-Web	Anthropic	User assistant	`claude-web`	Unclear	173,801
Cledara SaaS Management Agent	Cledara	User assistant	`cledarabot`	Unclear	—
Cloudflare Browser Run	Cloudflare	User assistant	`cloudflare browser run`	Unclear	—
Cloudflare Crawler	Cloudflare	Model training	`cloudflarebrowserrenderingcrawler/1.0`	Unclear	—
Cloudflare-AutoRAG	Cloudflare	Search index	`cloudflare-autorag`	Respects robots.txt	—
CloudVertexBot	Unknown	Data reseller	`cloudvertexbot`	Unclear	—
Code	Unknown	Browsing agent	`code`	Unclear	—
cohere-ai	Cohere	User assistant	`cohere ai`	Unclear	174,990
Cohere-Command	Unknown	Data reseller	`cohere-command`	Unclear	—
cohere-training-data-crawler	Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products	Data reseller	`cohere-training-data-crawler`	Unclear	—
Cotoyogi	ROIS	Model training	`cotoyogi`	Respects robots.txt	—
CragCrawler	CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services	Data reseller	`cragcrawler`	Unclear	—
Crawl4AI	Unknown	Undocumented	`crawl4ai`	Unclear	—
Crawlspace	Crawlspace	Data reseller	`crawlspace`	Respects robots.txt	—
Cursor	Cursor	Browsing agent	`cursor`	Unclear	—
Daric2	Cloudflare	Model training	`daric2`	Unclear	—
Daric3	Cloudflare	Model training	`daric3`	Unclear	—
Daric4	Cloudflare	Model training	`daric4`	Unclear	—
Datenbank Crawler	Datenbank	Data reseller	`datenbank crawler`	Unclear	—
DeepSeekBot	DeepSeek	Model training	`deepseekbot`	Ignores robots.txt	—
Devin	Devin AI	Browsing agent	`devin`	Respects robots.txt	—
Diffbot	Diffbot	Data reseller	`diffbot`	Unclear	145,203
Direqt Anomura	Direqt	Answer engine	`anomura`	Unclear	—
Doximity-Diffbot	Unknown	Data reseller	`doximity-diffbot`	Unclear	—
DuckAssistBot	DuckDuckGo	User assistant	`duckassistbot`	Unclear	—
EasyScan	codire GmbH	User assistant	`easyscan`	Unclear	—
Echobot Bot	Echobox	Data reseller	`echobot bot`	Unclear	—
EchoboxBot	Echobox	Data reseller	`echoboxbot`	Unclear	—
Element451Bot	Spark451, Inc. (Element451)	Answer engine	`element451bot`	Respects robots.txt	—
ExaBot	Unknown	Data reseller	`exabot`	Unclear	—
FacebookBot	Meta	Model training	`facebookbot`	Respects robots.txt	225,676
facebookexternalhit	Meta	User assistant	`facebookexternalhit`	Ignores robots.txt	—
Factset_spyderbot	Factset	Model training	`factset_spyderbot`	Unclear	—
FirecrawlAgent	Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications	Data reseller	`firecrawlagent`	Respects robots.txt	—
FirmlyAI Bot	Firmly	User assistant	`firmlyai bot`	Unclear	—
FishBot	FishBot	Model training	`fishbot`	Unclear	—
FriendlyCrawler	Unknown	Model training	`friendlycrawler`	Respects robots.txt	—
GeistHaus-PageFetcher	GeistHaus, a company developing AI systems for therapy and psychological assessment	User assistant	`geisthaus-pagefetcher`	Unclear	—
Gemini-AI	Unknown	Data reseller	`gemini-ai`	Unclear	—
Generic Bot	Unknown	Data reseller	`generic bot`	Unclear	—
Google-Agent	Google	Browsing agent	`google-agent`	Unclear	—
Google-CloudVertexBot	Google	Browsing agent	`cloudvertexbot`	Respects robots.txt	—
Google-Firebase	Google	User assistant	`google-firebase`	Unclear	—
Google-Gemini-CLI	Unknown	Browsing agent	`google-gemini-cli`	Unclear	—
Google-NotebookLM	Google	User assistant	`google-notebooklm`	Unclear	—
GoogleAgent-Mariner	Google	Browsing agent	`googleagent-mariner`	Unclear	—
GoogleAgent-URLContext	Google that retrieves web content on behalf of Gemini API users	User assistant	`googleagent-urlcontext`	Unclear	—
GoogleOther-Image	Google	Search index	`googleother-image`	Respects robots.txt	—
GoogleOther-Video	Google	Search index	`googleother-video`	Respects robots.txt	—
HenkBot	Unknown	Data reseller	`henkbot`	Unclear	—
Henry Shopping Agent	Henry Labs	User assistant	`henry shopping agent`	Unclear	—
HIFIBot	HIFI	User assistant	`hifibot`	Unclear	—
iAskBot	Unknown	Undocumented	`iaskbot`	Unclear	—
iaskspider	Unknown	Undocumented	`iaskspider`	Unclear	—
iaskspider/2.0	iAsk	User assistant	`iaskspider/2.0`	Ignores robots.txt	—
IbouBot	Ibou	Search index	`iboubot`	Respects robots.txt	—
ICC-Crawler	NICT	Model training	`icc-crawler`	Respects robots.txt	—
ImageSift	Unknown	Data reseller	`imagesift`	Unclear	—
ImagesiftBot	ImageSift	Search index	`imagesiftbot`	Respects robots.txt	130,584
imageSpider	Unknown	Data reseller	`imagespider`	Unclear	—
img2dataset	img2dataset	Model training	`img2dataset`	Unclear	—
Instapaper	Instant Paper	User assistant	`instapaper`	Unclear	—
ISSCyberRiskCrawler	ISS-Corporate	Model training	`isscyberriskcrawler`	Ignores robots.txt	—
kagi-fetcher	Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…	User assistant	`kagi-fetcher`	Unclear	—
Kagibot	Unknown	Search index	`kagibot`	Unclear	—
Kangaroo Bot	Unknown	Data reseller	`kangaroo bot`	Unclear	—
Kernel	Kernel	User assistant	`kernel`	Unclear	—
Kimi-User	Moonshot AI that fetches web content on behalf of users interacting with Kimi	User assistant	`kimi-user`	Unclear	—
KimiBot	Moonshot AI	Model training	`kimibot`	Unclear	—
KlaviyoAIBot	Klaviyo	User assistant	`klaviyoaibot`	Respects robots.txt	—
KunatoCrawler	Unknown	Undocumented	`kunatocrawler`	Unclear	—
laion-huggingface-processor	Unknown	Data reseller	`laion-huggingface-processor`	Unclear	—
LAIONDownloader	Large-scale Artificial Intelligence Open Network	Model training	`laiondownloader`	Ignores robots.txt	—
LCC	Unknown	Data reseller	`lcc`	Unclear	—
LINER Bot	Liner Bot	Model training	`linerbot`	Unclear	—
LinerBot	Unknown	User assistant	`linerbot`	Unclear	—
Linguee Bot	Linguee	Model training	`linguee bot`	Ignores robots.txt	—
LinkupBot	Unknown	Search index	`linkupbot`	Unclear	—
Make.com	Make.com	Model training	`make.com`	Unclear	—
Manus Bot	Manus	User assistant	`manus bot`	Unclear	—
Manus-User	Butterfly Effect, a company based in China	Browsing agent	`manus-user`	Unclear	—
Meta-ExternalAds	Unknown	Data reseller	`meta-externalads`	Unclear	—
meta-externalfetcher	Meta	User assistant	`meta-externalfetcher`	Unclear	—
meta-webindexer	Meta	User assistant	`meta-webindexer`	Unclear	—
MistralAI-User/1.0	Mistral AI	User assistant	`mistralai-user/1.0`	Respects robots.txt	—
Mozilla-Tabstack	Mozilla	Data reseller	`mozilla-tabstack`	Respects robots.txt	—
MyCentralAIScraperBot	Unknown	Data reseller	`mycentralaiscraperbot`	Unclear	—
NagetBot	Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts)	User assistant	`nagetbot`	Unclear	—
Nava Labs ASP (Dev)	Nava Labs	User assistant	`nava/`	Unclear	—
Navu	HivePoint, Inc.	Model training	`navubot`	Unclear	—
netEstate Imprint Crawler	netEstate	Data reseller	`netestate imprint crawler`	Unclear	—
newsai	Unknown	User assistant	`newsai`	Unclear	—
NotebookLM	Unknown	User assistant	`notebooklm`	Unclear	—
NovaAct	Unknown	Browsing agent	`novaact`	Unclear	—
Novellum AI Crawl	Novellum	Model training	`novellum`	Unclear	—
omgili	Webz.io	Model training	`omgili`	Respects robots.txt	—
omgilibot	Webz.io	Search index	`omgilibot`	Respects robots.txt	176,611
OpenAI	OpenAI	Undocumented	`openai`	Respects robots.txt	—
opencode	Unknown	Browsing agent	`opencode`	Unclear	—
PanguBot	the Chinese company Huawei	Data reseller	`pangubot`	Unclear	—
Panscient	Panscient	Model training	`panscient`	Respects robots.txt	—
panscient.com	Panscient	Model training	`panscient.com`	Respects robots.txt	—
Payhawk Invoice Fetching Agent	Payhawk	User assistant	`payhawk invoice fetching agent`	Unclear	—
payroll-bot	ADP	Model training	`adpresearchbot/`	Unclear	—
PetalBot	Huawei	Search index	`petalbot`	Respects robots.txt	—
PhindBot	phind	User assistant	`phindbot`	Unclear	—
Poggio-Citations	Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan…	User assistant	`poggio-citations`	Unclear	—
Poseidon Research Crawler	Poseidon Research	Search index	`poseidon research crawler`	Unclear	—
QAtechBot	QA.tech	User assistant	`qatechbot`	Unclear	—
QualifiedBot	Qualified	User assistant	`qualifiedbot`	Unclear	—
Querit-SearchBot	Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg…	Data reseller	`querit-searchbot`	Unclear	—
QueritBot	Querit, a company providing a search API for large language model integration	Data reseller	`queritbot`	Unclear	—
QuillBot	Quillbot	Undocumented	`quillbot`	Unclear	—
quillbot.com	Quillbot	Undocumented	`quillbot.com`	Unclear	—
Retool	Retool	User assistant	`retool/`	Unclear	—
RyeBot	Rye	User assistant	`ryebot`	Unclear	—
SBIntuitionsBot	SB Intuitions	Data reseller	`sbintuitionsbot`	Respects robots.txt	—
Scrapy	Zyte	Model training	`scrapy`	Unclear	—
Selectika AI	Selectika	Model training	`selectikascraper`	Unclear	—
SemrushBot-OCOB	Semrush	Data reseller	`semrushbot-ocob`	Respects robots.txt	—
SemrushBot-SWA	Semrush	User assistant	`semrushbot-swa`	Respects robots.txt	—
SemrushBotSwa	Semrush	Model training	`semrushbot-swa/`	Unclear	—
Shap-User	Unknown	User assistant	`shap-user`	Unclear	—
ShapBot	Parallel	Data reseller	`shapbot`	Respects robots.txt	—
Sidetrade indexer bot	Sidetrade	Model training	`sidetrade indexer bot`	Unclear	—
Spider	Unknown	Data reseller	`spider`	Unclear	—
Strivve Automation	Strivve	User assistant	`strivve automation`	Unclear	—
TavilyBot	Unknown	Data reseller	`tavilybot`	Unclear	—
Terra Cotta	Unknown	Data reseller	`terra cotta`	Unclear	—
TerraCotta	Ceramic AI	Data reseller	`terracotta`	Respects robots.txt	—
Thinkbot	Thinkbot	Data reseller	`thinkbot`	Ignores robots.txt	—
TikTokSpider	ByteDance	Model training	`tiktokspider`	Unclear	—
Timpibot	Timpi	Model training	`timpibot`	Unclear	106,901
TongyiBot	Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers	User assistant	`tongyibot`	Unclear	—
Trae	Unknown	Browsing agent	`trae`	Unclear	—
TwinAgent	Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…	Browsing agent	`twinagent`	Unclear	—
UseAI	Unknown	User assistant	`useai`	Unclear	—
VelenPublicWebCrawler	Velen Crawler	Model training	`velenpublicwebcrawler`	Respects robots.txt	—
Visually.io Shopify Editor	Visually.io	User assistant	`visually.io shopify editor`	Unclear	—
WARDBot	WEBSPARK	Data reseller	`wardbot`	Unclear	—
webzio	Unknown	Data reseller	`webzio`	Unclear	—
Webzio-Extended	Unknown	Data reseller	`webzio-extended`	Unclear	—
wpbot	QuantumCloud	Data reseller	`wpbot`	Unclear	—
WRTNBot	Unknown	Undocumented	`wrtnbot`	Unclear	—
Yahoo Gemini	Unknown	Data reseller	`yahoo gemini`	Unclear	—
YaK	Meltwater	Search index	`yak`	Unclear	—
YandexAdditional	Yandex	Model training	`yandexadditional`	Respects robots.txt	—
YandexAdditionalBot	Yandex	Model training	`yandexadditionalbot`	Respects robots.txt	—
YGS Group Falconer Scraper	Unknown	Model training	`ygs-scraper-bot`	Unclear	—
YiyanBot	Baidu that fetches web content for the yiyan	User assistant	`yiyanbot`	Unclear	—
YouBot	You.com	Answer engine	`youbot`	Respects robots.txt	136,659
ZanistaBot	Unknown	Search index	`zanistabot`	Unclear	—
$1	Unknown	Search index	`$1`	Unclear	—
1001FirmsBot	Unknown	Data reseller	`1001firmsbot`	Unclear	—
2ip	Unknown	Data reseller	`2ip`	Unclear	—
Abonti	Unknown	Data reseller	`abonti`	Unclear	—
Aboundexbot	Unknown	Search index	`aboundexbot`	Unclear	—
AccessibleWebBot	Unknown	Data reseller	`accessiblewebbot`	Unclear	—
AccompanyBot	Unknown	Data reseller	`accompanybot`	Unclear	—
Acoon	Unknown	Search index	`acoon`	Unclear	—
AdAuth	Unknown	Data reseller	`adauth`	Unclear	—
Adbeat	Unknown	Data reseller	`adbeat`	Unclear	—
AddThis	AddThis	Data reseller	`addthis`	Unclear	—
Adsbot	Unknown	Data reseller	`adsbot`	Unclear	—
Adscanner	Unknown	Data reseller	`adscanner`	Unclear	—
AdsTxtCrawler	Unknown	Data reseller	`adstxtcrawler`	Unclear	—
adstxtlab.com	Unknown	Data reseller	`adstxtlab.com`	Unclear	—
Aegis	Unknown	Data reseller	`aegis`	Unclear	—
AffiliateSpyBot	Unknown	Data reseller	`affiliatespybot`	Unclear	—
Agency Analytics Crawler	Agency Analytics	Data reseller	`aa/1.0`	Unclear	—
aHrefs Bot	Unknown	Data reseller	`ahrefs bot`	Unclear	—
Ahrefs Site Audit	Ahrefs	Data reseller	`ahrefssiteaudit`	Unclear	—
AhrefsBot	Ahrefs	Data reseller	`ahrefsbot`	Unclear	—
AirsoftdbBot	Airsoftdb AB	Data reseller	`airsoftdbbot`	Respects robots.txt	—
Alexa Crawler	Unknown	Search index	`alexa crawler`	Unclear	—
Algolia	Algolia	Search index	`algolia crawler`	Unclear	—
All Africa Crawler	AllAfrica Global Media	Search index	`allafrica`	Unclear	—
Alli AI Bot	Alli AI	Data reseller	`alliai/1.0`	Unclear	—
Allloadin Favicon Bot	Unknown	Data reseller	`allloadin favicon bot`	Unclear	—
AlltheWeb	Unknown	Data reseller	`alltheweb`	Unclear	—
AlphaXCrawl	Unknown	Data reseller	`alphaxcrawl`	Unclear	—
Amazon AdBot	Unknown	Data reseller	`amazon adbot`	Unclear	—
Amorank Spider	Unknown	Data reseller	`amorank spider`	Unclear	—
Analytics SEO Crawler	Unknown	Data reseller	`analytics seo crawler`	Unclear	—
Ant	Unknown	Data reseller	`ant`	Unclear	—
AppcastBot	Appcast	Data reseller	`appcastparsingbot`	Unclear	—
archive.org bot	Unknown	Data reseller	`archive.org bot`	Unclear	—
ArchiveBot	Unknown	Data reseller	`archivebot`	Unclear	—
ArchiveBox	Unknown	Data reseller	`archivebox`	Unclear	—
Arena Bot	Arena.im	User assistant	`arenabot`	Unclear	—
Arquivo Web Crawler	Arquivo	Data reseller	`arquivo-web-crawler`	Respects robots.txt	—
Arquivo.pt	Unknown	Data reseller	`arquivo.pt`	Unclear	—
ARSNova Filter System	Unknown	Data reseller	`arsnova filter system`	Unclear	—
Artemis Web Reader	capjamesg	Data reseller	`artemis web reader/`	Unclear	—
Artsdata Crawler	Culture Creates	Data reseller	`artsdata-crawler`	Respects robots.txt	—
Asana	Unknown	Data reseller	`asana`	Unclear	—
Ask Jeeves	Unknown	Search index	`ask jeeves`	Unclear	—
AspiegelBot	Unknown	Data reseller	`aspiegelbot`	Unclear	—
Attracta	Attracta	Data reseller	`attracta`	Unclear	—
Authory	Authory	Data reseller	`authory/`	Unclear	—
Automattic Analytics	Unknown	Data reseller	`automattic analytics`	Unclear	—
Backlink-Check.de	Unknown	Data reseller	`backlink-check.de`	Unclear	—
BacklinkCrawler	Unknown	Data reseller	`backlinkcrawler`	Unclear	—
BacklinksExtendedBot	Unknown	Data reseller	`backlinksextendedbot`	Unclear	—
BackupLand	Unknown	Data reseller	`backupland`	Unclear	—
Baidu ADS Server Proxy	Baidu	Search index	`baidu-ads`	Unclear	—
Baidu Spider	Unknown	Search index	`baidu spider`	Unclear	—
Baiduspider	Baidu	Search index	`baiduspider`	Unclear	—
Barkrowler	Babbar	Data reseller	`barkrowler`	Respects robots.txt	—
Bazqux	Bazqux	Data reseller	`bazqux`	Unclear	—
BazQux Reader	Unknown	Data reseller	`bazqux reader`	Unclear	—
BDCbot	Unknown	Data reseller	`bdcbot`	Unclear	—
BDFetch	Unknown	Data reseller	`bdfetch`	Unclear	—
BestChange Bot	BestChange	Data reseller	`bestchange`	Unclear	—
Bibliotheque Nacional de France Crawler	Bibliothèque nationale de France	Data reseller	`bnf.fr_bot`	Unclear	—
Bing Preview	Microsoft	User assistant	`bingpreview`	Unclear	—
BingBot	Microsoft	Search index	`bingbot`	Respects robots.txt	—
Birdcrawlerbot	Unknown	Data reseller	`birdcrawlerbot`	Unclear	—
BitlyBot	Unknown	Data reseller	`bitlybot`	Unclear	—
Blackboard Ally	Unknown	Data reseller	`blackboard ally`	Unclear	—
Blekkobot	Unknown	Search index	`blekkobot`	Unclear	—
BLEXBot	WebMeUp	Data reseller	`blexbot`	Unclear	—
BLEXBot Crawler	Unknown	Data reseller	`blexbot crawler`	Unclear	—
Bloglines	Unknown	Data reseller	`bloglines`	Unclear	—
Bloglovin	Unknown	Data reseller	`bloglovin`	Unclear	—
Blogtrottr	Blogtrottr	Data reseller	`blogtrottr`	Unclear	—
Bluesky Link Preview Service	Bluesky	User assistant	`bluesky/`	Unclear	—
BoardGamePrices Bot	KP Software Consult ApS	Data reseller	`boardgameprices bot`	Unclear	—
BoardReader	Unknown	Search index	`boardreader`	Unclear	—
BoardReader Blog Indexer	Unknown	Data reseller	`boardreader blog indexer`	Unclear	—
Botify	Botify	Data reseller	`botify`	Unclear	—
Bountii Bot	Unknown	Search index	`bountii bot`	Unclear	—
BrandVerity	Unknown	Data reseller	`brandverity`	Unclear	—
BrightEdge	Unknown	Data reseller	`brightedge`	Unclear	—
BUbiNG	Unknown	Data reseller	`bubing`	Unclear	—
Buck	Unknown	Search index	`buck`	Unclear	—
Buffer Link Preview Bot	Buffer	User assistant	`bufferlinkpreviewbot`	Unclear	—
BuiltWith	Unknown	Data reseller	`builtwith`	Unclear	—
Butterfly Robot	Unknown	Search index	`butterfly robot`	Unclear	—
Buttondown RSS-Feed-Parser	Buttondown	Data reseller	`buttondown`	Unclear	—
CaliberBot	Conductor	Data reseller	`caliperbot/1.0`	Unclear	—
Capital One Bot	Capital One	Data reseller	`capitalonebot`	Unclear	—
CareerBot	Unknown	Data reseller	`careerbot`	Unclear	—
Castopod	Unknown	Data reseller	`castopod`	Unclear	—
CATExplorador	Unknown	Search index	`catexplorador`	Unclear	—
Charlotte	Unknown	Data reseller	`charlotte`	Unclear	—
CheckMark Network	Unknown	Data reseller	`checkmark network`	Unclear	—
Choosito	Unknown	Data reseller	`choosito`	Unclear	—
Chrome-Lighthouse	Google	Data reseller	`chrome-lighthouse`	Unclear	—
Cincraw	Unknown	Data reseller	`cincraw`	Unclear	—
CISPA Web Analyzer	Unknown	Data reseller	`cispa web analyzer`	Unclear	—
CLASSLA-web	Unknown	Data reseller	`classla-web`	Unclear	—
ClearscopeBot	Mushi Labs	Data reseller	`clearscopebot/`	Respects robots.txt	—
Clickagy	Unknown	Data reseller	`clickagy`	Unclear	—
Cliqzbot	Unknown	Data reseller	`cliqzbot`	Unclear	—
CloudFlare AMP Fetcher	Unknown	Data reseller	`cloudflare amp fetcher`	Unclear	—
Cloudflare Custom Error Page Crawler	Cloudflare	Data reseller	`collapsify/`	Unclear	—
Cloudflare Custom Hostname Verification	Unknown	Data reseller	`cloudflare custom hostname verification`	Unclear	—
Cloudflare Digicert DCV	Cloudflare	Data reseller	`digicert dcv`	Unclear	—
Cloudflare Healthchecks	Cloudflare	Data reseller	`cloudflare-healthchecks`	Unclear	—
Cloudflare Prefetch	Cloudflare	Data reseller	`cloudflare-prefetch`	Unclear	—
Cloudflare Purge	Cloudflare	Data reseller	`cloudflare-purge/`	Unclear	—
Cloudflare SpeedTest	Cloudflare	Data reseller	`cloudflare speedtest`	Unclear	—
Cloudflare SSLDetector	Cloudflare	Data reseller	`cloudflare-ssldetector`	Unclear	—
Cloudflare Stream Webhook	Cloudflare	Data reseller	`cloudflare stream webhook`	Unclear	—
Cloudflare Web Scanner	Cloudflare	Search index	`cloudflarewebscanner`	Respects robots.txt	—
Cloudflare-Traffic-Manager	Cloudflare	Data reseller	`cloudflare-traffic-manager`	Unclear	—
CloudServerMarketSpider	Unknown	Data reseller	`cloudservermarketspider`	Unclear	—
CMU-Cylab	Carnegie Mellon University	Data reseller	`cmu-cylab`	Unclear	—
Cocolyzebot	Unknown	Data reseller	`cocolyzebot`	Unclear	—
cognitiveSEO Crawler	cognitive SEO Internet Marketing Tools.	Data reseller	`james bot`	Unclear	—
colly	Unknown	Data reseller	`colly`	Unclear	—
CommaFeed	Unknown	Data reseller	`commafeed`	Unclear	—
Comscore	Unknown	Data reseller	`comscore`	Unclear	—
ConvermaxBot	Convermax Corp.	Data reseller	`convermax/`	Unclear	—
Cookiebot	Unknown	Data reseller	`cookiebot`	Unclear	—
CoostiBot	Coosti ApS	Data reseller	`coostibot`	Respects robots.txt	—
CopyvioDetector	Unknown	Data reseller	`copyviodetector`	Unclear	—
Coveo Bot	Coveo	Search index	`coveobot`	Unclear	—
Crawldad	Unknown	Data reseller	`crawldad`	Unclear	—
Crawlson	Crawlson	Search index	`crawlson`	Respects robots.txt	—
Crawly Project	Unknown	Data reseller	`crawly project`	Unclear	—
creobot	creoline GmbH	User assistant	`creobot`	Unclear	—
CriteoBot	Unknown	Data reseller	`criteobot`	Unclear	—
CrowdTangle	Unknown	Data reseller	`crowdtangle`	Unclear	—
CrystalSemanticsBot	Unknown	Data reseller	`crystalsemanticsbot`	Unclear	—
CSS Certificate Spider	Unknown	Data reseller	`css certificate spider`	Unclear	—
CuteStat	CuteStat	Search index	`cutestat`	Respects robots.txt	—
Cxense	Cxense	Data reseller	`cxensebot/`	Unclear	—
CyberFind Crawler	Unknown	Data reseller	`cyberfind crawler`	Unclear	—
Cốc Cốc	Coccoc	Search index	`coccocbot`	Unclear	—
Cốc Cốc Bot	Unknown	Search index	`cốc cốc bot`	Unclear	—
DaspeedBot	Unknown	Data reseller	`daspeedbot`	Unclear	—
DataForSEO	DataForSEO	Data reseller	`rsiteauditor`	Unclear	—
DataForSEO Bot	DataForSEO	Data reseller	`dataforseobot`	Unclear	—
DataForSeoBot	Unknown	Data reseller	`dataforseobot`	Unclear	—
datagnionbot	Unknown	Data reseller	`datagnionbot`	Unclear	—
Datanyze	Unknown	Data reseller	`datanyze`	Unclear	—
Dataprovider	Unknown	Data reseller	`dataprovider`	Unclear	—
Dataprovider.com	Dataprovider.com	Search index	`dataprovider`	Unclear	—
Daum	Daum	Search index	`daum`	Unclear	—
Dazoobot	Unknown	Search index	`dazoobot`	Unclear	—
Deep SEARCH 9	Unknown	Data reseller	`deep search 9`	Unclear	—
Deepfield Genome	Unknown	Data reseller	`deepfield genome`	Unclear	—
deepnoc	Unknown	Data reseller	`deepnoc`	Unclear	—
DepSpid	Unknown	Data reseller	`depspid`	Unclear	—
DigiCert DCV	DigiCert	Data reseller	`digicert dcv bot`	Unclear	—
Discobot	Unknown	Search index	`discobot`	Unclear	—
discordBot	Discord, Inc.	User assistant	`discordbot/`	Unclear	—
Disqus	Unknown	Data reseller	`disqus`	Unclear	—
dlvr.it Feed Fetcher	dlvr.it	Data reseller	`dlvr.it/1.0`	Respects robots.txt	—
Domain Codex	Unknown	Data reseller	`domain codex`	Unclear	—
Domain Re-Animator Bot	Unknown	Data reseller	`domain re-animator bot`	Unclear	—
Domain Research Project	Unknown	Data reseller	`domain research project`	Unclear	—
DomainAppender	Unknown	Data reseller	`domainappender`	Unclear	—
DomainCrawler	Unknown	Data reseller	`domaincrawler`	Unclear	—
Domains Project	Unknown	Data reseller	`domains project`	Unclear	—
DomainStatsBot	Unknown	Data reseller	`domainstatsbot`	Unclear	—
DomCop Bot	Unknown	Data reseller	`domcop bot`	Unclear	—
DotBot	Unknown	Data reseller	`dotbot`	Unclear	—
Dubbotbot	Unknown	Data reseller	`dubbotbot`	Unclear	—
DuckDuckBot	DuckDuckGo	Search index	`duckduckbot`	Unclear	—
DuplexWeb-Google	Unknown	Data reseller	`duplexweb-google`	Unclear	—
Easou Spider	Unknown	Search index	`easou spider`	Unclear	—
eCairn-Grabber	Unknown	Data reseller	`ecairn-grabber`	Unclear	—
EchoFeed	Robb Knight	Data reseller	`echofeed`	Unclear	—
EFF Do Not Track Policy Checker	Unknown	Data reseller	`eff do not track policy checker`	Unclear	—
EFF Do Not Track Verifier	Unknown	Data reseller	`eff do not track verifier`	Unclear	—
EMail Exractor	Unknown	Data reseller	`email exractor`	Unclear	—
Embedly	Embedly	User assistant	`embedly`	Unclear	—
eMoney Advisor	eMoney Advisor	Data reseller	`emoneybot/`	Unclear	—
Entfer	Unknown	Data reseller	`entfer`	Unclear	—
Epivoz Crawler	Techmeme	Data reseller	`epivozcrawler/`	Unclear	—
evc-batch	Unknown	Data reseller	`evc-batch`	Unclear	—
Everyfeed	Unknown	Data reseller	`everyfeed`	Unclear	—
ExactSeek Crawler	Unknown	Search index	`exactseek crawler`	Unclear	—
Example3	Unknown	Data reseller	`example3`	Unclear	—
Eyeotabot	Unknown	Data reseller	`eyeotabot`	Unclear	—
eZ Publish Link Validator	Unknown	Data reseller	`ez publish link validator`	Unclear	—
EzoicBot	Ezoic Inc	Data reseller	`http://www.ezoic.com/bot.html`	Unclear	—
Ezooms	Unknown	Data reseller	`ezooms`	Unclear	—
Fastmail Bot	Fastmail	User assistant	`fastmailua`	Unclear	—
Faveeo	Unknown	Data reseller	`faveeo`	Unclear	—
FedReporter Bot for FFIEC	Fed Reporter, Inc.	Data reseller	`fedreporterdatabot`	Unclear	—
Feed Wrangler	Unknown	Data reseller	`feed wrangler`	Unclear	—
Feedbin	Feedbin	Data reseller	`feedbin`	Unclear	—
FeedBlitz	FeedBlitz LLC	Data reseller	`feedblitz`	Unclear	—
FeedBurner	Unknown	Data reseller	`feedburner`	Unclear	—
Feeder	Really Simple AB	Data reseller	`feeder.co`	Respects robots.txt	—
Feedly	Feedly	Data reseller	`feedly`	Unclear	—
FeedOtter	FeedOtter	Data reseller	`feedotter`	Unclear	—
Feedspot	Unknown	Data reseller	`feedspot`	Unclear	—
FeedWind Crawler	Mikle	Data reseller	`feedwind/`	Unclear	—
Femtosearch	Unknown	Data reseller	`femtosearch`	Unclear	—
Fever	Unknown	Data reseller	`fever`	Unclear	—
FindFiles.net	Unknown	Data reseller	`findfiles.net`	Unclear	—
Findxbot	Unknown	Data reseller	`findxbot`	Unclear	—
FIT Feeds Fetcher	follow.it	Data reseller	`follow.it`	Unclear	—
Flipboard	Unknown	Data reseller	`flipboard`	Unclear	—
FlipboardProxy	Flipboard	User assistant	`flipboardproxy`	Unclear	—
FlipboardRSS	Flipboard	Data reseller	`flipboardrss`	Unclear	—
FlyingPress	FlyingWeb LLC	Data reseller	`flyingpress`	Unclear	—
FN Legislative and Regulatory Bot	FiscalNote Holdings, Inc.	Data reseller	`fiscalnote`	Unclear	—
FontRadar	Unknown	Data reseller	`fontradar`	Unclear	—
fragFINN	Unknown	Data reseller	`fragfinn`	Unclear	—
Freespoke	Freespoke	Search index	`crawler.freespoke.com`	Unclear	—
FreshRSS	Unknown	Data reseller	`freshrss`	Unclear	—
FullFact Ingest	FullFact	Data reseller	`fullfact`	Unclear	—
Functionize	Unknown	Data reseller	`functionize`	Unclear	—
Funnelback	Squiz - FunnelBack	Search index	`funnelback`	Unclear	—
Gaisbot	Unknown	Data reseller	`gaisbot`	Unclear	—
GDNP	Unknown	Data reseller	`gdnp`	Unclear	—
GeedoBot	Unknown	Data reseller	`geedobot`	Unclear	—
GeedoShopProductFinder	Geedo SIA	Data reseller	`geedoshopproductfinder`	Respects robots.txt	—
GhostExplore	Ghost	Data reseller	`ghostexplore`	Unclear	—
Gigablast	Unknown	Search index	`gigablast`	Unclear	—
Gigabot	Gigablast	Search index	`gigabot`	Unclear	—
Github Camo	Unknown	Data reseller	`github camo`	Unclear	—
Gluten Free Crawler	Unknown	Data reseller	`gluten free crawler`	Unclear	—
Gmail Image Proxy	Unknown	Data reseller	`gmail image proxy`	Unclear	—
Golfe	Unknown	Data reseller	`golfe`	Unclear	—
Goo	Unknown	Search index	`goo`	Unclear	—
Google Area 120 Privacy Policy Fetcher	Unknown	Data reseller	`google area 120 privacy policy fetcher`	Unclear	—
Google Calendar Importer	Google	Data reseller	`google-calendar-importer`	Unclear	—
Google Cloud Scheduler	Unknown	Data reseller	`google cloud scheduler`	Unclear	—
Google Digital Asset Links	Google	Data reseller	`googleassociationservice`	Unclear	—
Google Favicon	Unknown	Data reseller	`google favicon`	Unclear	—
Google Feed Fetcher	Google	Data reseller	`google.com/feedfetcher`	Unclear	—
Google Image Proxy	Google	User assistant	`googleimageproxy`	Unclear	—
Google Images	Google	Search index	`googlebot-image`	Unclear	—
Google Page Renderer	Google	User assistant	`google-pagerenderer`	Unclear	—
Google Publisher Center	Google	Data reseller	`googleproducer`	Unclear	—
Google Scholar	Google	Search index	`googlebot-ia/`	Unclear	—
Google Search Console	Unknown	Data reseller	`google search console`	Unclear	—
Google StoreBot	Google	Data reseller	`google storebot`	Unclear	—
Google Videos	Google	Search index	`googlebot-video`	Unclear	—
Google Web Snippet	Google	User assistant	`google-pagerenderer`	Unclear	—
Google-AdWords-Express	Google	Data reseller	`google-adwords-express`	Unclear	—
Google-Adwords-Instant	Google	Data reseller	`google-adwords-instant`	Unclear	—
Google-Flights-Search-Spackle	Google	Search index	`google-flights-search-spackle`	Unclear	—
Google-ilp	Unknown	Data reseller	`google-ilp`	Unclear	—
Google-Read-Aloud	Unknown	Data reseller	`google-read-aloud`	Unclear	—
Google-Safety	Unknown	Data reseller	`google-safety`	Unclear	—
google-xrawler	Google	Search index	`google-xrawler`	Unclear	—
GoogleBot	Google	Search index	`googlebot`	Unclear	—
Googlebot News	Unknown	Search index	`googlebot news`	Unclear	—
Gowikibot	Unknown	Data reseller	`gowikibot`	Unclear	—
Gozle	Unknown	Data reseller	`gozle`	Unclear	—
Grapeshot	Unknown	Data reseller	`grapeshot`	Unclear	—
Gregarius	Unknown	Data reseller	`gregarius`	Unclear	—
GTmetrix	Unknown	Data reseller	`gtmetrix`	Unclear	—
HaloBot	Unknown	Data reseller	`halobot`	Unclear	—
Hatena	Hatena	Data reseller	`hatena`	Respects robots.txt	—
Hatena Bookmark	Unknown	Data reseller	`hatena bookmark`	Unclear	—
Hatena Favicon	Unknown	Data reseller	`hatena favicon`	Unclear	—
Headline	Unknown	Data reseller	`headline`	Unclear	—
Heexybot	Unknown	Search index	`heexybot`	Unclear	—
HelloWork	HelloWork Group	Data reseller	`helloworkjobpostingbot`	Unclear	—
Heritrix	Unknown	Data reseller	`heritrix`	Unclear	—
Heureka Feed	Unknown	Data reseller	`heureka feed`	Unclear	—
HEY Email Privacy Proxy	37signals	User assistant	`hey.com/imageproxy`	Unclear	—
htmlyse	Unknown	Data reseller	`htmlyse`	Unclear	—
httpx	Unknown	Data reseller	`httpx`	Unclear	—
HuaweiWebCatBot	Unknown	Data reseller	`huaweiwebcatbot`	Unclear	—
HubPages	Unknown	Data reseller	`hubpages`	Unclear	—
HubSpot	Unknown	Data reseller	`hubspot`	Unclear	—
HubSpot Feed Fetcher	HubSpot	Data reseller	`hubspot-feedfetcher`	Unclear	—
HubSpot Page Fetcher	HubSpot	User assistant	`hubspot page fetcher`	Unclear	—
HubSpotContentSearchBot	Unknown	Data reseller	`hubspotcontentsearchbot`	Unclear	—
Huckabuy Bot	Huckabuy	Data reseller	`huckabot/`	Unclear	—
Hype Machine	Hype Machine	Search index	`hype machine`	Unclear	—
HypeStat	Unknown	Data reseller	`hypestat`	Unclear	—
i-search-crawler	Scala Communications, Inc.	Search index	`i-search-crawler`	Respects robots.txt	—
Idealo	Idealo internet GmbH	Data reseller	`idealo-bot`	Respects robots.txt	—
IDG	Unknown	Data reseller	`idg`	Unclear	—
Iframely	Iframely	User assistant	`https://iframely.com/docs/about`	Unclear	—
IFTTT RSS Feed Service	IFTTT RSS Feed Servuce	Data reseller	`ifttt`	Unclear	—
IIS Site Analysis	Unknown	Data reseller	`iis site analysis`	Unclear	—
IndeedJobBot	Indeed	Data reseller	`indeedjobbot`	Respects robots.txt	—
Inetdex Bot	Unknown	Data reseller	`inetdex bot`	Unclear	—
Infegy	Unknown	Data reseller	`infegy`	Unclear	—
InfoRoom Reader	GreenerSoft	Data reseller	`inforoomreader`	Unclear	—
InfoTigerBot	Unknown	Data reseller	`infotigerbot`	Unclear	—
Innguma Fetcher	Innguma	Data reseller	`innguma`	Unclear	—
Innologica	Innologica	Data reseller	`inoreader.com`	Unclear	—
Inspici	Unknown	Data reseller	`inspici`	Unclear	—
InsytfulBot	Unknown	Data reseller	`insytfulbot`	Unclear	—
Intelligence X	Unknown	Data reseller	`intelligence x`	Unclear	—
Internet Archive	Internet Archive	Data reseller	`archive.org_bot`	Respects robots.txt	—
Internet Archive - Archive-It	Archive-It	Data reseller	`archive-it`	Unclear	—
InternetMeasurement	Unknown	Data reseller	`internetmeasurement`	Unclear	—
IONOS Crawler	Unknown	Data reseller	`ionos crawler`	Unclear	—
IP-Guide Crawler	Unknown	Data reseller	`ip-guide crawler`	Unclear	—
IPS Agent	Unknown	Data reseller	`ips agent`	Unclear	—
IsItWP	Unknown	Data reseller	`isitwp`	Unclear	—
iTMS	Unknown	Data reseller	`itms`	Unclear	—
JobboerseBot	Unknown	Data reseller	`jobboersebot`	Unclear	—
JobicyBot	Jobicy	Search index	`jobicybot`	Respects robots.txt	—
jobswithgptcom-bot	JobsWithGPT	Search index	`jobswithgptcom`	Unclear	—
JungleKeyThumbnail	Unknown	Data reseller	`junglekeythumbnail`	Unclear	—
KadoBot	Unknown	Data reseller	`kadobot`	Unclear	—
Kagi Bot	Kagi	Search index	`kagibot/`	Unclear	—
KakaoTalk Scrap	Kakao	User assistant	`kakaotalk-scrap`	Unclear	—
kb.dk_bot	Netarkivet	Data reseller	`netarkivindsamling`	Unclear	—
KeldanNewsCrawler	Keldan ehf.	Search index	`keldannewscrawler`	Unclear	—
Keys.so	Unknown	Data reseller	`keys.so`	Unclear	—
KlarnaBot	Unknown	Data reseller	`klarnabot`	Unclear	—
KomodiaBot	Unknown	Data reseller	`komodiabot`	Unclear	—
Konturbot	Unknown	Data reseller	`konturbot`	Unclear	—
Kouio	Unknown	Data reseller	`kouio`	Unclear	—
Kozmonavt	Unknown	Data reseller	`kozmonavt`	Unclear	—
KStandBot	Unknown	Data reseller	`kstandbot`	Unclear	—
LAC IA Harvester	Unknown	Data reseller	`lac ia harvester`	Unclear	—
Larbin web crawler	Unknown	Data reseller	`larbin web crawler`	Unclear	—
leak.info	Unknown	Data reseller	`leak.info`	Unclear	—
LetSearch	Unknown	Data reseller	`letsearch`	Unclear	—
Level9SearchBot	Level 9	Search index	`level9searchbot/`	Unclear	—
Library Of Congress Web Archiving	United States Library of Congress	Data reseller	`https://www.loc.gov/programs/web-archiving/for-s`	Unclear	—
LightspeedSystemsCrawler	Unknown	Data reseller	`lightspeedsystemscrawler`	Unclear	—
Linespider	LINE Corporation	Search index	`linespider`	Unclear	—
LinkBloom	Unknown	Data reseller	`linkbloom`	Unclear	—
LinkCheck	Siteimprove	Data reseller	`linkcheck by siteimprove`	Unclear	—
LinkChecker Bot	LinkChecker	Data reseller	`linkchecker.pro`	Respects robots.txt	—
LinkCheckerBot	LinkChecker.Pro	Data reseller	`linkcheckerbot`	Respects robots.txt	—
Linkdex Bot	Unknown	Search index	`linkdex bot`	Unclear	—
LinkedInBot	LinkedIn	User assistant	`linkedinbot`	Unclear	—
LinkpadBot	Unknown	Data reseller	`linkpadbot`	Unclear	—
LinksIndexerBot	Links Indexer	User assistant	`linksindexerbot`	Unclear	—
LinkWalker	Unknown	Data reseller	`linkwalker`	Unclear	—
LiveJournal	Unknown	Data reseller	`livejournal`	Unclear	—
Loomly Bot	Loomly	User assistant	`loomlybot`	Unclear	—
Lumar	Unknown	Data reseller	`lumar`	Unclear	—
LumeWebScan	Unknown	Data reseller	`lumewebscan`	Unclear	—
LumtelBot	Unknown	Data reseller	`lumtelbot`	Unclear	—
MaCoCu	Unknown	Data reseller	`macocu`	Unclear	—
MADBbot	Unknown	Data reseller	`madbbot`	Unclear	—
MagiBot	Peak Labs	Search index	`magibot`	Unclear	—
MagnetmeBot	Magnet.me	Data reseller	`magnet.me-web/`	Unclear	—
Mail.Ru Bot	Unknown	Search index	`mail.ru bot`	Unclear	—
MailRUBot	Mail Russia	User assistant	`mail.ru_bot`	Respects robots.txt	—
MakeMerryBot	Unknown	Data reseller	`makemerrybot`	Unclear	—
Marginalia	Unknown	Data reseller	`marginalia`	Unclear	—
Marginalia Search	Marginalia	Search index	`search.marginalia.nu`	Unclear	—
MariaDB/MySQL Knowledge Base	Unknown	Data reseller	`mariadb/mysql knowledge base`	Unclear	—
marketgoo	marketgoo	Data reseller	`marketgoo`	Respects robots.txt	—
Mars Finder	Mars Flag	Search index	`(dbot)`	Respects robots.txt	—
masscan	Unknown	Data reseller	`masscan`	Unclear	—
masscan-ng	Unknown	Data reseller	`masscan-ng`	Unclear	—
Meanpath Bot	Unknown	Search index	`meanpath bot`	Unclear	—
MediaMonitoringBot	MediaMonitoringBot	Data reseller	`mediamonitoringbot/`	Respects robots.txt	—
Mediatoolkit Bot	Unknown	Data reseller	`mediatoolkit bot`	Unclear	—
Mediatoolkitbot	Determ	Data reseller	`mediatoolkitbot`	Respects robots.txt	—
Mediumbot	Unknown	Data reseller	`mediumbot`	Unclear	—
MegaIndex	Unknown	Data reseller	`megaindex`	Unclear	—
MelonMesa Bot	MelonMesa	Data reseller	`centcombot`	Unclear	—
MeltwaterNews	Unknown	Data reseller	`meltwaternews`	Unclear	—
MetaInspector	Unknown	Data reseller	`metainspector`	Unclear	—
MetaJobBot	Unknown	Data reseller	`metajobbot`	Unclear	—
MicroAdBot	Unknown	Data reseller	`microadbot`	Unclear	—
Microsoft Preview	Microsoft	User assistant	`microsoftpreview/`	Unclear	—
MidasNetworkBot	Midas Network d.o.o.	Data reseller	`midasnetworkbot`	Respects robots.txt	—
MirrorWebCrawler	MirrorWeb Ltd	Data reseller	`mirrorweb`	Unclear	—
Mixnode	Unknown	Data reseller	`mixnode`	Unclear	—
MixRank Bot	Unknown	Data reseller	`mixrank bot`	Unclear	—
MJ12 Bot	Unknown	Search index	`mj12 bot`	Unclear	—
MJ12Bot	Majestic	Data reseller	`mj12bot`	Unclear	—
Mnogosearch	Unknown	Search index	`mnogosearch`	Unclear	—
Mojeek	Mojeek	Search index	`mojeekbot`	Unclear	—
MojeekBot	Unknown	Search index	`mojeekbot`	Unclear	—
Monitor Backlinks	Unknown	Data reseller	`monitor backlinks`	Unclear	—
MonitoRSS	MonitoRSS	Data reseller	`monitorss/`	Respects robots.txt	—
Monsidobot	Unknown	Data reseller	`monsidobot`	Unclear	—
MoodleBot Linkchecker	Unknown	Search index	`moodlebot linkchecker`	Unclear	—
Morningscore Bot	Unknown	Data reseller	`morningscore bot`	Unclear	—
MotoMinerBot	Motominer	Data reseller	`motominerbot`	Unclear	—
Moz dotbot	Moz	Data reseller	`dotbot`	Unclear	—
Moz rogerbot	Moz	Data reseller	`rogerbot-crawler`	Respects robots.txt	—
MRGbot	MRG Web Services srl	Search index	`mrgbot/`	Respects robots.txt	—
MSN	Microsoft	Search index	`msnbot/`	Unclear	—
MTRobot	Unknown	Data reseller	`mtrobot`	Unclear	—
Muck Rack Scraper	Muck Rack	Data reseller	`muckrack.com`	Unclear	—
MuckRack	Unknown	Data reseller	`muckrack`	Unclear	—
MuscatFerret	Unknown	Data reseller	`muscatferret`	Unclear	—
najdu.s.holubem.eu	Unknown	Data reseller	`najdu.s.holubem.eu`	Unclear	—
NalezenCzBot	Unknown	Data reseller	`nalezenczbot`	Unclear	—
NameProtectBot	Unknown	Data reseller	`nameprotectbot`	Unclear	—
nbertaupete95	Unknown	Data reseller	`nbertaupete95`	Unclear	—
Neevabot	Neeva	Search index	`neevabot`	Unclear	—
Netcraft Survey Bot	Unknown	Search index	`netcraft survey bot`	Unclear	—
netEstate	Unknown	Data reseller	`netestate`	Unclear	—
Netpeak Checker	Unknown	Data reseller	`netpeak checker`	Unclear	—
NetTrack	Unknown	Data reseller	`nettrack`	Unclear	—
Netvibes	Unknown	Data reseller	`netvibes`	Unclear	—
NETZZAPPEN	Unknown	Data reseller	`netzzappen`	Unclear	—
New York Times Newsgathering	The New York Times	Data reseller	`scraping@nytimes\.com`	Unclear	—
NewsBank	NewsBank	Data reseller	`newsbank`	Unclear	—
NewsBlur	NewsBlur	Data reseller	`newsblur`	Unclear	—
NewsGator	Unknown	Data reseller	`newsgator`	Unclear	—
Newslitbot	Unknown	Data reseller	`newslitbot`	Unclear	—
NewsNow	NewsNowUK	Search index	`newsnow`	Unclear	—
NiceCrawler	Unknown	Data reseller	`nicecrawler`	Unclear	—
NitroBot	NitroPack Ltd	Data reseller	`nitro-`	Unclear	—
NLCrawler	Unknown	Data reseller	`nlcrawler`	Unclear	—
Nooshub	Nooshub	Data reseller	`nooshub`	Unclear	—
Nostra	Nostra	Data reseller	`nostra`	Unclear	—
Notabot	Helpfeel	Search index	`notabot`	Unclear	—
Nutch-based Bot	Unknown	Data reseller	`nutch-based bot`	Unclear	—
Nuzzel	Unknown	Data reseller	`nuzzel`	Unclear	—
oBot	Unknown	Search index	`obot`	Unclear	—
Observer	Obsrvr	Data reseller	`obsrvr.net`	Respects robots.txt	—
Odnoklassniki Bot	Unknown	Data reseller	`odnoklassniki bot`	Unclear	—
OI-Crawler	Unknown	Data reseller	`oi-crawler`	Unclear	—
OMIM Litrack	Johns Hopkins University	Data reseller	`omim-litrack`	Unclear	—
Onalytica	Unknown	Data reseller	`onalytica`	Unclear	—
OnCrawl	Oncrawl	Data reseller	`www.oncrawl.com`	Unclear	—
Online Webceo Bot	Online Webceo	Data reseller	`online-webceo-bot`	Unclear	—
Opengraph Bot	Opengraph	User assistant	`opengraph.io/`	Unclear	—
OpenGraph.io	Unknown	Data reseller	`opengraph.io`	Unclear	—
Openindex Spider	Unknown	Search index	`openindex spider`	Unclear	—
OpenLinkProfiler	Unknown	Data reseller	`openlinkprofiler`	Unclear	—
OpenRobotsTXT	Unknown	Data reseller	`openrobotstxt`	Unclear	—
OpenRSS	OpenRSS	Data reseller	`openrss`	Unclear	—
OpenWebSpider	Unknown	Data reseller	`openwebspider`	Unclear	—
Orange Bot	Unknown	Search index	`orange bot`	Unclear	—
Orlo Link Preview	Orlo	User assistant	`orlo-linkpreview/`	Unclear	—
Oseox	Oseox	Data reseller	`oseox`	Respects robots.txt	—
OSZKbot	Unknown	Data reseller	`oszkbot`	Unclear	—
OurFamilyWizard	OurFamilyWizard	Data reseller	`cozi-icalendar-feedreader`	Unclear	—
Outbrain	Unknown	Data reseller	`outbrain`	Unclear	—
Overcast	Overcast Radio	Data reseller	`overcast/1.0 podcast sync`	Unclear	—
OWLer	Unknown	Data reseller	`owler`	Unclear	—
Ozon Web Grabber	Ozon	User assistant	`odklbot/`	Unclear	—
Pageburst	Unknown	Data reseller	`pageburst`	Unclear	—
Pagefreezer Website Archiving	Pagefreezer	Data reseller	`pagefreezer`	Unclear	—
PagePeeker	Unknown	Data reseller	`pagepeeker`	Unclear	—
PageThing	Unknown	Data reseller	`pagething`	Unclear	—
Pandalytics	Unknown	Data reseller	`pandalytics`	Unclear	—
PaperLiBot	Unknown	Search index	`paperlibot`	Unclear	—
Paqlebot	Unknown	Data reseller	`paqlebot`	Unclear	—
parse.ly	Unknown	Data reseller	`parse.ly`	Unclear	—
ParticleNewsBot	Unknown	Data reseller	`particlenewsbot`	Unclear	—
Peer39	Unknown	Data reseller	`peer39`	Unclear	—
Petal Bot	Unknown	Data reseller	`petal bot`	Unclear	—
Picsearch bot	Unknown	Search index	`picsearch bot`	Unclear	—
Pigafetta	Unknown	Data reseller	`pigafetta`	Unclear	—
Pinterest	Unknown	Data reseller	`pinterest`	Unclear	—
PinterestBot	Pinterest	Data reseller	`pinterest.com/`	Unclear	—
PiplBot	Unknown	Data reseller	`piplbot`	Unclear	—
Platebreaker	Platebreaker	Search index	`platebreaker`	Unclear	—
PlayStore-Google	Unknown	Data reseller	`playstore-google`	Unclear	—
Plukkie	Unknown	Data reseller	`plukkie`	Unclear	—
Pocket Casts Feed Parser	Pocket Casts	Data reseller	`pocket casts feed parser`	Unclear	—
Podroll Analyzer	Unknown	Data reseller	`podroll analyzer`	Unclear	—
Pompos	Unknown	Data reseller	`pompos`	Unclear	—
Potions	Potions	Search index	`potions/`	Unclear	—
prerender	Prerender, LLC	Data reseller	`prerender`	Unclear	—
Presearch	Unknown	Search index	`presearch`	Unclear	—
PressEngine Bot	PressEngine	User assistant	`pressenginebot`	Respects robots.txt	—
Pricey	Wilico	Data reseller	`pricey`	Unclear	—
PritTorrent	Unknown	Data reseller	`prittorrent`	Unclear	—
Project Patchwatch	Unknown	Data reseller	`project patchwatch`	Unclear	—
Project Resonance	Unknown	Data reseller	`project resonance`	Unclear	—
Protopage	Protopage Ltd	Data reseller	`protopage`	Unclear	—
Quantcast	Unknown	Data reseller	`quantcast`	Unclear	—
Quartr Crawler	Quartr	Data reseller	`quartr crawler`	Unclear	—
QuerySeekerSpider	Unknown	Data reseller	`queryseekerspider`	Unclear	—
Quora Bot	Unknown	Data reseller	`quora bot`	Unclear	—
Quora Link Preview	Unknown	Data reseller	`quora link preview`	Unclear	—
Qwantbot	Qwant	Search index	`qwantbot`	Unclear	—
Rainmeter	Unknown	Data reseller	`rainmeter`	Unclear	—
Rakuten Image extraction bot	Rakuten	Data reseller	`rakutenusabot-image`	Unclear	—
RamblerMail Image Proxy	Unknown	Data reseller	`ramblermail image proxy`	Unclear	—
RavenCrawler	Unknown	Data reseller	`ravencrawler`	Unclear	—
RDTvlokipBot	Theo CHARLET	Search index	`rdtvlokipbot`	Respects robots.txt	—
RedekenBot	Unknown	Data reseller	`redekenbot`	Unclear	—
ReqBin	Unknown	Data reseller	`reqbin`	Unclear	—
Research JLU	Unknown	Data reseller	`research jlu`	Unclear	—
Research Scan	Unknown	Data reseller	`research scan`	Unclear	—
Revvim	Revvim	Data reseller	`revvimgort`	Unclear	—
Rogerbot	Unknown	Data reseller	`rogerbot`	Unclear	—
ROI Hunter	Unknown	Data reseller	`roi hunter`	Unclear	—
RSiteAuditor	Unknown	Data reseller	`rsiteauditor`	Unclear	—
RSS API	RSS API (by Tibush GmbH)	Data reseller	`rssapi/`	Unclear	—
rss2tg_bot	Yellow Rubber Duck Consulting	Data reseller	`rss2tg bot`	Unclear	—
Ryowl	Unknown	Data reseller	`ryowl`	Unclear	—
SabsimBot	Unknown	Data reseller	`sabsimbot`	Unclear	—
SafeDNSBot	Unknown	Data reseller	`safednsbot`	Unclear	—
Sandoba//Crawler	Unknown	Data reseller	`sandoba//crawler`	Unclear	—
SBIder	Unknown	Data reseller	`sbider`	Unclear	—
Scamadviser External Hit	Unknown	Data reseller	`scamadviser external hit`	Unclear	—
Scour RSS Bot	Scour	Data reseller	`scour.ing`	Unclear	—
Scraping Robot	Unknown	Data reseller	`scraping robot`	Unclear	—
Screaming Frog SEO Spider	Unknown	Data reseller	`screaming frog seo spider`	Unclear	—
ScreenerBot	Unknown	Data reseller	`screenerbot`	Unclear	—
Screpy	screpy.com	Data reseller	`screpy`	Unclear	—
SearchAtlas Bot	Search Atlas	Data reseller	`searchatlas bot`	Unclear	—
Seekport	SISTRIX	Search index	`seekport`	Unclear	—
Sellers.Guide	Unknown	Data reseller	`sellers.guide`	Unclear	—
semaltbot	Unknown	Data reseller	`semaltbot`	Unclear	—
Semantic Scholar Bot	Unknown	Data reseller	`semantic scholar bot`	Unclear	—
Semrushbot	Semrush	Data reseller	`semrushbot`	Unclear	—
SemrushBotBacklinkAudit	Semrush	Data reseller	`semrushbot-ba`	Unclear	—
SemrushBotBacklinks	Semrush	Data reseller	`semrushbot/`	Unclear	—
SemrushBotLinkBuilding	Semrush	Data reseller	`semrushbot;`	Unclear	—
SemrushBotSI	Semrush	Data reseller	`semrushbot-si`	Unclear	—
Senuto	Unknown	Data reseller	`senuto`	Unclear	—
SEO Audit Check Bot	SEO Audit Check	Data reseller	`seo-audit-check-bot`	Unclear	—
seo4ajax	Capsule Code	Data reseller	`seo4ajax.com`	Unclear	—
Seobility	Unknown	Data reseller	`seobility`	Unclear	—
SeobilityBot	Seobility	Data reseller	`seobilitybot`	Unclear	—
SEOENGBot	Unknown	Data reseller	`seoengbot`	Unclear	—
SEOkicks	Unknown	Data reseller	`seokicks`	Unclear	—
SeolytBot	Unknown	Data reseller	`seolytbot`	Unclear	—
Seoscanners.net	Unknown	Data reseller	`seoscanners.net`	Unclear	—
SERankingBacklinksBot	SE Ranking	Data reseller	`serankingbacklinksbot`	Unclear	—
Serendeputy Bot	Unknown	Data reseller	`serendeputy bot`	Unclear	—
SerpstatBot	Serpstat	Data reseller	`serpstatbot`	Unclear	—
ServerHunterSpider	Server Hunter	Data reseller	`serverhunterspider`	Unclear	—
Seznam	Seznam	Search index	`seznambot`	Unclear	—
Seznam Bot	Unknown	Search index	`seznam bot`	Unclear	—
Seznam Email Proxy	Unknown	Data reseller	`seznam email proxy`	Unclear	—
Seznam Zbozi.cz	Unknown	Data reseller	`seznam zbozi.cz`	Unclear	—
sfFeedReader	Unknown	Data reseller	`sffeedreader`	Unclear	—
ShopAlike	Unknown	Data reseller	`shopalike`	Unclear	—
Shopify Partner	Unknown	Data reseller	`shopify partner`	Unclear	—
Shortwave Image Fetcher	Shortwave Communications Inc.	User assistant	`shortwave image fetcher`	Unclear	—
SilverReader	Unknown	Data reseller	`silverreader`	Unclear	—
Simbiat Software	Unknown	Data reseller	`simbiat software`	Unclear	—
Sirdata	Unknown	Data reseller	`sirdata`	Unclear	—
SISTRIX Crawler	Unknown	Data reseller	`sistrix crawler`	Unclear	—
SISTRIX Optimizer	Unknown	Data reseller	`sistrix optimizer`	Unclear	—
SiteAuditBot	Semrush	Data reseller	`siteauditbot`	Unclear	—
Sitebulb	Unknown	Data reseller	`sitebulb`	Unclear	—
SiteCheckerBotCrawler	Unknown	Data reseller	`sitecheckerbotcrawler`	Unclear	—
SiteGuru	SiteGuru	Data reseller	`sitegurucrawler`	Respects robots.txt	—
Siteimprove	Unknown	Search index	`siteimprove`	Unclear	—
Siteimprove Crawl	Siteimprove	Data reseller	`sitecheck-sitecrawl`	Unclear	—
SitemapParser-VIPnytt	Unknown	Data reseller	`sitemapparser-vipnytt`	Unclear	—
SiteOne Crawler	Unknown	Data reseller	`siteone crawler`	Unclear	—
SiteScore	Unknown	Data reseller	`sitescore`	Unclear	—
SiteSearch360	SEMKNOX GmbH	Search index	`sitesearch360`	Unclear	—
SiteSucker	Unknown	Data reseller	`sitesucker`	Unclear	—
Skroutz ImageBot	Skroutz S.A.	Data reseller	`skroutz imagebot`	Unclear	—
SkroutzBot	Skroutz S.A.	Data reseller	`skroutzbot`	Unclear	—
Skype	Skype	User assistant	`skypeuripreview`	Unclear	—
Slack Image Proxy	Slack	User assistant	`slack-imgproxy`	Unclear	—
Slackbot	Unknown	Data reseller	`slackbot`	Unclear	—
SMTBot	Unknown	Search index	`smtbot`	Unclear	—
SMTnet PM Bot	SMTnet	Search index	`smtnetpmbot/`	Respects robots.txt	—
Snapchat Ads	Unknown	Data reseller	`snapchat ads`	Unclear	—
Snapchat Proxy	Unknown	Data reseller	`snapchat proxy`	Unclear	—
Sogou Spider	Unknown	Search index	`sogou spider`	Unclear	—
Soso Spider	Unknown	Search index	`soso spider`	Unclear	—
sourcedash	Source Dashboard	Search index	`sourcedash`	Respects robots.txt	—
Sparkler	Unknown	Data reseller	`sparkler`	Unclear	—
Spawning AI	Unknown	Data reseller	`spawning ai`	Unclear	—
SpiderLing	Unknown	Data reseller	`spiderling`	Unclear	—
Spinn3r	Unknown	Data reseller	`spinn3r`	Unclear	—
SplitSignalBot	Unknown	Data reseller	`splitsignalbot`	Unclear	—
Sprinklr	Unknown	Data reseller	`sprinklr`	Unclear	—
Sputnik Bot	Unknown	Data reseller	`sputnik bot`	Unclear	—
Sputnik Favicon Bot	Unknown	Data reseller	`sputnik favicon bot`	Unclear	—
Sputnik Image Bot	Unknown	Data reseller	`sputnik image bot`	Unclear	—
Spyglasses	Spyglasses	Data reseller	`spyglassesbot`	Unclear	—
sqalix	Unknown	Data reseller	`sqalix`	Unclear	—
StackScopeBot	Unknown	Data reseller	`stackscopebot`	Unclear	—
start.me	Unknown	Data reseller	`start.me`	Unclear	—
Startpagina Linkchecker	Unknown	Search index	`startpagina linkchecker`	Unclear	—
Statabot	StataCorp LLC	Data reseller	`statabot`	Unclear	—
Statista	Unknown	Data reseller	`statista`	Unclear	—
StatistikAustria	Statistik Bot	Data reseller	`statistikaustria`	Unclear	—
StatOnline.ru	Unknown	Data reseller	`statonline.ru`	Unclear	—
StatsDroneBot	StatsDrone	Data reseller	`statsdronebot`	Unclear	—
StatusNestBacklinkSpider	Unknown	Data reseller	`statusnestbacklinkspider`	Unclear	—
Steam Chat	Valve Software	User assistant	`steamchat`	Unclear	—
Steve Bot	Unknown	Data reseller	`steve bot`	Unclear	—
Stract	Unknown	Data reseller	`stract`	Unclear	—
Sublinq	Unknown	Data reseller	`sublinq`	Unclear	—
Substack Content Fetch	Unknown	Data reseller	`substack content fetch`	Unclear	—
SuggestBot	Unknown	Data reseller	`suggestbot`	Unclear	—
Superfeedr Bot	Unknown	Data reseller	`superfeedr bot`	Unclear	—
SurdotlyBot	Unknown	Data reseller	`surdotlybot`	Unclear	—
Survey Bot	Unknown	Search index	`survey bot`	Unclear	—
svEye	Unknown	Data reseller	`sveye`	Unclear	—
Swiftbot	Unknown	Data reseller	`swiftbot`	Unclear	—
Swisscows Favicons	Unknown	Data reseller	`swisscows favicons`	Unclear	—
Synapse	Unknown	Data reseller	`synapse`	Unclear	—
t3versions	Unknown	Data reseller	`t3versions`	Unclear	—
Taboolabot	Unknown	Data reseller	`taboolabot`	Unclear	—
TactiScout	Unknown	Data reseller	`tactiscout`	Unclear	—
Tag Inspector	Unknown	Data reseller	`tag inspector`	Unclear	—
Tarmot Gezgin	Unknown	Search index	`tarmot gezgin`	Unclear	—
tchelebi	Unknown	Data reseller	`tchelebi`	Unclear	—
Telegram Bot	Telegram	User assistant	`telegrambot`	Unclear	—
TestCrawler	Unknown	Data reseller	`testcrawler`	Unclear	—
The British Library Legal Deposit Bot	Unknown	Data reseller	`the british library legal deposit bot`	Unclear	—
The Trade Desk Content	Unknown	Data reseller	`the trade desk content`	Unclear	—
TigerBot	Unknown	Data reseller	`tigerbot`	Unclear	—
TinEye	Unknown	Data reseller	`tineye`	Unclear	—
TinEye Crawler	Unknown	Search index	`tineye crawler`	Unclear	—
Tiny Tiny RSS	Unknown	Data reseller	`tiny tiny rss`	Unclear	—
Toutiao	Bytedance	Search index	`zhanzhang.toutiao.com`	Unclear	—
TraceMyFile	Unknown	Search index	`tracemyfile`	Unclear	—
Trellis-Services	Mediavine	Data reseller	`trellis-services`	Unclear	—
Trendiction Bot	Unknown	Data reseller	`trendiction bot`	Unclear	—
Trendsmap	Unknown	Data reseller	`trendsmap`	Unclear	—
TrustBot	Unknown	Data reseller	`trustbot`	Unclear	—
TrustedSite	TrustedSite (formerly McAfee SECURE)	Data reseller	`trustedsite`	Unclear	—
Tumblr	Automattic	User assistant	`tumblr/`	Unclear	—
TurnitinBot	Turnitin	Data reseller	`turnitin`	Unclear	—
TweetedTimes Bot	Unknown	Data reseller	`tweetedtimes bot`	Unclear	—
Tweetmeme Bot	Unknown	Data reseller	`tweetmeme bot`	Unclear	—
Twingly Recon	Unknown	Data reseller	`twingly recon`	Unclear	—
twitterbot	Twitter	User assistant	`twitterbot/`	Unclear	—
Twurly	Unknown	Data reseller	`twurly`	Unclear	—
UCSB Network Measurement	Unknown	Data reseller	`ucsb network measurement`	Unclear	—
UkrNet Mail Proxy	Unknown	Data reseller	`ukrnet mail proxy`	Unclear	—
uMBot	Unknown	Data reseller	`umbot`	Unclear	—
UniversalFeedParser	Unknown	Data reseller	`universalfeedparser`	Unclear	—
upday	upday GmbH & Co. KG	Data reseller	`upday/`	Respects robots.txt	—
URLAppendBot	Unknown	Data reseller	`urlappendbot`	Unclear	—
URLSuMaBot	Unknown	Data reseller	`urlsumabot`	Unclear	—
Vagabondo	Unknown	Data reseller	`vagabondo`	Unclear	—
ValidBot	Unknown	Data reseller	`validbot`	Unclear	—
Velen Public Web Crawler	Unknown	Data reseller	`velen public web crawler`	Unclear	—
VeryHip	Unknown	Data reseller	`veryhip`	Unclear	—
videootv Bot	Digital Green	Data reseller	`videootvbot`	Respects robots.txt	—
VirusTotal Cloud	Unknown	Data reseller	`virustotal cloud`	Unclear	—
Visual Site Mapper Crawler	Unknown	Data reseller	`visual site mapper crawler`	Unclear	—
VK Robot	Unknown	Data reseller	`vk robot`	Unclear	—
VK Share Button	Unknown	Data reseller	`vk share button`	Unclear	—
VORTEX	Unknown	Data reseller	`vortex`	Unclear	—
vuhuvBot	Unknown	Search index	`vuhuvbot`	Unclear	—
W3 Validator Services	World Wide Web Consortium (W3C)	Data reseller	`w3c-checklink`	Unclear	—
WebbCrawler	Unknown	Data reseller	`webbcrawler`	Unclear	—
WebCEO	Unknown	Data reseller	`webceo`	Unclear	—
WebDataStats	Unknown	Data reseller	`webdatastats`	Unclear	—
Webliobot	Unknown	Data reseller	`webliobot`	Unclear	—
Weborama	Unknown	Search index	`weborama`	Unclear	—
WebPros	Unknown	Data reseller	`webpros`	Unclear	—
Website-info	Unknown	Data reseller	`website-info`	Unclear	—
WebSpiderMount	AspenTechLabs Inc	Data reseller	`webspidermount`	Unclear	—
webtru	Unknown	Data reseller	`webtru`	Unclear	—
Webwiki	Unknown	Data reseller	`webwiki`	Unclear	—
WellKnownBot	Unknown	Data reseller	`wellknownbot`	Unclear	—
WeSEE:Search	Unknown	Search index	`wesee:search`	Unclear	—
WeViKaBot	Unknown	Data reseller	`wevikabot`	Unclear	—
WhatCMS	Unknown	Data reseller	`whatcms`	Unclear	—
WhereGoes	Unknown	Data reseller	`wheregoes`	Unclear	—
Who.is Bot	Unknown	Data reseller	`who.is bot`	Unclear	—
Wibybot	Unknown	Data reseller	`wibybot`	Unclear	—
WikiDo	Unknown	Search index	`wikido`	Unclear	—
Willow Internet Crawler	Unknown	Data reseller	`willow internet crawler`	Unclear	—
WindowsForum-AI	WindowsForum.com	Search index	`windowsforum-ai`	Unclear	—
WireReaderBot	Unknown	Data reseller	`wirereaderbot`	Unclear	—
WMF Citoid	Wikimedia Foundation	User assistant	`citoid/wmf`	Unclear	—
WMF Zotero Translation Server	Wikimedia Foundation	User assistant	`zoterotranslationserver/wmf`	Unclear	—
WooRank	Unknown	Search index	`woorank`	Unclear	—
WordCountBot	Weglot	User assistant	`wordcountbot`	Unclear	—
Workona	Unknown	Data reseller	`workona`	Unclear	—
Wotbox	Unknown	Search index	`wotbox`	Unclear	—
wp.com feedbot	Unknown	Data reseller	`wp.com feedbot`	Unclear	—
WPMU DEV	Unknown	Data reseller	`wpmu dev`	Unclear	—
XoviBot	Unknown	Data reseller	`xovibot`	Unclear	—
xtate	Unknown	Data reseller	`xtate`	Unclear	—
XY Archive Compliance Bot	XY Archive Compliance	Data reseller	`xy-archive-compliance`	Unclear	—
YaCy	Unknown	Search index	`yacy`	Unclear	—
Yahoo Japan SEO Crawler	Yahoo	Data reseller	`j-brw`	Unclear	—
Yahoo Link Preview	Yahoo	User assistant	`yahoo link preview`	Unclear	—
Yahoo Mail	Yahoo	User assistant	`yahoomailproxy`	Unclear	—
Yahoo Slurp	Yahoo	Search index	`yahoo! slurp`	Respects robots.txt	—
Yahoo! Cache System	Unknown	Data reseller	`yahoo! cache system`	Unclear	—
Yahoo! JAPAN	Yahoo! JAPAN	Search index	`j-dlc/`	Respects robots.txt	—
Yahoo! Japan $1	Unknown	Data reseller	`yahoo! japan $1`	Unclear	—
YahooCacheSystem	Yahoo	User assistant	`yahoocachesystem`	Unclear	—
YahooMailProxy	Yahoo	User assistant	`yahoomailproxy`	Unclear	—
YandexBot	Yandex	Search index	`http://yandex.com/bots`	Unclear	—
YandexBot-MirrorDetector	Unknown	Data reseller	`yandexbot-mirrordetector`	Unclear	—
Yeti	Naver	Search index	`yeti/`	Unclear	—
Yeti/Naverbot	Unknown	Search index	`yeti/naverbot`	Unclear	—
YioopBot	Unknown	Search index	`yioopbot`	Unclear	—
yoozBot	Unknown	Search index	`yoozbot`	Unclear	—
Youdao Bot	Unknown	Search index	`youdao bot`	Unclear	—
Yourls	Unknown	Data reseller	`yourls`	Unclear	—
Yunyun Bot	Unknown	Search index	`yunyun bot`	Unclear	—
Zaldamo	Unknown	Data reseller	`zaldamo`	Unclear	—
Zao	Unknown	Data reseller	`zao`	Unclear	—
Ze List	Unknown	Data reseller	`ze list`	Unclear	—
Zeno	Unknown	Data reseller	`zeno`	Unclear	—
Zookabot	Unknown	Data reseller	`zookabot`	Unclear	—
Zoombot	SEOZoom s.r.l.	Data reseller	`zoombot`	Unclear	—
ZoomInfo	ZoomInfo	Search index	`zoominfobot`	Unclear	—
ZoominfoBot	Unknown	Data reseller	`zoominfobot`	Unclear	—
ZumBot	Zum Internet Corp	Search index	`zumbot`	Unclear	—

⬇ Export CSV ⬇ Export JSON

How to block AI crawlers in robots.txt

Place a robots.txt file at the root of your domain (https://example.com/robots.txt). List each crawler you want to exclude, then a single Disallow: /. To block every known AI bot at once, use the generator above — it stays current as new crawlers appear. Note that robots.txt is a request, not enforcement; combine it with server- or CDN-level rules if you need hard blocking.

Frequently asked questions

What is an AI crawler (AI bot)?

An AI crawler is an automated agent that fetches web pages to power AI products — training large language models, answering questions in AI search (ChatGPT, Perplexity, Gemini), or acting on a user's behalf. Our database tracks 228 AI crawlers plus adjacent search/SEO/feed bots (930 in total).

How do I block AI crawlers?

Add the crawlers you want to exclude to your site's /robots.txt with a User-agent line each and a Disallow: /. Use the generator on this page to copy-paste (or download) a ready-made robots.txt that blocks every known AI bot, kept up to date automatically.

Do AI crawlers respect robots.txt?

Most major crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot) state that they honour robots.txt. Some are undocumented or ignore it. Each entry in our database shows the crawler's stated robots.txt policy.

What is the GPTBot user agent, and how do I block it?

GPTBot is OpenAI's training crawler. Block it with User-agent: GPTBot then Disallow: / in robots.txt. See the GPTBot profile for its full user-agent string and how many sites block it.

What's the difference between a training crawler and an answer engine?

Training crawlers (GPTBot, CCBot, Google-Extended) collect content to train models — blocking them opts you out of training. Answer engines (OAI-SearchBot, PerplexityBot) fetch content to cite in live AI answers — blocking them removes you from AI search results (AI visibility).

Can I tell if a bot is really who it claims to be?

User-agents can be spoofed. We verify major crawlers by forward-confirmed reverse DNS and, where available, official IP ranges — the profile pages show a verification badge for bots we've confirmed on our own network.

Which AI bots do most websites block?

Across 48M+ live websites we measure, training crawlers are blocked far more than answer engines — see the Observatory for the live ranking and per-country / per-category breakdowns.

How complete and how current is this list?

It aggregates every open source (ai.robots.txt, the Cloudflare verified-bot directory, matomo device-detector and more), deduplicated with source attribution, and it grows automatically: any unrecognised bot we catch fetching our llms.txt / ai.txt is reviewed and added. Currently 930 crawlers.

Categories All →

Technologies All →

Countries All →

Languages All →