Meta · AI crawler
2,014,939 websites block meta-externalagent
That’s 4.152% of the 48,534,257 live websites we index — the #7 most-blocked AI crawler.
Model trainingRespects robots.txtAI Data Scrapers
"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly."
Official documentation ↗
🛰️ Observed crawling piperic.com unverified
68,984 requests · last seen 3m ago · 50 IP(s)
Fetched: HTML (50790) · other (18194)
Block meta-externalagent in robots.txt
Add this to your site’s /robots.txt:
User-agent: meta-externalagent
Disallow: /
Who blocks meta-externalagent
Top countries blocking it
Top categories blocking it
← All AI bots