Cloudflare · AI crawler
Cloudflare Web Scanner
Search indexRespects robots.txtSearch Engine Crawler
Cloudflare Web Scanner is a well-behaved crawler that checks domain liveness, content type, robots.txt directives, and other web properties to map the shape of the web.
Official documentation ↗
🛰️ We haven’t observed Cloudflare Web Scanner crawling piperic.com yet. This page updates automatically when it does.
Block Cloudflare Web Scanner in robots.txt
Add this to your site’s /robots.txt:
User-agent: Cloudflare Web Scanner
Disallow: /
← All AI bots