AI content licensing & pay-per-crawl
AI crawlers read the open web to train models and answer questions — mostly for free. A new layer lets publishers charge for that access instead of just blocking it: Cloudflare Pay-Per-Crawl, TollBit, the open x402 protocol, and direct licensing deals. This page maps who is already positioned to monetize AI crawling — measured from our index of 49,080,155 live websites.
2,509,220
sites already block ≥1 AI crawler
1,986,951
of those are behind Cloudflare — Pay-Per-Crawl is one toggle away
2,940,755
reserve or prohibit AI-training rights in policy / ToS
Live figures from public robots.txt, ToS and CDN signals across our index. Not a measurement of traffic or revenue — see scope below.
Four ways to monetize AI crawling
Cloudflare Pay-Per-Crawl
Returns HTTP 402 (Payment Required) to AI bots and settles payment automatically. Built into Cloudflare — for the 15.9M sites already behind it, it's a dashboard toggle.
TollBit
A marketplace that meters and bills AI-crawler access to your content, with per-request pricing and analytics. Works regardless of your CDN.
x402 protocol
An open, HTTP-native micropayment standard (the revived 402 status): the crawler pays per request over the wire. Vendor-neutral and self-serve.
Direct licensing deals
Negotiated agreements — the route large publishers have taken with OpenAI and others. Highest value, but bespoke and slow; best for sites with significant, distinctive content.
Find the opportunity cohort
These open our live index, filtered — the content sites best positioned to charge for AI access. Export any list, or check a single site.
Blocks GPTBot and is on Cloudflare →
1,986,951 sites — one Cloudflare setting from Pay-Per-Crawl
Every site that blocks GPTBot →
The full, exportable directory of sites signalling AI-access intent
Business & finance sites blocking AI →
Filter by category — swap in technology, news, education and more
Check a single site →
Read any domain's AI-crawler policy live and see its monetization signal
What this measures — and what it doesn't
Honesty matters here. We measure only public signals: which AI bots a site blocks in robots.txt, its stated AI-training policy and Terms of Service, and whether it sits behind Cloudflare. We do not have other sites' server logs, so we never estimate their AI-crawler traffic or revenue, and we don't claim a site has actually enabled Pay-Per-Crawl, TollBit or signed a deal. This is a positioning signal from public data — a starting point for a decision, not a measurement of one.
Export the opportunity cohort
Get the full, filterable list of sites signalling AI-access intent — by category, country and tech — as CSV, JSON or XLSX.
Export the list ↗
Frequently asked questions
What is AI pay-per-crawl?
Pay-per-crawl lets a website charge AI crawlers for access instead of simply blocking or allowing them. Cloudflare's Pay-Per-Crawl returns an HTTP 402 (Payment Required) to AI bots and settles payment automatically; services like TollBit and the open x402 protocol do the same at the request level. It turns AI scraping — which sites currently give away for free or block outright — into a metered, paid API.
How do I know if my site could monetize AI crawling?
Two public signals suggest you're positioned for it: you already block AI crawlers in robots.txt (you've shown intent to control access), and your content is the kind AI wants (news, reference, data, docs). If you're also behind Cloudflare, Pay-Per-Crawl is a single dashboard setting. Our AI Crawler Checker reads these signals for any domain and flags the opportunity.
Is this the same as blocking GPTBot?
No. Blocking removes you from AI training and answers for free. Pay-per-crawl keeps the door open but attaches a price — the AI company can still crawl you, but it pays. Many publishers move from blanket blocking to metered access once they realise the traffic has value.
Does Piperic measure how much AI-bot traffic or revenue a site gets?
No — and we're explicit about that. We measure only public signals: which AI bots a site blocks in robots.txt, its stated AI-training policy and Terms of Service, and whether it sits behind Cloudflare. We do not have other sites' server logs, so we never estimate their AI-crawler traffic, revenue or whether they've actually enabled pay-per-crawl. The opportunity view is a positioning signal from public data, not a measurement.
What are the ways to license AI access to my content?
Four main routes today: (1) Cloudflare Pay-Per-Crawl — an HTTP-402 toll for AI bots, built in for sites on Cloudflare; (2) TollBit — a marketplace that meters and bills AI crawler access; (3) the x402 protocol — an open, HTTP-native micropayment standard; and (4) direct licensing deals, like the agreements large publishers have signed with OpenAI and others. The first three are self-serve; the fourth is negotiated.