Pay Per Crawl is a model for charging bots per piece of content
When an AI bot arrives at a site, the question is no longer just "let it in or keep it out". In the approach Cloudflare calls Pay Per Crawl, there is a third answer. The bot can take the content, but it pays for every page it takes. The site owner sets a flat price per domain; if the bot accepts that price it gets the page, and if it does not, it doesn't.
The model is currently in private beta. It is not a switch anyone can flip in the dashboard; you have to apply to join. Cloudflare acts as the Merchant of Record in this flow, meaning it is the party that collects the fee from the bot operator and passes it on to the publisher. The minimum price is $0.001 per request, and the fee only applies when the bot actually receives the content, that is, on requests that return HTTP 200.
What does the HTTP 402 status code do in this model?
402 Payment Required is a status code that has long been reserved in the HTTP standard but has sat largely unused in practice. Its meaning is simple. The server is not refusing the request; it is saying it expects payment in exchange for the resource. 404 says "it doesn't exist", 403 says "you're not allowed", and 402 says "it exists, but it has a price".
Cloudflare has turned this code into a short negotiation language with bots. When a bot with no intent to pay requests paid content, it receives a 402 response with the price in the crawler-price header. If the bot accepts that price, it repeats the request with the crawler-exact-price header. The reverse is also possible. The bot states up front, with crawler-max-price, the maximum it is willing to pay; if the price is below that limit, the content comes back directly with a 200, and the crawler-charged header states how much was charged.
There is a distinction here that is often confused. In Cloudflare's AI Crawl Control dashboard, you can choose 402 as the block response code returned to bots, and this option is available on paid plans. But returning a 402 on its own does not make money. That response only conveys the message "this content is paid". The layer that carries the price, collects the payment and handles settlement is separate and part of the private beta program. A site that selects 402 in the dashboard and expects revenue has, in practice, simply blocked the bot.
How is the identity of a paying bot verified?
For charging to work, the bot's identity has to be provable. A user agent string is easy to spoof, so it is not possible to bill a user agent. Cloudflare uses the Web Bot Auth method for this. A bot operator joining the program generates an Ed25519 key pair, publishes its public key in a publicly accessible JWK directory, registers with Cloudflare and signs each request with the signature-agent, signature-input and signature headers.
This requirement has an important practical consequence. For an unregistered bot, a "charge" setting effectively means "block". A crawler that is not connected to the payment infrastructure sees the price, cannot pay and cannot get the content. So the real question in the charging decision is not "how much should I ask for" but "how much of the bot traffic reaching my site is actually able to pay".
What gap does the x402 protocol fill?
The 402 code on its own is an incomplete sentence. It says "payment required" but does not answer how much, in which currency, to where, and how payment is proven. x402 is an open protocol defined precisely to fill that gap. Cloudflare and Coinbase set up a foundation to steward the standard.
The flow is built on three headers. Along with the 402, the server sends the payment schemes it accepts, the price, the network and the destination address as a base64-encoded JSON object in the PAYMENT-REQUIRED header. The client authorizes the payment and repeats the request with the PAYMENT-SIGNATURE header. The server then reports the result of settlement with PAYMENT-RESPONSE. There is no account creation, API key or subscription step. That is why the protocol is designed for machine-to-machine use rather than for human users.
Cloudflare has also announced a layer that takes this protocol beyond crawlers. Monetization Gateway aims to make it possible to charge for any resource behind its network. Its scope covers web pages, datasets, APIs and MCP tools. Cloudflare describes payments as made in stablecoins that flow directly to the seller's wallet, so they do not depend on Cloudflare's billing layer. Access is rolling out through an early access list, and like Pay Per Crawl, it is not generally available.
The site owner's three options and the cost of each
The decision comes down to three options, and all three have a cost. There is no free option; only the cost you pay changes.
| Option | What it gains | What it costs | What it requires |
|---|---|---|---|
| Allow | Content can reach answer surfaces, and the chance of being mentioned as a brand and a source is preserved | Access brings no compensation on the copyright side, and the full crawl load stays with the publisher | No additional setup |
| Charge | Access creates a revenue opportunity and lays a technical foundation for licensing negotiations | Any bot not connected to the payment infrastructure cannot get the content, and visibility shrinks accordingly | Private beta participation and a bot registered for identity verification |
| Block | Crawl load and unauthorized use decrease, and the decision is applied in a single step | Visibility on the relevant surface ends completely, with no revenue in return | Can be applied even on the free plan |
The visibility cost of shutting off access
The most expensive mistake in this decision is treating all AI bots as a single category and blocking them all at once. A crawler collecting training data for a model and a crawler looking for sources while generating an answer to a user's question do not come for the same purpose, and blocking them does not have the same consequences.
On OpenAI's side, the distinction is clearly defined. GPTBot comes to collect content that may be used in training foundation models. OAI-SearchBot is used to surface sites in the results of ChatGPT's search features, so a site that blocks this bot does not appear in ChatGPT's search answers. ChatGPT-User does not crawl automatically; it fetches the page a user asks for at that moment.
On Google's side, the distinction works through robots.txt and follows a different logic. Google-Extended is a signal that determines whether content Googlebot has already fetched may be used for training and grounding in Gemini models. Google's documentation states its limits clearly. Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Because AI Overviews and AI Mode are features within Search, the path to those surfaces runs through Googlebot.
The practical conclusion is this. Closing off training and closing off answer surfaces are separate decisions. Stopping training crawlers over copyright concerns does not necessarily reduce search visibility. But putting a price on the crawler that looks for sources while generating answers means a direct loss of visibility if that crawler is not connected to the payment infrastructure. It is impossible to make this distinction without knowing which bots visit the site and how often. The free AI crawl audit tool is the place to start if you want to see which AI bots reach your site and how they are being handled.
Who Pay Per Crawl makes sense for
Where the model works, three things are true at the same time. First, the site has content that cannot easily be reproduced. Primary data, long archives, original research and numerical content produced from your own measurements fit this description. Second, bot traffic to this content reaches a meaningful volume. Third, the site's business model does not depend on visibility on that surface.
This combination is usually found at large publishers, organizations that run databases, academic and industry archives, and platforms that hold price or product data. For publishers like these, charging is really the technical counterpart of a licensing negotiation. Setting a price is a way of saying "this content is not free" before sitting down at the table. The way the private beta works fits this, too: participation goes through an application or an existing enterprise account representative.
Realistic expectations for small sites
For small and mid-sized sites, the honest answer is this: the model is not a revenue line today. The arithmetic shows why. The minimum charge is $0.001 per request. At that price, a thousand paid crawl requests are worth one dollar. Bot traffic to corporate brochure sites or service sites stays far below that volume. Raising the price is not a solution either, because the higher the price, the fewer bots choose to pay.
On the other hand, what the same site could lose is concrete. On service and local business sites, demand starts when users encounter the brand inside an answer. Not appearing on answer surfaces at all is a measurable loss for these sites. The equation tips in favor of charging for large archives, but not for small sites.
For small sites, the right order is control and visibility, not charging. Reducing unnecessary crawl load through robots.txt and crawl budget tuning, managing training and answer crawlers separately, and then working to make the content selectable as a source in answers offers a better return. That second part falls within the scope of AI visibility work.
Preparation to do before charging
The pricing decision comes at the end of the sequence. There are steps to complete before it, and most of them are valuable even if you never move on to charging.
- Measurement. Find out which AI bots visit the site, how often they make requests and what response they currently receive. The decision is made by looking at this table.
- Separation. Split training crawlers and search and answer crawlers into two separate lists. You are not obliged to make the same decision for both lists.
- Declaring preferences. Content Signals directives in robots.txt make your preferences on the use of content for search, answer generation and model training machine-readable. In Cloudflare's managed robots.txt file, the default line is
Content-signal: search=yes, ai-train=no, use=reference. This is a declaration, not a technical barrier, and compliance is voluntary. - Infrastructure. Because the charging flow runs through Cloudflare, the site's traffic has to pass through that network. For a site that is not yet connected, setting up Cloudflare is the first step.
- Decision. Put the measurement table next to the business model, and it becomes clear which option actually pays off. For most sites, the answer will not be charging.
Frequently Asked Questions
Can every site use Pay Per Crawl today?
No. The bot monitoring and blocking side of AI Crawl Control is available on all plans, and customizing the block response code is available on paid plans. Pay Per Crawl, which includes payment collection, is in private beta and participation requires an application.
Does simply returning a 402 generate revenue?
No. A 402 response is only a signal that the content is paid. For payment to actually happen, you need both the layer that handles pricing and collection and a verified bot connected to that flow. Without both, a 402 is in practice a block response.
Does charging affect visibility in Google Search?
Pay Per Crawl applies to AI crawlers, while Googlebot is a search crawler and falls outside that scope. However, a broadly written block rule can also catch search crawlers, so after writing a rule, you need to verify that Googlebot still has access.
Are x402 and Cloudflare's Pay Per Crawl the same thing?
No. Both are based on the HTTP 402 code. Pay Per Crawl uses Cloudflare's own crawler-price headers, and collection runs through Cloudflare's Merchant of Record role. x402 is an open protocol, used on Cloudflare's side together with Monetization Gateway, and defined so that payment flows directly to the seller's wallet.
What is the difference between Content Signals and charging?
Content Signals is a declaration of preference and, like robots.txt, relies on voluntary compliance. It does not technically stop a crawler that ignores the declaration. Charging and blocking, on the other hand, are enforced at the network level, meaning the party serving the request actually imposes the decision.
