Pay per crawl

Pay per crawl is Cloudflare's mechanism for charging automated agents to fetch a page. It turns the access question from a binary — allow or block — into a price, using a status code that has been reserved for the purpose since HTTP was young and almost never used.

Cloudflare's documentation describes it as a feature of AI Crawl Control that enables site owners to control and monetize AI crawler access to content by setting a price per zone. Cloudflare acts as the Merchant of Record and provides the underlying technical infrastructure. As of the documentation's 28 July 2026 revision the feature is in closed beta.

The mechanism

The exchange is deliberately plain HTTP. Each time an AI crawler requests content, one of two things happens: it presents payment intent via request headers and receives normal HTTP 200 access, or it receives an HTTP 402 Payment Required response carrying the price.

That is the whole protocol surface. A crawler that knows nothing about the scheme simply gets a 402 and can decide what to do; a crawler that participates signals intent on the request. Both sides need accounts — the documentation has separate paths for a site owner enabling it and setting a price, and for an AI owner setting up an account, verifying their crawler and connecting payment before discovering and crawling payable content.

Blocking still overrides charging

The interaction rule is worth committing to memory before configuring anything, because it is the opposite of what people assume. Cloudflare states that if you block an AI crawler from a zone via its WAF or Bot Management products, those rulesets override pay per crawl's charge feature and the blocked crawler will not have access.

So a firewall rule is not a fallback that collects money when it fires — it is a hard denial that removes the paid path entirely. Wanting to sell access and blocking the buyer are configurations that can coexist quietly, and the block wins.

Why it exists

Cloudflare's framing of the problem is that AI crawlers often consume vast amounts of web content; some provide mutual benefit to content owners by indexing content for search engines, while others engage in activities such as content scraping without permission. The resulting landscape, it says, leaves content owners with limited options for managing AI crawlers or receiving compensation for automated access to their intellectual property.

The important observation for anyone thinking about visibility is the middle clause. The document itself distinguishes crawlers that return value through indexing from those that do not — which is the same distinction that makes a blanket block a bad default. Refusing everything also refuses the agents whose whole job is to be able to cite you.

What it changes about measurement

Pay per crawl is the clearest example of a broader fact: robots.txt is no longer a complete description of who may read a site. A directive file is a request honoured at the crawler's discretion; an edge that returns 402 is an enforcement point the crawler cannot argue with.

Two consequences follow for anybody auditing access.

First, a robots.txt that permits an agent is an upper bound on access, not a measurement of it. The page may still be behind a payment wall, a bot rule or a rate limit that robots.txt says nothing about. Any study of AI crawler permissions — including our own — measures written policy, not reachability.

Second, the honest test of whether an agent can read you is server logs. A crawler that is charged, blocked or challenged leaves a different trace from one that is served, and no amount of reading configuration files substitutes for looking at what actually arrived.

Deciding

The decision this feature forces is not really a pricing decision, it is an editorial one: which of your content is a product you sell, and which is the argument that makes people buy the product. Documentation, reference material and anything you want quoted back to a buyer belongs in the second category, where being read is the point and charging for it is charging for your own distribution. An archive with standalone value is a different case.

Because the feature is in closed beta and its documentation is dated, treat the specifics above as a snapshot of 28 July 2026 rather than as settled behaviour, and re-check the source before building on it.