Pay per crawl

Pay per crawl is Cloudflare's mechanism for charging automated agents to fetch a page. It turns the access question from a binary, allow or block, into a price, using an HTTP status code reserved for the purpose since the early web and almost never used.

Cloudflare's documentation describes it as a feature of AI Crawl Control that lets site owners control and monetize AI crawler access by setting a price per zone. Cloudflare acts as the Merchant of Record. As of the documentation's 28 July 2026 revision, the feature is in closed beta.

How does pay per crawl work?

The exchange is plain HTTP. Each time an AI crawler requests content, one of two things happens: it presents payment intent in request headers and receives a normal HTTP 200, or it receives HTTP 402 Payment Required with the price.

That is the whole protocol surface. A crawler that knows nothing about the scheme gets a 402 and decides what to do; one that participates signals intent on the request. Both sides need accounts: the site owner enables the feature and sets a price, and the AI company sets up an account, verifies its crawler and connects payment.

Does a firewall block override pay per crawl?

Yes, and this is the opposite of what people assume. Cloudflare states that if you block an AI crawler via its WAF or Bot Management products, those rules override pay per crawl's charge feature and the blocked crawler has no access.

So a firewall rule is not a fallback that collects money when it fires. It is a hard denial that removes the paid path. Wanting to sell access and blocking the buyer can coexist quietly in one configuration, and the block wins.

Which crawlers would you actually be charging?

Look at your own traffic before setting a price. Here is ours. attensira.com allows every major AI crawler in robots.txt and counts requests at the edge. In the 30 days to 4 October 2026, by self-identified user agent, the split was lopsided.

Meta-ExternalAgent, a training crawler, made 131,105 of the 143,601 requests, the large majority, and almost all of them in the last seven days. OpenAI's three agents made 7,536, mostly for search; Bytespider, also training, made 3,505; PerplexityBot made 621, bingbot 541 and Anthropic's two agents 104. The chart at the top of this page draws the gap to scale.

The crawler that took the most from us by far collects training data and returns no citation. The crawlers behind the answer engines that do cite us asked for a small fraction of that. That asymmetry is the honest case for pay per crawl: the traffic volume and the value returned are not related. We still allow all of them, because being readable to every engine is our product's argument; a publisher whose archive is the product could rationally decide otherwise. See Meta-ExternalAgent and PerplexityBot for what each documents about itself.

Why does pay per crawl exist?

Cloudflare's framing is that AI crawlers consume vast amounts of content; some provide mutual benefit by indexing for search engines, while others scrape without permission, leaving content owners with few options for managing access or being paid.

The middle clause is the important one. Cloudflare's own description separates crawlers that return value through indexing from those that do not, the same distinction that makes a blanket block a bad default. Refusing everything also refuses the agents whose whole job is to be able to cite you.

What does pay per crawl change about measurement?

It is the clearest sign that robots.txt no longer fully describes who may read a site. A directive file is a request honoured at the crawler's discretion; an edge returning 402 is enforcement.

Two consequences follow. A robots.txt that permits an agent is an upper bound on access, not a measurement of it; the page may still sit behind a payment wall, a bot rule or a rate limit. And the honest test of whether an agent can read you is server logs: a crawler that is charged, blocked or challenged leaves a different trace from one that is served.

What goes wrong with pay per crawl?

• Charging the crawlers that cite you. Pricing search and user-triggered agents prices your own distribution.
• A block left in place. It silently overrides the charge, so nobody pays and nobody reads.
• Reading robots.txt as the full policy. Edge rules decide what actually happens.
• Building on beta behaviour. The feature is in closed beta; re-check the documentation before relying on it.

Does Attensira manage pay per crawl?

No. Attensira does not set Cloudflare rules or prices. What it gives you is the two columns the pricing decision needs, side by side. The first is cost: with crawler logs installed in your edge middleware, each operator's requests are counted and classed by why the agent came (training, search indexing or a live answer) and by whether a person triggered the fetch. The second is return: whether the engine behind that operator names and cites you in the answers Attensira reads every day. On our own site the two columns disagree completely. Meta-ExternalAgent sent the most requests of any agent and feeds no engine Attensira reads, while Perplexity's crawler sent a few hundred requests and Perplexity answers cited attensira.com in 85 of 2,175 answers in the same 30 days. If you charge, charge on that evidence, per agent.