Robots.txt

Robots.txt is the oldest instruction a site gives a machine: a text file at /robots.txt that says which crawler may fetch what. For twenty years it mostly concerned search engines. Now it is also where a site decides how it deals with AI, crawler by crawler, and the decisions are easy to get wrong because one company can run several crawlers with different jobs.

How does robots.txt work?

The file is a list of groups. Each group names one or more user-agent tokens, then the paths that are allowed or disallowed for them. A crawler uses the most specific group that names it and falls back to User-agent: * if none does. Within a group the longest matching path wins.

The rules were standardised in 2022 as RFC 9309, the Robots Exclusion Protocol. It asks crawlers to parse at least the first 500 kibibytes, to follow at least five redirects to reach the file, and not to trust a cached copy for more than 24 hours. That last rule is why a change is not instant.

Two limits matter more than the syntax:

• It is a request. Nothing in the protocol stops a fetch. Reputable operators publish that they honour it; enforcement needs your server or CDN.
• It controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. Keeping a page out of an index is the job of a noindex rule, in the page or in an X-Robots-Tag header, and the crawler has to be allowed in to read it.

What does a real AI-era robots.txt look like?

Here is part of ours, as attensira.com serves it from this site's code on 4 October 2026. It has 34 groups and is 1,743 bytes long:

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /api/

User-agent: CCBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

Sitemap: https://attensira.com/sitemap-index.xml

Every AI search, answer and training crawler gets its own group and is allowed everywhere except the API. Three crawlers are shut out completely: Common Crawl's CCBot and two SEO-tool crawlers. The last line points crawlers at the sitemap. Naming each crawler, rather than relying on *, means a later rule for one of them cannot accidentally change the rest.

A file says who may come; only the edge says who came. Attensira counts crawler requests in edge middleware, because crawlers run no JavaScript, and labels each by the user agent it claimed. Any client can claim any name, so our own counts, 143,601 AI-crawler requests in the 30 days to 4 October 2026, are directional, not verified.

How do you allow AI search but block AI training?

Write separate groups for each operator's crawlers. OpenAI documents GPTBot for training, OAI-SearchBot for surfacing sites in ChatGPT search and ChatGPT-User for fetches a person triggers. Disallowing GPTBot asks OpenAI not to train on the pages; disallowing OAI-SearchBot removes them from ChatGPT search answers. Anthropic, Perplexity and Google split their crawlers the same way, under different names. Google's training opt-out, Google-Extended, is a robots.txt token only; Google's AI Overviews are fed by ordinary Googlebot.

Across the web, sites block training more than search. In Attensira's study of the top 10,000 Tranco domains, run on 3 September 2026 over 8,036 robots.txt files, 14.3% blocked GPTBot at the root, against 5.8% for OAI-SearchBot and 1.9% for Googlebot. Common Crawl's CCBot, a training-data crawler, was the most blocked of all at 15.1%.

What mistakes cost sites their AI citations?

• A blanket block written for training. Disallow: / under a rule that also matches the search crawler removes you from answers as a side effect.
• Blocking a crawler you meant to throttle. Rate problems belong in your server or CDN, not in a disallow.
• Blocking CSS and JavaScript paths that a renderer needs to see the page.
• Expecting an instant effect. Allow a day for any change to reach the operators.
• Assuming robots.txt is the whole story. CDN bot rules and firewalls block crawlers without a line in robots.txt, which is why a file that says "allowed" overstates real access.

How does Attensira help with robots.txt?

Attensira is the service of an SEO agency at the price of a tool, and for robots.txt that means two things. The free robots.txt generator builds a file with a rule for each AI crawler, starting from "allow the ones that can cite you, block the ones that only train". In a workspace, the crew's web engineer reads your crawler logs and your robots rules, and when a rule is costing you answers it proposes the fix as a pull request; robots.txt at the root is one of the files it may edit, and nothing merges without your approval. The AI crawler guide covers each operator's crawlers in full.