How to write robots.txt rules for AI bots

What does a real AI-aware robots.txt look like?

Here is ours, abridged. This is what attensira.com served on 4 October 2026.

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /api/

# ...one group each for ClaudeBot, Claude-User, Claude-SearchBot,
# PerplexityBot, Perplexity-User, Meta-ExternalAgent, Googlebot,
# Google-Extended, Applebot, Applebot-Extended, Bingbot and others...

User-agent: CCBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://attensira.com/sitemap-index.xml
License: https://attensira.com/license.xml

Four decisions are visible in it.

Every agent we care about has its own group. Even where the rules are identical to the wildcard group, naming the agent makes the decision explicit, and it means the decision survives if someone later tightens the wildcard.

The API is closed to everyone. /api/ serves nothing a crawler should read, so every group disallows it.

Three crawlers are shut out completely: Common Crawl's CCBot, and two SEO tools' crawlers, AhrefsBot and SemrushBot.

Training agents are allowed. GPTBot, ClaudeBot and Meta-ExternalAgent can read the site. For a company that wants models to know its name, we see no reason to opt out. Your decision may differ; the training versus search chapter covers what each choice costs.

It also has weaknesses, which we found while writing this guide. It is long, because each group repeats the same two lines; a group can list several user-agent lines, and ours could be a third of the length. And it has no group for some agents operators now document, such as Mistral's indexing and training agents, so those fall to the wildcard. The rest of this chapter is the rules that make a file like this do what you mean.

How is a robots.txt file structured?

A file is a list of groups. Each group starts with one or more User-agent: lines naming the crawlers it applies to, followed by Allow: and Disallow: rules giving path prefixes.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Here three training agents share one group and are kept out of the whole site, and every other crawler may read everything except /admin/. Lines such as Sitemap: stand outside groups and apply to the whole file.

Which group does a crawler obey?

The one that names it. RFC 9309, the robots.txt standard, says a crawler matches its product token against the user-agent lines case-insensitively and uses the group that matches. Only if no group names it does it use the * group.

This is the rule most often misunderstood. If you add a group for GPTBot, the rules under * stop applying to GPTBot entirely. A site that disallowed /private/ under * and then added User-agent: GPTBot with Allow: / has just let GPTBot into /private/.

If several groups name the same crawler, their rules are combined into one, the RFC says. Two groups for GPTBot in different places in the file act as one group. That is a common result of files edited by several people over time, and a good reason to keep one group per agent.

Which rule wins inside a group?

The most specific one: the rule whose path matches the URL with the most characters. RFC 9309 calls it the match with the most octets. So in:

User-agent: OAI-SearchBot
Disallow: /docs/
Allow: /docs/public/

/docs/public/guide is allowed, because /docs/public/ is the longer match, and /docs/internal/notes is disallowed. Order in the file does not matter.

What do status codes do to your robots.txt?

More than most teams realise. RFC 9309 sets two very different outcomes:

So a robots.txt that times out under load, or errors behind a misconfigured CDN, can block every compliant AI crawler, search agents included, until it recovers. Serve it as a static file, cache it at the edge, and monitor it like any page that matters. RFC 9309 also says crawlers must parse at least 500 KiB of it, so an enormous file risks being cut off.

A related trap sits outside robots.txt entirely. Bot-protection and firewall rules at your CDN can challenge or block crawlers on every page while robots.txt says they are welcome. The crawler then sees an error or a challenge page instead of your content, and robots.txt never comes into it. If an allowed agent never appears in your logs, check the firewall before the file. Perplexity notes that firewall configuration changes can take time to propagate, as robots.txt changes do.

How long do changes take to apply?

About a day for most operators, and longer for some.

• OpenAI says it can take about 24 hours for its systems to adjust.
• Perplexity and Meta say changes can take up to 24 hours.
• RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.
• Amazon says Amazonbot may use a cached robots.txt from the previous 30 days.

After fixing an accidental block, wait at least a day, then check your server logs for the agent's return. Pages dropped from an AI product's index while the block was live come back only when the agent recrawls them, which you cannot force.

Which directives beyond Allow and Disallow work?

Sitemap is widely supported and worth including; it helps crawlers find pages your internal links do not reach.

Crawl-delay is not in RFC 9309. Anthropic says its bots support it; Amazon says Amazonbot does not. Treat it as a request some crawlers honour. If load is the real problem, rate-limit by agent on the server.

Tokens without crawlers, Google-Extended and Applebot-Extended, work like any other group but are read by the main Google and Apple crawlers as a training opt-out. They never fetch anything themselves.

New lines such as License: carry licensing terms for content, as ours does. Crawlers that do not recognise a line ignore it, so it does no harm.

What can robots.txt not do?

Three things, and each one trips people up.

It is not access control. RFC 9309 says its rules are not a form of access authorization. Compliant crawlers honour them; others need not. Anything private belongs behind authentication.

It does not bind every user-triggered fetcher. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.

It does not remove what was already collected. A new rule applies to future crawls, not to pages already in an index or a training set.

Which robots.txt mistakes cost AI visibility?

Five, roughly in order of how often they appear in files we read.

A wildcard block written to stop training. User-agent: * with Disallow: / keeps out GPTBot and ClaudeBot, and also OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot. The site disappears from every AI search product that honours robots.txt. If the goal is a training opt-out, name the training agents and leave the wildcard open.

The right intent under the wrong token. A rule meant for training addressed to OAI-SearchBot instead of GPTBot opts the site out of ChatGPT search and leaves training untouched. Check each token against the operator's description of it, not against its name.

A named group that forgets the wildcard's rules. Adding a group for one agent drops every rule under * for that agent. Private paths disallowed under * must be repeated in each named group.

A file that errors. A robots.txt behind a misconfigured redirect, a login wall or a slow origin can return an error to crawlers while looking fine in a browser. Under the standard, an error means "disallow everything".

Stale tokens and missing new ones. Operators add and retire agents. A file written a year ago may name agents that no longer exist and miss ones that do, as ours did.

How do you write rules for a large site?

By section, with the patterns the standard supports. RFC 9309 defines two special characters in paths: * matches any sequence of characters, and $ anchors the end of a URL. So Disallow: /*?sort= keeps crawlers out of sorted duplicates of every listing page, and Disallow: /*.pdf$ keeps them out of PDF files without touching pages whose names contain "pdf".

Decide per section first, then write the rules. Typical choices: product, pricing, documentation and articles open to search and user agents; account, cart, checkout and internal search results closed to everyone; anything licensed or paid decided separately for training agents. Then keep the file in version control, with a comment above each group saying who decided it and why. A robots.txt nobody can explain is one nobody dares to fix.

How do you check your rules before you ship them?

1. Read the file as each agent. For each token you named, list the groups that apply and the rule that wins for your key paths.
2. Test a few URLs per agent. The crawler URL inspector shows how each agent's rules apply to a given URL, and the robots.txt generator writes a file with each agent named.
3. Check the status code of /robots.txt from outside your network, and its response time.
4. Watch the logs for a few days after shipping. An agent you meant to allow that disappears is the fastest sign of a mistake.

The next chapter, on verifying a crawler, covers making sure the agents in your logs are who they say they are.