For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Free tool

robots.txt generator for AI crawlers

The AI robots.txt generator builds a robots.txt with a rule for each AI crawler: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest. It starts by allowing the crawlers that can put you in an answer and blocking the ones with no documented answer use; change any of them, then copy the file. Free, no sign-up.

Start freeBook a demo

7 days free · card required

Or use the free tool below

Build your file

In your browser · Free, no sign-up
Which AI crawlers may fetch your site?

Every crawler below is documented by its operator, with what the operator says it is for. The defaults allow the crawlers that can put your pages in an answer and block the ones with no documented answer use. Each description links to the operator's own page, read on 4 October 2026. Change any of them and the generated file updates as you go.

13 of 20 allowed. Ticked means Allow: /.

13 of 20 allowed · see the file

robots.txt

# robots.txt built with https://attensira.com/tools/robots-generator

# OpenAI, no answer use documented
User-agent: GPTBot
Disallow: /

# OpenAI, can put you in an answer
User-agent: OAI-SearchBot
Allow: /

# OpenAI, can put you in an answer
User-agent: ChatGPT-User
Allow: /

# Anthropic, no answer use documented
User-agent: ClaudeBot
Disallow: /

# Anthropic, can put you in an answer
User-agent: Claude-SearchBot
Allow: /

# Anthropic, can put you in an answer
User-agent: Claude-User
Allow: /

# Perplexity, can put you in an answer
User-agent: PerplexityBot
Allow: /

# Perplexity, can put you in an answer
User-agent: Perplexity-User
Allow: /

# Google, can put you in an answer
User-agent: Google-Extended
Allow: /

# Google, can put you in an answer
User-agent: Googlebot
Allow: /

# Microsoft, can put you in an answer
User-agent: Bingbot
Allow: /

# Apple, can put you in an answer
User-agent: Applebot
Allow: /

# Apple, no answer use documented
User-agent: Applebot-Extended
Disallow: /

# Meta, no answer use documented
User-agent: meta-externalagent
Disallow: /

# Meta, can put you in an answer
User-agent: meta-webindexer
Allow: /

# Amazon, no answer use documented
User-agent: Amazonbot
Disallow: /

# Amazon, can put you in an answer
User-agent: Amzn-SearchBot
Allow: /

# Mistral AI, can put you in an answer
User-agent: MistralAI-Index
Allow: /

# ByteDance, no answer use documented
User-agent: Bytespider
Disallow: /

# Common Crawl, no answer use documented
User-agent: CCBot
Disallow: /

# Every other crawler
User-agent: *
Allow: /

Save it as robots.txt at the root of your domain, then open /robots.txt in a browser to check it is served as plain text.

Share
attensira.com/robots.txt3 of 34 groups
  1. User-agent: GPTBotAllow: /Disallow: /api/Trainingthe default blocks it
  2. 18 more groups, PerplexityBot among them
  3. User-agent: Google-ExtendedAllow: /Disallow: /api/Groundingsame as the default
  4. 10 more groups
  5. User-agent: CCBotDisallow: /Trainingsame as the default
  6. AhrefsBot, SemrushBot and *
  7. Sitemap: https://attensira.com/sitemap-index.xmlSitemapdeclared
The live file, read directly, n=1 run ·

What goes in the file?

  1. One group per documented crawler

    Each crawler whose operator documents its token gets one Allow: / or Disallow: /, which you choose.

  2. A comment on every group

    One line: the operator, and whether the crawler can put you in an answer, so every group in the file can be explained.

  3. The catch-all, your sitemap and your rules

    User-agent: * allows everything else; your Sitemap: line and your own rules go at the end.

What does our own robots.txt allow?

Our live attensira.com/robots.txt, read on 4 October 2026, against what this generator writes by default.

We allow GPTBot and ClaudeBot, which OpenAI and Anthropic document as training crawlers. This generator's defaults would block both, because training does not put you in today's answer. Google-Extended is allowed in our file and by the default, because Google documents it as governing grounding in Gemini Apps too. Our file still blocks CCBot, so it is a choice, not a blanket allow.

That is the point of the toggles. The defaults keep every crawler that can put you in an answer and drop the ones with no documented answer use; the right file is the one whose every line you can explain.

The file is the excerpt at the top of this page. Read on 4 Oct 2026.

What does the generator write?

One group per crawler, for the 20 crawlers whose operators document their user-agent token: 13 that can put a page in an answer (search indexes, on-demand fetchers and grounding controls) and 7 with no documented answer use. Each group is a single Allow: / or Disallow: /, with a one-line comment naming the operator and whether the crawler can put you in an answer. A catch-all User-agent: * group allows everything else, and your sitemap line and your own rules go at the end.

Nothing is invented. A token that no operator documents is not on the list, because a rule for a crawler that does not exist does nothing and makes the file harder to read. Every description in the list above links to the operator's own page, all read on 4 October 2026.

Which crawlers should you allow?

The ones that can put you in an answer, almost always. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes your pages from the index those assistants search, so they can no longer cite you. The same holds for meta-webindexer (Meta AI), MistralAI-Index (Mistral) and Amzn-SearchBot (Amazon's search experiences). The on-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because a person asked for it.

The training crawlers are a real choice. Blocking GPTBot, ClaudeBot or Amazonbot keeps your pages out of future training sets; it does not remove you from today's answers, because each of those operators documents a separate search token. Some publishers block training to protect their content. Others allow it so future models know them. The default here blocks training; our own site allows it, and the example above shows the file.

Google-Extended is not a training-only switch. Google documents it as governing both Gemini training and grounding in Gemini Apps and Vertex AI, so blocking it can keep you out of those answers. It does not affect Google Search or AI Overviews. The default allows it.

robots.txt is a request, not a lock. Documented crawlers honor it; anything that ignores it has to be stopped at your server or CDN.

Who keeps the file right after today?

Inside Attensira, a robots change reaches your repository as a pull request for you to review. Merging it is your call. GitHub delivery.

Pull requestyour-site

robots.txt: one rule per documented AI crawler

attensirawaits for your review

public/robots.txt

  1. unchanged: # OpenAI: the search index behind ChatGPT search
  2. unchanged: User-agent: OAI-SearchBot
  3. unchanged: Allow: /
  4. added: # Anthropic: training crawler
  5. added: User-agent: ClaudeBot
  6. added: Disallow: /
Web engineer agentWeb engineer · would hand you a pull request like thisExample · the kind of change this finding calls for, not one that shipped

How do you put the file live?

  1. Copy the file and save it as robots.txt at the root of your domain: https://example.com/robots.txt. One file per host; a subdomain needs its own.
  2. Open it in a browser. It must load as plain text with a 200 status. A 404 means every crawler is allowed everywhere; an HTML page in its place means the file was never deployed.
  3. Run the crawler URL inspector on a page you care about, to confirm each crawler gets the verdict you meant and a real response.
  4. Merge, do not overwrite. If your current file carries rules for paths (an admin area, a staging folder), keep them in the “your own rules” box.

Questions people ask about AI crawlers and robots.txt

If I block GPTBot, will ChatGPT stop citing me?

Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one behind ChatGPT search. Blocking GPTBot keeps your pages out of training; blocking OAI-SearchBot is what removes them from search answers.

What is Google-Extended?

A control token, not a separate crawler. Googlebot does the fetching; a Google-Extended rule tells Google whether those pages may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Blocking it does not affect Google Search, according to Google's crawler documentation (read 4 October 2026).

Why is there no rule for Claude-Web or anthropic-ai?

Because Anthropic does not document them as current tokens. The generator only writes rules for crawlers whose operators document them, so the file does not fill up with rules for crawlers that do not exist.

How long until a change takes effect?

Crawlers re-read robots.txt on their own schedule. Google says it generally caches the file for up to 24 hours, and longer when it cannot refresh it; see how Google interprets robots.txt (read 4 October 2026).

Does this tool store anything?

No. The file is built in your browser as you tick the boxes. Nothing is sent to us.

This writes the file. Attensira's crew keeps it right.

Start freeBook a demo

7 days free · card required