AI Robots.txt Generator
Generate robots.txt with AI crawler controls - allow or block GPTBot, Claude, Google AI
Control AI Access: Select which AI crawlers can access your site. Allowing AI crawlers helps your content appear in ChatGPT, Claude, and other AI search results. Upload the generated robots.txt to your site's root directory.
AI Crawlers
|
GPTBot
AllowedOpenAI's ChatGPT crawler
GPTBotChatGPT-User
AllowedChatGPT browsing feature
ChatGPT-UserClaude-Web
AllowedAnthropic's Claude crawler
Claude-WebClaudeBot
AllowedAnthropic's Claude bot
ClaudeBotGoogle-Extended
AllowedGoogle's AI training crawler
Google-ExtendedPerplexityBot
AllowedPerplexity AI crawler
PerplexityBotApplebot-Extended
AllowedApple Intelligence crawler
Applebot-ExtendedAmazonbot
AllowedAmazon's AI crawler
AmazonbotDiffbot
AllowedDiffbot AI crawler
DiffbotFacebookBot
AllowedMeta's AI crawler
FacebookBotImagesiftBot
AllowedAI image indexing bot
ImagesiftBotOmgilibot
AllowedOmgili AI crawler
OmgilibotBytespider
BlockedByteDance (TikTok) crawler
BytespiderStatus:
✓ 12 Allowed✗ 1 Blocked
Additional Settings
Optional: Link to your XML sitemap
Optional: Add custom robots.txt directives
Generated robots.txt
Copy and paste to your site
# robots.txt generated by Attensira # https://attensira.com/tools/robots-generator # Allowed AI Crawlers # GPTBot - OpenAI's ChatGPT crawler User-agent: GPTBot Allow: / # ChatGPT-User - ChatGPT browsing feature User-agent: ChatGPT-User Allow: / # Claude-Web - Anthropic's Claude crawler User-agent: Claude-Web Allow: / # ClaudeBot - Anthropic's Claude bot User-agent: ClaudeBot Allow: / # Google-Extended - Google's AI training crawler User-agent: Google-Extended Allow: / # PerplexityBot - Perplexity AI crawler User-agent: PerplexityBot Allow: / # Applebot-Extended - Apple Intelligence crawler User-agent: Applebot-Extended Allow: / # Amazonbot - Amazon's AI crawler User-agent: Amazonbot Allow: / # Diffbot - Diffbot AI crawler User-agent: Diffbot Allow: / # FacebookBot - Meta's AI crawler User-agent: FacebookBot Allow: / # ImagesiftBot - AI image indexing bot User-agent: ImagesiftBot Allow: / # Omgilibot - Omgili AI crawler User-agent: Omgilibot Allow: / # Blocked AI Crawlers # Bytespider - ByteDance (TikTok) crawler User-agent: Bytespider Disallow: / # Standard Search Engines User-agent: * Allow: /
How to Use:
- 1.Click the "Copy" button above to copy the robots.txt content
- 2.Create a file named "robots.txt" in your site's root directory
- 3.Paste the content and save the file
- 4.Upload to your site (e.g., https://yoursite.com/robots.txt)
- 5.Verify it's accessible by visiting yoursite.com/robots.txt
Where to go next
- Check what your live robots.txt already says to each AI crawlerBefore you replace the file, see which crawlers the current one allows and which line decides it.
- See which AI crawlers actually fetched your pagesrobots.txt states your intent. Crawler logs are the evidence of what the bots did with it.
- Validate the sitemap you list in the Sitemap: lineA robots.txt pointing at a malformed sitemap wastes the one directive crawlers reliably follow.
- Build an llms.txt to sit alongside itrobots.txt permits; llms.txt curates. The two answer different questions and both are worth publishing.
- Glossary of AI crawler and AI search termsDefinitions for user agent, crawl budget, AI crawler and the rest of the vocabulary used here.
- Track whether assistants name your brand once crawling is allowedPermitting the crawlers is step one. Prompt tracking measures whether it changed any answers.