For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Crawler directory

The AI crawler and bot directory

The Attensira crawler directory has one page for every AI crawler, fetcher and agent whose operator documents it, built only from that documentation. Each page gives the robots.txt token, the user-agent strings, the directive to copy and how to verify a visit. Sort them by job: training, search indexing, one person's question, or the operator's own product.

Start freeBook a demo

7 days free · card required

crawlers, fetchers and agents documented
55
entries checked to 5 Oct 2026
operators, each from its own documentation
22
publish an IP range file you can verify against
37
of 55 agents
AI crawler requests to attensira.com, 30 days
143,601
labelled by claimed user agent
Analyst agentAnalystBot directory and our AI traffic log, Attensira MCP · to

Which kind of crawler am I looking for?

Each category is its own page, with every agent in it compared side by side.

How many of the most visited sites block each crawler?

From our robots.txt study of the 10,000 most visited domains: the share whose robots.txt disallows each agent at the site root.

Disallowed at the rootn=8,036 robots.txt files
Top 10,000 domains (Tranco), by agent · run
NameDisallowed at the root
CCBot15.1%
GPTBot14.3%
ClaudeBot13.4%
meta-externalagent12.4%
Google-Extended12.3%
PerplexityBot8.2%
ChatGPT-User7.1%
meta-externalfetcher6.2%
OAI-SearchBot5.8%
Perplexity-User5.7%
Claude-SearchBot5.5%
Claude-User5.5%
MistralAI-User5.4%
Google-NotebookLM3.8%
bingbot2.2%
Googlebot1.9%
Web engineer agentWeb engineer · Writes robots rules like theseRobots.txt access study, our own fetch · · n=8,036 robots.txt files

CCBot is the most blocked here: 1,215 of 8,036 sites disallow it at the root. The least blocked, Googlebot, is disallowed by 156.

Google-NotebookLM is now Google-GeminiNotebook. The chart keeps the former token because that is what the sites' robots.txt files name.

A site that never names an agent is counted as unspecified, not as allowing it.

What is each fetch for?

Almost every agent here is doing one of four things, and which one decides what allowing or blocking it costs you. Each token links to its entry.

Search indexing14

Builds the index a search engine or an assistant searches when it answers. Blocking it removes you from the pool a citation is drawn from.

User-triggered fetch12

Fetches one page because a person just did something: asked an assistant, shared a link, or verified a site. The assistants' fetchers are the most direct path from a page to a cited answer.

Training8

Collects text that may enter a training corpus, or is the opt-out token for it. Blocking it removes no page from any assistant's search index or citations, though a model can still name you from what it already learned.

Product operations21

Works for the operator's own product: ad checks, SEO tools, shopping, safety and uptime checks. None of it decides whether an AI answer cites you.

What does a robots.txt that separates training from search look like?

attensira.com's own file, read from the code that serves it: AI search and training crawlers allowed by name, three crawlers blocked, the API kept out of everyone's budget.

Allowed by name
30 AI and search crawlers
Blocked site-wide
CCBot, AhrefsBot, SemrushBot
Kept out of every crawl
/api/

Read it live at attensira.com/robots.txt. Build your own with the robots.txt generator.

robots.txt · attensira.com30 allowed by name · 3 blocked
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /api/

# ...and 28 more AI and search crawlers allowed by name, the same way.

User-agent: CCBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: *
Allow: /
Disallow: /api/
Web engineer agentWeb engineer · Writes robots rules like theseExcerpt of the file this site serves at /robots.txt ·

Which AI crawlers actually fetch a site like ours?

Our own AI traffic log for the last 30 days. One training crawler made nine in ten of the requests, while the search crawlers that put pages in front of answers came far less often. Every row is labelled by the user agent the request claimed; none was verified by IP.

Share of AI crawler requests to attensira.comn=143,601 requests
AI traffic log, by operator · to
NameShare of AI crawler requests to attensira.com
Meta (Meta-ExternalAgent)91.3%
OpenAI (OAI-SearchBot, GPTBot, ChatGPT-User)5.2%
ByteDance (Bytespider)2.4%
Perplexity (PerplexityBot)0.4%
Microsoft (bingbot)0.4%
Google (Googlebot)0.1%
Anthropic (ClaudeBot, Claude-User)0.1%
Analyst agentAnalyst · Reads the traffic logAI traffic log, read over the Attensira MCP · · n=143,601 requests

What else do people ask?

How is this directory built?
An agent gets a page only when its operator publishes documentation naming it and saying what it does. Every user-agent string, robots.txt token and verification method is transcribed from that documentation, with the URL and the date it was read printed on the page.
Should I block AI crawlers?
Decide by what the fetch is for. Blocking a training crawler such as GPTBot or ClaudeBot removes no page from any assistant's search index or citations, though a model can still name you from what it learned in training. Blocking a search crawler such as OAI-SearchBot or PerplexityBot, or a user-triggered fetcher, removes your pages from the pool an assistant cites from.
Which crawlers do I need to allow to appear in ChatGPT, Perplexity and Claude?
Their search crawlers and their user-triggered fetchers: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-SearchBot and Claude-User for Claude. Their training crawlers, GPTBot and ClaudeBot, are separate tokens you can refuse without losing citations.
Can I trust the user agent in my server logs?
No. A user-agent header is free text that any client can send, and AI crawler names are the ones scrapers borrow most. Confirm a request against the operator's published IP ranges or reverse DNS, which each entry here lists, or says the operator publishes none.
How does Attensira count AI crawler visits?
In your own edge middleware, on Growth and Business, because AI crawlers do not execute JavaScript and a browser beacon never sees them. The board reports requests, pages and when each agent was last seen, as directional coverage data: user agents are self-reported, so Attensira does not claim a visit was verified.
Why is a bot I have seen in my logs missing?
Tokens that circulate on third-party bot lists with no traceable operator page are left out on purpose. Naming one in robots.txt costs nothing, but nobody can honestly say what the rule achieves.

Put the crew to work on your brand.

Start freeBook a demo

7 days free · card required