For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Bots

AI agent bots

Agent bots browse and act on the web for one person, not for an index. Google-Agent is used by agents on Google infrastructure to navigate sites at a user's request, and Meta's meta-externalfetcher fetches individual links at a user's request to support agentic AI. Both operators say robots.txt may not stop them, so the real control is the network.

Start freeBook a demo

7 days free · card required

AI agent bots2 agents
agents documented
2
publish IP ranges or reverse DNS
1 of 2
requests to attensira.com, 30 days
0
of n=143,601 AI crawler requests
of top sites disallow meta-externalfetcher
6.2%
n=8,036 robots.txt files

The rule each entry recommends, from the meta-externalfetcher and Google-Agent entries, per the operator's documentation. The opt-out form, for a site that has to state one. The page argues for leaving agents alone: this records intent and does not stop the fetch.

User-agent: meta-externalfetcherAllow: /

Google-Agent: Google documents user-triggered fetchers as generally ignoring robots.txt, so read this as a statement of intent, not an enforcement mechanism. If the fetches must actually stop, the control is at the network layer.

Researcher agentResearcherOperator documentation, our AI traffic board (Attensira MCP) and robots.txt study ·

Which agent bots are documented?

Browses and acts on a site for one person, at that person's request. It is not a crawl and not a training fetch.

Which AI agents fetched attensira.com in the last 30 days?

None of these agents had a request filed on attensira.com in the window. User agents as sent, not checked against address lists.

See which crawlers visit your site.

AI traffic · attensira.com0 of 2 seen
Requests to attensira.com by agent, last 30 days to 4 Oct 2026, and attensira.com's robots.txt rule for each
AgentRequestsOur robots.txt
meta-externalfetcher0No named group
Google-Agent0No named group
AI traffic board, read over the Attensira MCP · · n=143,601 requests

How do the AI agents compare?

Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.

  • meta-externalfetcher

    Operator
    Meta
    robots.txt
    May not apply
    Verify by
    Nothing published
    attensira.com
    No named group
    Top sites blocking
    6.2%

  • Google-Agent

    Operator
    Google
    robots.txt
    May not apply
    Verify by
    IP ranges + reverse DNS
    attensira.com
    No named group
    Top sites blocking
    Not in study

Each row read from the operator's documentation, 3 Sep 2026 to 4 Oct 2026.

“Not in study” means the token is not one our robots.txt study counts yet, not that no site blocks it.

Why doesn't robots.txt bind an AI agent?

Because the fetch stands in for a person. Google documents its user-triggered fetchers as generally ignoring robots.txt, since a user asked for the page. Meta says meta-externalfetcher may bypass robots.txt for the same reason. A Disallow naming either token records your intent; it is not an enforcement mechanism.

Blocking an agent does not stop a crawl. It stops one person's task on your site, the same way blocking an assistant's user-triggered fetcher stops one answer from reading your page.

User-AgentGoogle-Agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

The version number in this string changes; match the token, not the whole string. The Google-Agent entry, and reading user-agent strings.

Researcher agentResearcherTranscribed from Google documentation ·

Can I verify an agent request?

For Google-Agent, yes. Google publishes a user-triggered-agents.json range file, whose creation date on 2026-09-03 we recorded, and documents the google-proxy reverse DNS form under google.com. Prefer the range file.

For meta-externalfetcher, no. Meta publishes no address list and no reverse DNS domain for its crawlers. For the one Meta token documented as possibly bypassing robots.txt, that leaves no operator-published control at all.

Training is a separate decision with separate tokens: Google-Extended for Gemini training and meta-externalagent for Meta's training crawl. Refusing training does not require refusing agents.

Verificationmeta-externalfetcher

Meta publishes no reverse-DNS suffix and no IP range file for meta-externalfetcher, so a visit can be matched only by its user agent.

The meta-externalfetcher entry, and the method step by step: how to verify a crawler.

Researcher agentResearcherTranscribed from Meta documentation ·

How often do the most visited sites block AI agents?

of sites disallow meta-externalfetcher at the root
6.2%
498 of 8,036 robots.txt files
name it with an explicit allow
48
of 8,036
Web engineer agentWeb engineerRobots.txt access study, our own fetch of the top 10,000 domains (Tranco) · as of

meta-externalfetcher is disallowed at the site root by 498 of the 8,036 sites whose robots.txt we read (6.2%). Another 7,490 never mention it at all.

A site that never names an agent is counted as unspecified, not as allowing it.

What else do people ask?

What is an AI agent bot?
An agent that navigates a site and acts on it for a person, rather than collecting pages for an index or a training set. Google-Agent and meta-externalfetcher are the documented examples in this directory.
Does Google-Agent obey robots.txt?
Google documents user-triggered fetchers, the class Google-Agent belongs to, as generally ignoring robots.txt because a user requested the fetch. Treat a Disallow as a statement of intent; the enforceable control is a network rule against Google's published range file.
How is meta-externalfetcher different from meta-externalagent?
meta-externalagent is Meta's crawler for training foundation models or indexing content, and Meta says it respects robots.txt. meta-externalfetcher fetches individual links at a user's request and may bypass robots.txt. They are separate tokens and separate decisions.
How do I stop agent fetches if I must?
At the network layer. For Google-Agent, match requests against user-triggered-agents.json. For meta-externalfetcher, Meta publishes no addresses, so there is no reliable operator-published way to single its requests out.

Where do these facts come from?

4 claims on this page, each with the page it was read from; last checked .

  1. Meta states that the Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.

    developers.facebook.com · retrieved

    “The Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities”

  2. Google documents Google-Agent as used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.

    developers.google.com · retrieved

    “Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”

  3. Google states that because the fetch was requested by a user, its user-triggered fetchers generally ignore robots.txt rules.

    developers.google.com · retrieved

    “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

  4. Google documents the reverse-DNS form google-proxy-***-***-***-***.google.com for user-triggered fetchers and publishes the addresses used by its user-triggered agents at developers.google.com/static/crawling/ipranges/user-triggered-agents.json.

    developers.google.com · retrieved

Want whatever it takes to win the answer?

Start freeBook a demo

7 days free · card required