Bots
meta-externalfetcher: the AI fetcher
meta-externalfetcher is the Meta agent that fetches a specific link because a person asked Meta AI something, including AI agents completing tasks on websites. Meta says it may bypass robots.txt because the user requested the fetch. Meta publishes no address list, so it cannot be verified or blocked precisely. Leave it allowed, and protect sensitive flows server-side.
7 days free · card required
0requests filed as meta-externalfetcher · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| meta-externalfetcher (this page) | 0 | Wildcard |
| Google-Agent | 0 | Wildcard |
None of the 2 had a request filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block meta-externalfetcher in robots.txt?
Paste this into the robots.txt at the root of your domain. The operator says the agent may not read robots.txt, so the group records your preference; enforce anything sensitive on the server. Beside it, the group attensira.com serves meta-externalfetcher today.
# Leave the fetcher allowed. Meta documents it as possibly# bypassing robots.txt, so a Disallow would record a# preference, not enforce one.User-agent: meta-externalfetcherAllow: /# No group names meta-externalfetcher, so it reads the# wildcard group.User-agent: *Allow: /Disallow: /api/We do not name meta-externalfetcher. It falls through to the wildcard group, which allows everything except /api/. Its operator states robots.txt may not apply to it, so this group is a preference at most.
Read the whole file, or write yours with the robots.txt generator.
Who runs meta-externalfetcher, and what is it for?
Meta runs it, and we file it under user-triggered fetch. Fetches one link because a person asked Meta AI about it or an agent is completing a task for them. Meta says it may bypass robots.txt for that reason.
From Meta’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Does meta-externalfetcher obey robots.txt?
Meta says it "may bypass robots.txt rules", so a group naming it records a preference rather than enforcing one; if you write one anyway, allow 24 hours before reading behaviour, since Meta caches the file that long.5 Which agents to allow, and what each costs, is in what to allow and what to block.
What controls still work on an agent that may skip robots.txt?
Meta's description includes helping AI navigate websites to complete tasks for users. A fetcher reads one page; an agent completing a task follows links, submits forms and comes back. With the protocol layer disclaimed and no address list to block on, what is left is server-side and does not depend on who is asking:
- Rate-limit by behaviour, not by user-agent string.
- Require authentication on anything that changes state. An agent working through a task will follow whatever is reachable.
- Assume any unauthenticated URL may be read by an automated client that did not consult your robots.txt.
How does meta-externalfetcher differ from Perplexity-User?
Both are fetchers a person sets off, and both operators say robots.txt may not hold: Perplexity says Perplexity-User "generally ignores" it. The differences are in what you get and what you can check. Perplexity documents a link to your page in the answer and publishes the addresses Perplexity-User fetches from. Meta makes no linking commitment for this token and publishes no addresses.
The two need the same server-side posture, but only Perplexity's traffic can be counted with confidence. Meta's citation commitment sits with meta-webindexer, which is the token to allow deliberately if you want Meta AI to link to you.
How do I verify that a visit really came from meta-externalfetcher?
You cannot. Meta advises allowlisting "either the user agent strings or the IP addresses (more secure) used by the crawler", then publishes no addresses and no reverse-DNS domain. For meta-externalagent that only means unconfirmed counts, because robots.txt still blocks it. Here it removes the last control, so treat every count as claimed traffic. Attensira's AI traffic report labels these requests by the agent they claim to be. Why a header proves nothing is in verifying a crawler.
Meta publishes no reverse-DNS suffix and no IP range file for meta-externalfetcher, so a visit can be matched only by its user agent.
ResearcherTranscribed from Meta documentation · How often do sites block meta-externalfetcher?
Measured by us, not reported by Meta: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow meta-externalfetcher at the root
- 6.2%
- 498 of 8,036 domains
- allow it by name
- 0.6%
- 48 domains
- never mention it
- 93.2%
- 7,490 domains
What user-agent string does meta-externalfetcher send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check Meta’s page before relying on the exact text.
What does allowing meta-externalfetcher cost?
Allowing it costs one fetch per user request and is how a page someone asks Meta AI about gets read. Blocking it in robots.txt may not stop it, because Meta says it can bypass robots.txt for user-requested fetches. Protect logins and checkout flows on the server instead.
meta-Should I allow meta-externalfetcher in my situation?
Concrete cases, and what each one implies for this agent.
You disallowed the token and the fetches continue.
Stop treating robots.txt as the control here; move to server-side measures that do not depend on identifying the client.
Meta documents this crawler as possibly bypassing robots.txt because it performs user-requested fetches. With no published address list, there is nothing to allowlist or block precisely either. Rate limits, authentication on sensitive flows, and not putting anything behind an unauthenticated URL you would mind an agent reading are the controls that still work.
An agent appears to be walking a multi-step flow on your site under this token.
Check that every step that mutates state requires authentication, then decide whether to rate-limit.
Meta's own description includes helping AI navigate websites to complete tasks for users, so an agent traversing a flow is the documented behaviour rather than an anomaly. The risk is not the reading; it is any state-changing endpoint reachable without a login.
Your log tool reports one combined figure for every token beginning meta-external.
Split them. The prefix hides three different jobs.
meta-externalagent is training and indexing, meta-externalads is advertising, and meta-externalfetcher is user-triggered and documented as possibly bypassing robots.txt. A combined row averages a training crawl, an ad crawl and a live agent fetch into a number that answers no question you would actually ask.
What else do people ask?
Does meta-externalfetcher crawl my site on its own?
Is this an AI agent browsing my site?
Should I treat this traffic as a visitor?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Meta states that the Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
developers.facebook.com · retrieved
“The Meta-ExternalFetcher crawler fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities”
Meta states that the Meta-ExternalFetcher crawler may bypass robots.txt rules because it performs fetches that were requested by the user, and names it alongside FacebookExternalHit as an exception in its robots.txt section.
developers.facebook.com · retrieved
“Accordingly, this crawler may bypass robots.txt rules.”
Meta documents that the user-agent string seen in log files for this crawler will be similar to "meta-externalfetcher/1.1", or the same token followed by a bracketed link to Meta's crawler documentation, and documents separate meta-externalagent and meta-externalads tokens sharing the same prefix.
developers.facebook.com · retrieved · changes often, check the source
Meta's crawler documentation advises adding to an allow list either the user agent strings or the IP addresses used by the crawler, describing the addresses as more secure, but publishes no address list and no reverse-DNS domain for its crawlers.
developers.facebook.com · retrieved
“Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler.”
Meta asks site owners to allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache its contents for up to 24 hours.
developers.facebook.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required