Bots
meta-externalagent: allow or block it?
meta-externalagent is Meta's AI crawler. Meta documents it as crawling for uses such as training foundation models or indexing content directly. It respects robots.txt, with changes taking up to 24 hours. Block it to keep your text out of Meta's training; allow it if you accept that. Meta publishes no address list, so visits cannot be verified.
7 days free · card required
131,105meta-externalagent requests to attensira.com
to 873 pages · Meta #1 of 7 operators
| Operator | Share of requests | Requests |
|---|---|---|
| Meta (this page) | 131,105 | |
| OpenAI | 7,536 | |
| ByteDance | 3,505 | |
| Perplexity | 621 | |
| Microsoft | 541 | |
| Google Search | 189 | |
| Anthropic | 104 |
User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP · · n=143,601 requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block meta-externalagent in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves meta-externalagent today.
# Meta's own documented example, narrowed to a block.User-agent: meta-externalagentDisallow: /User-agent: Meta-ExternalAgentAllow: /Disallow: /api/We name meta-externalagent and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs meta-externalagent, and what is it for?
Meta runs it, and we file it under training. Meta documents it as crawling for uses such as training foundation models or indexing content directly. Meta AI's citation crawler is meta-webindexer, a separate token.
From Meta’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Why is meta-externalagent filed as a training crawler?
Meta's one sentence on this token names two jobs: training foundation AI models, and improving products by indexing content directly.1 Nothing in the documentation lets a site allow the indexing half and decline the training half, so this page records the more consequential of the two.
OpenAI splits the same ground across two tokens: GPTBot for content that may be used in training,7 and OAI-SearchBot for ChatGPT's search features.8 If your rule is "no training, keep citations", the matching configurations are Disallow for GPTBot with OAI-SearchBot allowed, and Disallow for meta-externalagent with meta-webindexer allowed.6
Which Meta tokens does a meta-externalagent block leave alone?
A group naming meta-externalagent reaches that token and no other. Meta names two crawlers that may bypass robots.txt: meta-externalfetcher, because it performs user-requested fetches, and FacebookExternalHit, during security or integrity checks.3 meta-externalagent is not on that list.
All of Meta's tokens share the meta-external prefix, so a substring log filter merges the training crawler with the user-triggered fetcher. Match meta-externalagent as a whole word, case-insensitively. More blocks are in the robots.txt chapter, and our robots.txt generator writes Meta's tokens alongside every other operator's.
How do I verify that a visit really came from meta-externalagent?
You cannot. Meta advises allowlisting "either the user agent strings or the IP addresses (more secure) used by the crawler", then publishes no address list and no reverse-DNS domain for any of its crawlers.5 OpenAI, by contrast, publishes GPTBot's addresses at openai.com/gptbot.json.9
A meta-externalagent count is therefore a count of requests that claimed the token. Label it that way in any log report, including Attensira's AI traffic report. Why claimed and verified traffic differ is covered in verifying a crawler.
Meta publishes no reverse-DNS suffix and no IP range file for meta-externalagent, so a visit can be matched only by its user agent.
ResearcherTranscribed from Meta documentation · How often do sites block meta-externalagent?
Measured by us, not reported by Meta: the robots.txt of the 10,000 highest-ranked domains on the Tranco list for 2026-09-01. A domain that never mentions the token wrote no policy for it.
The method and every row, with the CSV.
- disallow meta-externalagent at the root
- 12.4%
- 994 of 8,036 domains
- allow it by name
- 1.5%
- 117 domains
- never mention it
- 86.2%
- 6,925 domains
What user-agent string does meta-externalagent send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check Meta’s page before relying on the exact text.
How much does meta-externalagent fetch from attensira.com?
Allowing it costs load. From the same read as the traffic frame at the top of this page (Attensira's AI traffic board for attensira.com, 30 days to 4 October 2026, n=143,601 AI crawler requests): 131,105 requests filed as Meta-ExternalAgent across 873 pages, against 7,536 from OpenAI's agents together and 621 from PerplexityBot. Meta alone sent most of the AI crawler requests we received. None of the Meta requests can be verified, because Meta publishes no address list, so the number is claimed traffic. It is still the clearest reason to put a rate limit in front of a token before you allow it.
What does allowing meta-externalagent cost?
Allowing it means your text may enter Meta's training data. Blocking it costs no Meta AI citations, because those come from meta-webindexer and meta-externalfetcher, which take separate rules. Meta says robots.txt changes can take up to 24 hours to apply.
meta-externalagent/1.1Agents filed beside meta-externalagent
Should I allow meta-externalagent in my situation?
Concrete cases, and what each one implies for this agent.
You want out of Meta's foundation-model training but want Meta AI to still cite you.
Disallow meta-externalagent and allow meta-webindexer.
Meta documents meta-webindexer separately, as navigating the web to improve Meta AI search result quality, and says allowing it helps Meta cite and link to your content in Meta AI's responses. meta-externalagent is the token whose documented use cases include training foundation AI models. Two tokens, two decisions.
A tool reports meta-externalagent traffic and you want to confirm it is really Meta.
Accept that you cannot confirm it from operator-published data, and treat the count as claimed rather than verified.
Meta publishes no address list and no reverse-DNS domain for its crawlers. Reporting an unverified count as a verified one is the failure this directory exists to prevent, so the honest label on a Meta row in a log report is "claimed".
You added the disallow rule this morning and Meta is still crawling.
Give it 24 hours before changing anything.
Meta states that crawlers may cache the contents of robots.txt for up to 24 hours and asks for that window before changes take effect. Editing the rule again inside that window produces a second cached generation and makes the behaviour harder to read, not easier.
What else do people ask?
Does meta-externalagent respect robots.txt?
How long does a robots.txt change take to work?
How does meta-externalagent differ from GPTBot?
Is this the same as facebookexternalhit?
Where do these facts come from?
9 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Meta states that the Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
developers.facebook.com · retrieved
“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”
Meta documents that the user-agent string seen in log files for this crawler will be similar to "meta-externalagent/1.1", or the same token followed by a bracketed link to Meta's crawler documentation.
developers.facebook.com · retrieved · changes often, check the source
Meta states that by configuring robots.txt a site can specify how Meta web crawlers interact with it, that blocking is done by adding a disallow for the relevant crawler, and names only Meta-ExternalFetcher and FacebookExternalHit as crawlers that may bypass robots.txt. Meta's own example block is a group naming meta-externalagent.
developers.facebook.com · retrieved
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.”
Meta asks site owners to allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache its contents for up to 24 hours.
developers.facebook.com · retrieved
“Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours.”
Meta's crawler documentation advises adding to an allow list either the user agent strings or the IP addresses used by the crawler, describing the addresses as more secure, but publishes no address list and no reverse-DNS domain for its crawlers.
developers.facebook.com · retrieved
“Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler.”
Meta documents Meta-WebIndexer separately as navigating the web to improve Meta AI search result quality, and states that allowing it in robots.txt helps Meta cite and link to a site's content in Meta AI's responses.
developers.facebook.com · retrieved
“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.”
OpenAI documents GPTBot as the agent used to crawl content that may be used in training its generative AI foundation models.
developers.openai.com · retrieved
“crawl content that may be used in training our generative AI foundation models”
OpenAI documents OAI-SearchBot, a separate robots.txt token from GPTBot, as the agent used to surface websites in search results in ChatGPT's search features.
developers.openai.com · retrieved
“surface websites in search results in ChatGPT's search features”
OpenAI publishes the IP ranges GPTBot fetches from as a machine-readable file at openai.com/gptbot.json, one of four per-agent range files.
developers.openai.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required