meta-webindexer vs meta-externalagent: search quality against training data

The short answer

Meta documents two automated crawlers with overlapping names and non-overlapping jobs. meta-webindexer improves Meta AI search result quality. meta-externalagent crawls for use cases such as training foundation AI models or improving products by indexing content directly.

If your objection is to training, meta-externalagent is the token that answers it. If your concern is how Meta AI represents you in an answer, meta-webindexer is the one that matters. And if you were about to disallow both because they both start with "meta", stop: one of them is the crawler that keeps Meta AI's picture of your company current.

The scope problem

The two entries are not symmetrical, and the asymmetry is the most useful thing on this page.

meta-webindexer has one documented job. It navigates the web to improve Meta AI search result quality for users, and Meta says it analyses online content to enhance the relevance and accuracy of Meta AI. That is a narrow, checkable statement, and a publisher can reason about it cleanly: allow it and Meta AI's answers about you are built on current content; block it and they are built on whatever else Meta has.

meta-externalagent has two documented jobs under one token — training foundation AI models, and improving products by indexing content directly. Those are different bargains. The first is the one publishers write policies about. The second is closer to what meta-webindexer does, and a publisher who is comfortable being indexed but not trained on has no way to say so. The robots.txt line is all-or-nothing.

This is the practical shape of most crawler frustration, and it is worth naming precisely rather than complaining about vaguely. The complaint is not that Meta crawls. It is that the granularity of the control does not match the granularity of the objection, so a publisher expressing a narrow position has to overshoot. Compare Mistral, which names its indexing crawler and its training crawler separately and states that the first does not feed the second: same industry, same year, a control surface that fits the actual question.

What you can verify, and what you cannot

Meta's crawler documentation publishes user-agent strings and no IP-based verification guidance. That is the significant gap here, and it puts Meta behind the peer group: OpenAI publishes a per-crawler address file, Anthropic publishes one covering its crawlers, Mistral publishes one for its indexing agent, Amazon publishes one per bot.

The consequence is concrete. A user-agent string is a header, and anyone can type a header. A request identifying as meta-externalagent/1.1 might be Meta, or it might be a scraper that discovered a Meta-branded user agent draws fewer blocks and fewer challenges than an honest one. Without a published address list there is no second signal to check it against.

So the reasoning you can actually support is limited. If you disallow meta-externalagent and continue seeing it in your logs, the correct conclusion is not "Meta ignores robots.txt." It is "somebody claiming to be Meta is fetching this, and I cannot tell who." Treat unverifiable branded traffic as a capacity problem — rate-limit it at the edge, watch what it takes — and do not build a compliance narrative on top of a forgeable string.

The third agent changes the reading

Meta documents a third crawler, meta-externalfetcher, and says it may bypass robots.txt rules because user-initiated requests drive its functionality.

That disclosure is genuinely creditable. Operators who quietly ignore robots.txt for user-triggered fetches are common; operators who write it down are not. It is also, unavoidably, information about how much a robots.txt directive achieves at this operator overall. Two of Meta's three agents honour the file. The third does not, and it is the one that fetches a link because a person asked for it.

The practical reading is that a robots.txt block at Meta is a block on systematic crawling, not on all Meta-originated access to your pages. If your reason for blocking is bandwidth, that distinction barely matters — user-triggered fetches are one page at a time. If your reason is that you do not want Meta's systems touching your content at all, robots.txt does not deliver that, and no amount of tightening the file will.

The robots.txt lines

Refuse the training-and-indexing agent, keep Meta AI search quality current. This is the configuration most companies who have thought about it want:

User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Allow: /

Refuse both systematic crawlers, accepting that Meta AI's picture of you goes stale and that the user-triggered fetcher is documented as possibly ignoring this file anyway:

User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Disallow: /

And the explicit allow, worth committing because it turns an accident into a decision the next maintainer can read:

User-agent: meta-externalagent
Allow: /

User-agent: meta-webindexer
Allow: /

Two mechanics that cause more harm than any policy. Robots.txt matches on the token, so the User-agent: line takes meta-webindexer and not the whole string with the version and the documentation link appended. And robots.txt is per host: the apex domain, the docs subdomain and the help centre each serve their own file. The documentation host is usually the most citable and the least reviewed.

The version trap, in an unusually sharp form

Meta's published short-form user agents are meta-webindexer/1.1 and meta-externalagent/1.1 — the version is part of the string as published, not an optional suffix. That makes copying it into a log filter the natural thing to do, and it makes the resulting filter version-pinned by default.

When Meta increments to 1.2, every dashboard and firewall rule built on the published string stops matching, silently. The failure looks exactly like a crawler that lost interest in your site, which is the most misleading possible presentation of a monitoring bug. Match on the bare token, meta-webindexer, and let the version fall wherever it falls.

Why Meta AI is worth caring about even if you do not use it

There is a temptation among technical audiences to skip Meta entirely. The assistant is not the one your engineers open, it does not show up in developer surveys, and the crawler traffic is easy to dismiss as somebody else's problem.

That reasoning holds only if your buyers look like your engineers. Meta AI is embedded in messaging and social products with an audience measured in a large fraction of the connected world, and a great many of the people who will eventually evaluate your product are inside those apps rather than inside a terminal. For a company selling to small businesses, to marketers, to operations teams or to consumers, the assistant that answers inside a messaging app is not a niche surface — it is the default one.

The asymmetry of the decision follows from that. Allowing meta-webindexer costs you some crawl bandwidth and gives Meta AI current information about what you sell. Blocking it saves the bandwidth and leaves the assistant working from whatever it already has, which for most companies means an older, thinner and less flattering picture. There is no version of that trade where blocking a search-quality crawler improves how you are described.

The training decision is genuinely a decision, and reasonable people land on both sides of it. The search-quality decision usually is not, and it is worth separating the two rather than letting a policy discussion about training quietly settle a distribution question nobody asked about.

A note on auditing this across hosts

Because meta-externalagent bundles two use cases, and because neither agent can be verified by address, the highest-value work here is not choosing the directive. It is confirming the directive is the same everywhere you serve content.

Most companies discover, when they finally look, that their apex domain has a considered robots file and their documentation host has whatever the framework shipped with. Sometimes that default is permissive, in which case a carefully-argued training block is not in effect on the host with the most content. Sometimes it is restrictive, in which case a company that believes it is opted in has been invisible to Meta AI for as long as the docs site has existed. Both outcomes are common, both are silent, and both are found in about ten minutes by fetching every robots file you serve and reading them side by side.

What neither directive controls

Neither is retroactive: content already collected is already collected, and a robots commit today does not reach back into a trained model. Neither reaches copies of your content hosted by somebody else under their robots file. Neither governs what a person pastes into a chat window. And neither is a licence — a robots directive is a machine-readable preference to a well-behaved client, not a contract.

It is also worth re-reading Meta's crawler page on a cadence. It has grown as agents were added, and a robots file written to be exhaustive against last year's roster is merely partial today. The failure is silent, which is what makes a calendar reminder the only reliable defence.

Confirming the change landed

A robots.txt edit is an intention. Your access log is the outcome, and with no address list to check against, logs are the only evidence available for these two agents. The question after any change is whether the agent you disallowed stopped fetching and the one you allowed kept fetching — per URL, since a path-scoped rule behaves nothing like a host-scoped one.

Attensira's crawler logs record which AI agent fetched which URL and when. To read your current rules before changing them, the robots.txt generator and the bot access score will tell you what your file permits today.

Related comparisons

For meta-externalagent against the training crawler it is most often compared with, read meta-externalagent vs GPTBot. For the same split done with job-named tokens, see MistralAI-Index vs MistralAI-Training. And for the two largest assistant vendors' search crawlers, see Claude-SearchBot vs OAI-SearchBot and GPTBot vs OAI-SearchBot.