Bots
meta-webindexer: the AI search crawler
meta-webindexer is Meta's search crawler for Meta AI. Meta documents it as improving Meta AI search results and says allowing it helps Meta cite and link your content in Meta AI's answers. It is separate from meta-externalagent, the training-side token. It respects robots.txt. Allow it if you want Meta AI citations; Meta publishes no way to verify visits.
7 days free · card required
0requests filed as meta-webindexer · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| meta-webindexer (this page) | 0 | Wildcard |
| OAI-SearchBot | ≈6,480 | Allowed |
| PerplexityBot | 621 | Allowed |
| Amzn-SearchBot | 0 | Parent's rule |
| Claude-SearchBot | 0 | Allowed |
| MistralAI-Index | 0 | Wildcard |
| YouBot | 0 | Allowed |
2 of 7 had requests filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block meta-webindexer in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves meta-webindexer today.
# Stay citable in Meta AI while keeping out of Meta's# training crawl.User-agent: meta-webindexerAllow: / User-agent: meta-externalagentDisallow: /# No group names meta-webindexer, so it reads the wildcard# group.User-agent: *Allow: /Disallow: /api/We do not name meta-webindexer. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs meta-webindexer, and what is it for?
Meta runs it, and we file it under search indexing. Meta documents it as improving Meta AI search results and says allowing it helps Meta cite and link your content. Training sits with meta-externalagent, a separate token.
From Meta’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
Which Meta crawlers sit on the AI answer path?
Meta documents five crawlers. Three concern AI answers:
- meta-webindexer: Meta AI search quality and citation.
- meta-externalagent: training foundation AI models or indexing content directly.5
- meta-externalfetcher: individual links at a user's request, documented as possibly bypassing robots.txt.
The other two, facebookexternalhit and meta-externalads, handle link previews and advertising.
How do I keep Meta AI citations while refusing training?
Allow meta-webindexer and disallow meta-externalagent, as in the block above. The split exists only because Meta gives the two jobs separate tokens. User-agent: Meta covers neither, because robots.txt matches product tokens, not company names.
Check allow-lists too. If your robots.txt blocks * and allows a list of named agents, a list that names meta-externalagent but not meta-webindexer has refused the one Meta crawler that Meta ties to citations. The full set of blocks is in the robots.txt chapter, and our generator writes them per token.
How do I verify that a visit really came from meta-webindexer?
You cannot. Meta advises allowlisting "either the user agent strings or the IP addresses (more secure) used by the crawler", then publishes neither an address list nor a reverse-DNS domain. You can allow the token and count requests that claim it, but nothing confirms any one of them was Meta.
That is no reason to block it, since the citation upside is Meta's own statement. It is a reason to keep the count in a separate column from traffic you can prove. Why AI tokens are often spoofed is covered in verifying a crawler.
Meta publishes no reverse-DNS suffix and no IP range file for meta-webindexer, so a visit can be matched only by its user agent.
ResearcherTranscribed from Meta documentation · What user-agent string does meta-webindexer send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check Meta’s page before relying on the exact text.
What does allowing meta-webindexer cost?
Allowing it costs crawl load and is what Meta says helps it cite and link your content in Meta AI's answers. Blocking it gives up those citations. It does not decide training: that is meta-externalagent, which you can block separately.
meta-webindexer/1.1Should I allow meta-webindexer in my situation?
Concrete cases, and what each one implies for this agent.
A blanket "block Meta" rule went into your robots.txt after a news cycle about AI training.
Narrow it — allow meta-webindexer, keep the disallow on meta-externalagent.
Meta documents these as separate tokens with separate jobs, and says allowing the indexer helps Meta cite and link to your content. A blanket block removes the citation path while leaving the training question exactly where it was, because the training use case belongs to the other token.
You allowed the token and want to know whether Meta AI is actually reading you.
Count it, but label the count claimed rather than verified.
Meta publishes no way to confirm a request came from Meta. Every other operator in this cluster publishes addresses, reverse DNS, or signatures. Reporting an unverifiable token as verified traffic is precisely the sourcing failure this directory exists to prevent.
Your rules for the two Meta tokens contradict each other because one lives under a wildcard group.
Give each Meta token its own explicit group.
Under RFC 9309 a token that has its own group obeys only that group, and a token without one falls back to the wildcard group. Name one Meta token and not the other, and the two get different rules without anyone deciding so. With Meta this bites twice, because the tokens share a prefix and are easy to believe you have already covered.
What else do people ask?
How long does a robots.txt change take to reach meta-webindexer?
Is meta-webindexer a training crawler?
Does it obey robots.txt?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Meta states that the Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users, that Meta analyses online content to enhance the relevance and accuracy of Meta AI, and that allowing Meta-WebIndexer in robots.txt helps Meta cite and link to a site's content in Meta AI's responses.
developers.facebook.com · retrieved
“The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users.”
Meta documents that the user-agent string seen in log files for this crawler will be similar to "meta-webindexer/1.1", or the same token followed by a bracketed link to Meta's crawler documentation.
developers.facebook.com · retrieved · changes often, check the source
Meta states that blocking its crawlers is done by adding a disallow for the relevant crawler to robots.txt, names only Meta-ExternalFetcher and FacebookExternalHit as crawlers that may bypass robots.txt, and asks for up to 24 hours for changes to take effect because crawlers may cache the file for that long.
developers.facebook.com · retrieved
“In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.”
Meta's crawler documentation advises adding to an allow list either the user agent strings or the IP addresses used by the crawler, describing the addresses as more secure, but publishes no address list and no reverse-DNS domain for its crawlers.
developers.facebook.com · retrieved
“Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler.”
Meta documents Meta-ExternalAgent as a separate crawler that crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
developers.facebook.com · retrieved
RFC 9309 section 2.2.1 states that crawlers obey the groups whose user-agent line matches their product token, combined if there are several, and obey the group with the "*" user-agent only if no matching group exists.
rfc-editor.org · retrieved
“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”
Want whatever it takes to win the answer?
7 days free · card required