MistralAI-Index vs MistralAI-Training: the cleanest crawler split published

The short answer

Mistral has published the crawler documentation everyone else's should look like. There is a crawler called MistralAI-Index whose job is indexing, a crawler called MistralAI-Training whose job is training, and a sentence saying the first one is not used for the second. You can read the robots file of a site that uses these tokens and know what its owner decided, without opening a single vendor page.

That sounds like a small thing. It is not. At most operators the boundary between "collecting for the index" and "collecting for the model" is something a publisher infers from a crawler's name, a marketing page and a certain amount of hope. Here it is written down.

Naming as a control surface

The convention matters more than any individual fact on the page. GPTBot tells you which company is crawling. MistralAI-Index tells you what the crawl is for. The first name requires documentation to interpret; the second one carries its own meaning into every robots.txt it appears in.

The practical consequence shows up in code review. A change that adds Disallow: / under MistralAI-Training is self-explanatory to the next person who reads the file, including the version of you who has forgotten this decision in eighteen months. A change that adds the same line under a vendor-branded token is a decision whose reasoning lives outside the repository, which is to say it is a decision that will eventually be reversed by accident.

If you maintain robots files for several hosts, this is an argument for a comment convention of your own. Name the question each block answers, not the vendor it addresses, because the vendor will add another agent and the question will stay the same.

What each crawler is documented to do

MistralAI-Index performs automated crawling of the web for indexing purposes only, feeding Mistral's search product, and Mistral states plainly that it is not used for generative AI training. Both halves of that sentence are load-bearing. The first tells you the crawl is systematic rather than triggered by a person. The second closes the gap that publishers otherwise fill with suspicion.

MistralAI-Training crawls content to build datasets for training Mistral's generative models, and Mistral documents that webmasters can disallow this user agent in their robots.txt file. It is a short, unglamorous entry that does the job: here is the agent, here is what it takes, here is how you say no.

Both are documented as governed by robots.txt, and both send user-agent strings that follow the same shape — the vendor's standard Mozilla prefix, then compatible; MistralAI-Index/1.0 or compatible; MistralAI-Training/1.0, then a link back to the very page these facts come from. A crawler that cites its own documentation in its user agent is doing something genuinely useful for the person reading a log at two in the morning.

The verification asymmetry

Here is the one place the symmetry breaks, and it breaks in the direction publishers will find annoying.

Mistral publishes an address file for MistralAI-Index at mistral.ai/mistralai-index-ips.json. The crawler page lists no equivalent for MistralAI-Training. So the agent whose behaviour publishers most want to audit — the one collecting material for a model, the one people write policies about — is the one you can only identify by a string that anyone on the internet can type into a header.

This is not hypothetical. Scrapers routinely wear the user agents of reputable crawlers, because a header that reads like a well-known bot gets fewer blocks and fewer challenges. Without an address list, a request claiming to be MistralAI-Training is a claim you cannot check. If you have disallowed that token and you keep seeing it in your logs, the honest conclusion is not "Mistral is ignoring robots.txt" — it is "somebody is using Mistral's name, and I have no way to tell which."

The workable response is to treat unverifiable crawler identity as a load question rather than a policy question. Rate-limit it at the edge, watch what it fetches, and do not build a compliance narrative on top of a forgeable string.

The robots.txt lines

Refuse training, keep indexing. This is the configuration Mistral's own naming invites, and it is the one most companies actually want:

User-agent: MistralAI-Training
Disallow: /

User-agent: MistralAI-Index
Allow: /

Refuse everything automated while leaving the user-triggered fetcher to its own rules:

User-agent: MistralAI-Training
Disallow: /

User-agent: MistralAI-Index
Disallow: /

And the version worth committing even though it changes nothing, because it converts a default into a documented decision that survives staff turnover:

User-agent: MistralAI-Training
Allow: /

User-agent: MistralAI-Index
Allow: /

Two mechanical notes, both of which cause more real damage than any vendor's policy. Robots.txt matches on the token, so the User-agent: line takes MistralAI-Index and not the whole Mozilla string. And robots.txt is per host and per scheme — the apex domain, the documentation subdomain and the help centre each serve their own file and each carry their own answer. For most technical companies the documentation host is both the most citable and the least audited.

What the version numbers will do to you

Both agents publish a /1.0 version today. Nobody has ever had to handle an increment, which is exactly why the increment will hurt. A firewall rule, a log dashboard or a WAF signature written against MistralAI-Index/1.0 matches perfectly right up until Mistral ships 1.1, at which point it silently matches nothing and the dashboard shows a crawler that appears to have stopped visiting.

Match on the token and on the self-identifying URL, both of which are stable, and never on the full string. This is the single most common cause of a company concluding that an AI crawler abandoned their site.

Why the explicit negative is worth more than it looks

"Not used for generative AI training" is nine words, and it changes the shape of the decision a publisher has to make.

Without it, a search crawler from a company that also trains models is an act of faith. The crawler's stated purpose is indexing; the company's business is models; and nothing in the documentation forbids the indexing corpus from becoming training material later. A cautious publisher, reasoning about incentives rather than promises, blocks both tokens and accepts the loss of visibility as the price of certainty. That reasoning is not paranoid — it is what most policy teams do when handed an ambiguous document.

With the negative in writing, the cautious publisher has something else to point at. It is still not a contract, and a determined operator could revise the page tomorrow. But it is a public, dated, quotable commitment from the operator's own documentation, and a commitment that specific is expensive to walk back quietly. Mistral gets a materially better answer from careful publishers than an equally trustworthy operator who left the sentence out.

That is the practical argument for anyone at a crawler operator reading this: the sentence you decline to write is the sentence a publisher will assume the worst about. Ambiguity does not read as neutrality. It reads as the option being kept open.

Reading a robots file that uses these tokens

One underrated benefit of job-named tokens is auditing somebody else's site — a competitor, an acquisition target, a partner whose content you are about to depend on.

A robots.txt that disallows MistralAI-Training and allows MistralAI-Index tells you the owner has thought about this and holds a specific position: happy to be found, unwilling to be training material. A file that disallows both is either a considered refusal or, far more often, the output of a blanket AI-crawler blocklist copied from a blog post. A file that mentions neither has made no decision at all, which for most of the web is the honest state of affairs.

You cannot get that read from a file full of vendor-branded tokens without cross-referencing four documentation pages. That is the whole argument for the naming convention, and it is the reason this pairing is worth a page even though the decision it describes takes two minutes to make.

What these tokens cannot do

Neither directive is retroactive. If Mistral's models already contain your content, a robots.txt commit today does not remove it, and no operator claims otherwise.

Neither reaches copies you do not serve. Syndicated posts, mirrors, aggregators and partner republications live behind other people's robots files, and for a widely-quoted publisher the copy someone else hosts is often the more heavily crawled one.

Neither controls what a person pastes into a chat window. Mistral documents a third agent, MistralAI-User, precisely because user-initiated fetches are a different category with different rules.

And neither is a licence. A robots directive is a machine-readable preference made to a well-behaved client. It is a good-faith mechanism between publishers and operators, and Mistral has held up its side of it more legibly than most. It is not a contract, and treating it as one is a category error that has cost people real money.

Confirming your change did anything

A robots.txt edit is a statement of intent. Your access log is the fact. After changing either token, the question worth answering is whether the agent you disallowed stopped fetching and the agent you allowed kept fetching — per URL, because a path-scoped rule behaves nothing like a host-scoped one.

Attensira's crawler logs record which agent fetched which URL and when, so a Mistral robots change can be checked against observed behaviour instead of assumed. To read your current rules first, the robots.txt generator and the bot access score will tell you what your file permits today.

Related comparisons

For the same training-versus-retrieval split at the two largest assistant vendors, read GPTBot vs OAI-SearchBot and ClaudeBot vs GPTBot. For the two vendors' search crawlers against each other, see Claude-SearchBot vs OAI-SearchBot. And for a training crawler whose output is a public archive rather than one company's model, see CCBot vs GPTBot.