BM25

Where do exact words still win?

On questions built around one exact identifier. The table above holds two questions we track for attensira.com, about the same subject, read over the Attensira MCP for the 30 days to 4 October 2026. We have a page about GPTBot whose title and first sentence contain the exact string "GPTBot". The first question names that string; the second describes the same territory in general words. Our page was linked in 12 of 66 answers to the first and in none of 60 to the second.

The sources follow the same line. For "what is GPTBot", the most influential source in ChatGPT's answers was developers.openai.com, OpenAI's own page about the crawler, whose name is the query. For the generic question it was github.com, in a long tail where no site held more than 10% of the citations.

We do not know how ChatGPT retrieves pages, and two questions prove nothing on their own. But the pattern is the one BM25 exists to produce: a question built around one exact identifier rewards the pages that contain it literally. A generic question rewards whichever pages are most broadly relevant, and there are hundreds of those.

What is BM25?

BM25, short for Best Matching 25, is a ranking function for keyword search. Given a query, it scores each document from three ingredients:

1. Term frequency. How often each query word appears in the document, with diminishing returns. The second occurrence adds less than the first, and the twentieth adds almost nothing.
2. Inverse document frequency. How rare the word is across the whole collection. A match on "GPTBot" counts for far more than a match on "the" or "crawler".
3. Document length. A match in a short document counts for more than the same match buried in a very long one.

It is a refinement of TF-IDF, developed in information-retrieval research in the 1990s, and is still the default keyword scorer in widely used search libraries. Anthropic's contextual retrieval write-up describes it simply: a ranking function that uses lexical matching to find precise word or phrase matches, especially effective for unique identifiers and technical terms.

Why do AI systems still use BM25?

Because embeddings are good at meaning and bad at exactness. Anthropic's example is a search for "Error code TS-999". An embedding model may return pages about error codes in general and miss the one page containing that code. BM25 finds it.

So many retrieval pipelines run both and merge the results:

• BM25 finds the top chunks by exact match;
• vector search finds the top chunks by meaning;
• the two lists are combined and deduplicated, often with rank fusion, and a reranking step may reorder them before the model reads them.

Anthropic measured the effect on its benchmarks. With context added to each chunk, embeddings alone cut the top-20 retrieval failure rate from 5.7% to 3.7%. Adding BM25 cut it to 2.9%. Adding a reranker cut it to 1.9%.

What does BM25 mean for how you write pages?

Exact strings are findable; paraphrases are not. If a buyer will search for a product name, a model number, an error code, a plan name or a standard, that string must appear on your page as plain text, ideally in the heading and the first sentence of the passage that answers it.

• Write "GPTBot", not only "OpenAI's training crawler".
• Write the version number, not only "the latest release".
• Keep names consistent. A product called three different things on three pages splits its matches three ways.
• Put the identifier near the answer, because retrieval works on chunks, and a chunk without the word cannot match it.

What are the objections to writing for BM25?

"Keyword matching is old SEO." Keyword stuffing is old SEO, and BM25's saturation curve is one reason it stopped working. Using the exact name of the thing you are explaining is just precision.

"AI understands synonyms now." Meaning-based retrieval does. That is why hybrid systems keep a lexical half for the cases where meaning is not enough.

"So we should repeat our keyword everywhere." The second mention adds less than the first. Once, in the right sentence, does most of the work.

How does Attensira use this?

Attensira tracks the exact phrasings your buyers use, including names, codes and product terms, and reads which pages each engine cites for each phrasing. When a precise question goes to a competitor's page, the crew checks whether your page states the exact term where the answer lives, fixes it, and ships the change after your approval. The free audit lets you phrase its eight questions with your own exact terms and shows, per engine, whether each answer named you.