BM25

BM25 — "Best Matching 25" — is a ranking function for lexical search: it scores documents by how well their actual words match the words in a query. It predates every neural retrieval technique in common use and remains a component of most of them, because it is good at the one thing vectors are bad at.

Anthropic's description is a useful working definition: BM25 is a ranking function that uses lexical matching to find precise word or phrase matches, and is particularly effective for queries that include unique identifiers or technical terms.

What it does to term frequency

BM25 builds on TF-IDF, which measures how important a word is to a document within a collection: a term that appears often in one document and rarely across the corpus is a strong signal. BM25 refines it in two ways that matter in practice. It considers document length, so a long page does not score higher merely by containing more words. And it applies a saturation function to term frequency, which — as Anthropic puts it — helps prevent common words from dominating the results.

Saturation is the interesting half. Under raw term frequency, a page repeating a phrase forty times scores roughly forty times a page saying it once. Under BM25, the marginal value of each repetition falls away quickly, so the twentieth mention is worth almost nothing. Keyword stuffing has been arithmetically pointless against this class of scorer for a long time.

The failure that keeps it alive

The reason BM25 still appears in modern retrieval stacks is a specific weakness of embeddings. Vectors excel at capturing semantic relationships and can miss crucial exact matches.

Anthropic's example: a user queries "Error code TS-999" in a technical support database. An embedding model might find content about error codes in general but could miss the exact TS-999 match. BM25 looks for that specific string and finds it.

Generalise that and you have a large class of real queries — part numbers, SKUs, API method names, version strings, drug names, statute numbers, error codes, people's names. These are precisely the queries where the right answer is one document and a topically similar document is worthless.

Hybrid retrieval

Because the two methods fail in different directions, most serious systems run both. The pipeline Anthropic describes:

1. Break the corpus into chunks of no more than a few hundred tokens.
2. Create TF-IDF encodings and semantic embeddings for those chunks.
3. Use BM25 to find the top chunks by exact match.
4. Use embeddings to find the top chunks by semantic similarity.
5. Combine and deduplicate the two result sets using rank fusion.
6. Add the top-K chunks to the prompt.

Its measured conclusion, across codebases, fiction, ArXiv papers and science papers, is that embeddings plus BM25 beats embeddings alone: contextual embeddings alone cut the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%, while adding contextual BM25 took the reduction to 49%, or 2.9%. A reranking pass on top took it to 67%.

What follows for what you publish

The lexical half of hybrid retrieval is the half a writer can influence most directly, and it argues against a habit that good editing usually encourages.

Elegant prose replaces a name with a pronoun, a version number with "the current release", and a product code with "the model in question". A lexical index cannot match any of those. If a query is likely to contain a literal string, that string has to appear literally on your page — spelled the way people write it, including the punctuation and the casing they use.

Three concrete habits:

• Write identifiers out. TS-999, v4.2, the full product name, the model number. Once per section is enough; saturation means more is wasted.
• Include the variants people actually type, where they are genuinely different terms rather than the same word repeated — a stuffed title is caught by both a scorer and an editor.
• Do not rely on the heading to carry the term into a chunk that no longer contains it, which is the same chunking discipline seen from the lexical side.

BM25 is thirty-year-old information retrieval, and it is a live part of how a modern assistant finds the passage it quotes. Writing that a keyword matcher can find is not a legacy concern.