apprun.icuGuides

Guide · Documents & retrieval

BM25 vs embeddings: which is enough for a small set of documents?

BM25 ranks passages by the exact words they share with a question. Embeddings rank them by closeness of meaning. For a small document set, BM25 is a reasonable working start, and a short test shows whether it misses answers that matter.

By apprun.icuUpdated Read 7 minIncludes a free browser toolSources 5 cited
On this page
  1. The short answer
  2. What BM25 and embeddings each match
  3. When exact-term search is enough
  4. When meaning matching helps
  5. Cost, privacy and offline trade-offs
  6. How to decide with a ten-question test
  7. What you lose without embeddings
  8. Common mistakes
  9. Common questions
  10. Sources and further reading
A question takes a keyword path or a meaning path, then a test decidesYour questionWords, or meaning?Keyword pathBM25 word matchMeaning pathEmbeddings: nearbyvectorsWhen is it enough?Test ten realquestions
Both paths rank passages; the ten-question test shows whether the keyword path is enough.

What BM25 and embeddings each match

BM25 matches words, and embeddings match meaning. BM25 is a keyword ranking formula: it scores a passage higher when it contains the query's words, when they appear more often, in a shorter passage, and when they are rare across the collection. The words have to appear in the text. The PDF to Knowledge Pack uses BM25 for its search, and the PDF search guide walks through the formula step by step.

An embedding is a list of numbers that a model computes from a passage. Passages with similar meaning get lists that sit close together, so a question can reach a passage that shares none of its words. The match is made between numbers, which makes the reason for a given ranking harder to read than a word count.

When exact-term search is enough

Exact-term search can be enough when your questions reuse the document's own vocabulary, which is common for names, identifiers, quotes and error codes. Anthropic's engineering post on contextual retrieval, published 19 September 2024, says BM25 is "particularly effective for queries that include unique identifiers or technical terms" [Anthropic: Introducing Contextual Retrieval]. In its example, an embedding model "might find content about error codes in general, but could miss the exact" match that BM25 finds. When the answer is a code such as TS-999, the exact token is the thing to match.

BEIR, a benchmark that evaluated retrieval models across 18 datasets, reports: "Our results show BM25 is a robust baseline" [BEIR benchmark paper, arXiv 2021]. Its abstract describes 18 datasets from diverse text retrieval tasks and domains and does not mention chunking, so read the finding as evidence about those benchmarks, not about the pages of your PDF.

When meaning matching helps

Meaning matching helps when people ask in words the document does not use. Here is an invented example. A policy line reads "Refund eligibility: requests within 30 days of purchase." Someone asks, "Can I get my money back?" The two share no content word, so BM25 has nothing to match. An embedding model may place the question near that line, but that depends on the model and has to be tested on your own text.

Anthropic's post recommends combining the two. It reports that "Embeddings+BM25 is better than embeddings on their own" across its own tests, which compared several embedding models and dataset types. These are one company's experiments, so they show that combining can help, not that it always does. BEIR is also cautious about BM25 as a ceiling: "re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs". BM25 is a strong baseline, not the top performer on average.

Cost, privacy and offline trade-offs

Much of the cost difference comes from the model. BM25 needs no model to rank your text. Embeddings need a model to turn every chunk and every question into a vector, so their cost depends on that model and on where it runs, which also decides the privacy and offline picture.

Trade-offs between BM25 and embeddings; qualitative, since the sources give no per-query price or speed figures
FactorBM25 on your textEmbeddings
What it needsThe text and a word-splitting stepA model that turns passages and questions into vectors
What it storesWord statistics for each chunkA vector for each chunk
Cost per questionLow; no model callDepends on the model and where it runs
PrivacyStays wherever the search code runsA local model keeps text on your device; a hosted model sends text to its provider
Offline useWorks once the page is loadedWorks only if the model runs on your device
Typical missQuestions whose words are absent from the textExact codes or names can be missed, as Anthropic's example shows

How to decide with a ten-question test

Run the exact-word method first, then decide from its misses on the same ten questions. Ten is a practical floor the Chunking Strategy Comparator suggests, and that tool warns that small samples are noisy.

  1. Write ten questions the way real users would ask them, not in the document's jargon.
  2. For each question, mark the passage that answers it. The Chunking Strategy Comparator works this way. It tests lexical retrieval only, so it can show exact-word results but not embedding results.
  3. Record whether the answering passage appears in the top results. Use the same chunks and the same number of results for every method you try. If your source is a PDF, the PDF to Knowledge Pack runs BM25 over page-sized chunks, so each result names one page you can check.
  4. Label each miss. If the question shares no content word with its answer, it is a meaning miss. If it shares words and still ranks low, look at chunking and length first.
  5. Decide before you test a second method. If the misses are meaning misses and those answers matter, test embeddings or a combination on the same ten questions. If almost every answer is found, keep the exact-word method.
Example with invented questions and passages: four of ten rows shown
QuestionWords shared with the answer passageFound by BM25?Miss type
What is the notice period?"notice period"YesNone
What does TS-999 mean?"TS-999"YesNone
Can I get my money back?NoneNoMeaning miss
How long are backups kept?None; the passage says "Retention for snapshots is 35 days."NoMeaning miss

In these invented rows both misses share no words with their answers, so a second method is worth testing, but only after the full ten-question run.

What you lose without embeddings

You lose matches that depend on meaning rather than shared words. A question that paraphrases the text, uses a synonym or is written in another language will not reach a passage that shares no vocabulary with it. BM25 cannot rank what lacks the question's words; it can only order what contains them.

You keep exact names, codes and quotes, a score you can check by counting words, and no model to host or call. Whether those matter more than meaning matching depends on your documents, and the ten-question test is how to find out.

Common mistakes

  • Reading BEIR as proof that BM25 always wins. BEIR calls BM25 a robust baseline, while re-ranking and late-interaction models do better on average at higher computational cost.
  • Carrying chunk-size results from embedding tests over to BM25. Results measured with one retriever do not automatically transfer to another.
  • Testing with three questions. The comparator suggests at least ten, and small samples are noisy.
  • Treating k1 = 1.2 and b = 0.75 as tuned values. They are defaults in Elasticsearch and Lucene, and the Robertson and Zaragoza review says that "many experiments suggest a somewhat lower value of k1 and a somewhat higher value of b".
  • Using a lexical test to choose an embedding model. The comparator's results show structural effects, such as cut points and size, not which embedding model to buy.

Common questions

Is BM25 outdated compared with embeddings?

No source here calls it outdated. BEIR describes BM25 as "a robust baseline" across its 18 datasets, and it reports that re-ranking and late-interaction models do better on average, at high computational cost. BM25 is a strong baseline rather than always the strongest method.

Is BM25 or embeddings better for a small set of PDFs?

Neither wins in general, and no cited source measures small sets. As a working rule, BM25 can be enough when your questions reuse the document's own terms. When questions are worded differently from the text, test embeddings on the same ten questions. The sources here do not give a document count at which one method takes over.

What is the difference between BM25 and TF-IDF?

TF-IDF weights a word by how often it appears in a document and how rare it is across the collection. BM25 builds on TF-IDF and adds document-length normalization and a saturation function on term frequency, as Anthropic's contextual retrieval post explains.

Can I combine BM25 and embeddings?

Yes. Anthropic reports that "Embeddings+BM25 is better than embeddings on their own" across its own tests, which compared several embedding models. Run both, merge the results, and test the merged list on the same ten questions.

Does BM25 understand what my question means?

No. It counts and weights matching words, so a passage with the right words can rank high even when its sentence says something different. Read the top result, not only its rank.

How many questions should I use for the test?

The Chunking Strategy Comparator suggests at least ten and warns that tiny samples are noisy. Write the questions before you run any search, use real wording, and keep the same set for each method you compare.

Sources and further reading

Share this guide