The 30-minute plan for one document
This plan tests one document. You can run the chunking and scoring in the Chunking Strategy Comparator, which applies seven strategy settings to the same questions. The strategy definitions and published numbers are in the strategy guide. If your source is a PDF, extract the text and check its reading order first with the PDF to Knowledge Pack.
| Step | Minutes | What you produce |
|---|---|---|
| 1. Write 10 to 15 questions from real use | 0 to 8 | A list written before you look at any chunk |
| 2. Mark the answer passage for each question | 8 to 14 | The shortest span that answers it |
| 3. Choose three to five settings | 14 to 18 | Settings that differ on one axis at a time |
| 4. Run each with the same retriever and k | 18 to 22 | One result per setting |
| 5. Read the missed questions | 22 to 27 | The reason each miss happened |
| 6. Write the decision | 27 to 30 | The setting to build with, and what to retest |
Steps 1 and 2 shape the result most.
Write the questions first, then mark the answers
Take questions from support tickets, search logs or the questions you would ask the document yourself. Write them before you look at any chunk. Once you have seen where the cuts fall, the questions drift toward the cuts you already like, and the test then confirms your own choice.
The gold passage is the shortest span that answers the question. A long span is hard for any small chunk to contain whole, so an intact score partly measures answer length. Keep answers to a sentence or two where you can.
Ten questions is a practical floor for this plan: below that, one question moves results a lot. Questions built from section headings are an easy shortcut, but they may favour cuts that follow headings. Mix them with questions a reader would actually type.
What each number means in this test
Found, intact and tokens returned use the top k chunks for each question. Split is judged over every chunk of the strategy, because a cut is a property of the chunking, and MRR uses the full ranking. Mid-sentence cuts describe the whole strategy, not one question.
| Measure | Definition in this test | Read it as |
|---|---|---|
| Found@k | Share of questions where a top-k chunk overlaps at least half of the answer's characters | Did the retriever surface the answer at all? |
| Intact@k | Share of questions where one top-k chunk contains the whole answer | Did the cut keep the answer in one piece? |
| Split | Share of questions where no single chunk holds the answer, but two or more together cover it, judged over every chunk | A boundary cut through the answer |
| MRR | Mean of 1 divided by the rank of the first chunk that overlaps the answer at all, with 0 when none does | Higher means the match sits nearer the top |
| Tokens returned | Mean, over questions, of the tokens in the top-k chunks | What you would send the model for that question |
| Mid-sentence chunks | Share of chunks that start or end inside a sentence, for the whole strategy | Quality of the chunk boundaries |
The MRR convention used here
Sources define reciprocal rank as a score "based on the reciprocal of the rank of the highest-scoring relevance document," but they give no formula and no rule for a question with no match. This site's convention, which the tool implements, is this: the tool ranks the chunks that share at least one indexed term with the question, best first, and MRR takes 1 divided by the rank of the first chunk that overlaps the answer at all. Overlap here means any shared characters, not the 50 percent test used for found@k. A question with no such chunk scores 0, and that zero stays in the average. A chunk that overlaps the answer but shares no indexed term with the question is never ranked, so it contributes nothing.
Read the result without over-reading it
Compare settings only after the run, and after you have read the misses. With 10 questions, each question moves a rate by 10 percentage points; with 15, by about 7. A two-question gap is not a finding. Chroma's 472-query evaluation still reported standard deviations of about 30 points on recall, computed across all queries and corpora, which shows how much results vary from question to question.
| What you see | What it usually means | What to try next |
|---|---|---|
| Found is low for every setting | The questions, marked answers or wording may not match the document | Check three missed questions by hand; reword only if a real user would ask that way |
| Split is high under fixed-size, low under sentence or heading-aware | Boundaries are cutting answers | Try a larger size or a boundary-respecting cut, and compare tokens returned |
| Intact is high, and tokens returned are high too | Large chunks or overlap keep answers whole at the cost of repeated text | Try a smaller size or less overlap, then compare again |
| Found is equal, but MRR differs | The answer is returned but ranks lower in some settings | Check which chunks rank above the match for those questions |
| Heading-aware wins only on heading questions | The test rewards the structure it was built around | Add questions that do not reuse section titles |
| Two settings differ by one or two questions | Noise at this sample size | Add questions before you choose |
| The best setting changes when k changes | The result depends on how much context you send | Fix k at the value you will use in production |
Run the same check after any change to the document, because a new section can move cut points for every setting. The Chunking Strategy Comparator keeps the retriever and k fixed across settings, so a change in the result points at the chunking.
What lexical retrieval can and cannot tell you
This test uses a lexical retriever. BM25 scores chunks by the words they share with the question. Its defaults in Lucene 9.9.0 and Elasticsearch are k1 = 1.2 and b = 0.75, which are library defaults rather than values tuned to your documents. The tool's BM25 and TF-IDF also match word stems, drop common English stop words and ignore one-letter terms. Anthropic describes BM25 as one that "uses lexical matching to find precise word or phrase matches," which makes it strong on identifiers. Its example is an error code, TS-999, that an embedding search could miss. The BM25 versus embeddings guide covers that trade-off in more depth.
Because the retriever stays fixed, the test isolates what the chunks do: where cuts fall, how large the pieces are, whether a heading gives context, and how many tokens come back. Those are the effects it can show.
It cannot show how a paraphrased question fares, because lexical matching scores shared words. It cannot choose an embedding model, and a setting that wins under BM25 may lose under embeddings. Anthropic found that embeddings plus BM25 beat embeddings alone in its experiments, so a production system may need both. Run the same questions through your embedding model before you commit.
What this test cannot tell you, and common mistakes
The test does not judge the model's answer. Found and intact describe retrieval only. It also cannot prove that a setting is best. NVIDIA's best token size changed between datasets, and Microsoft describes its own figures as starting points.
- Writing questions after seeing the chunks. The test then rewards the cuts you already prefer.
- Copying a chunk size from a tutorial without checking the library version you install.
- Using long answer spans, which turns intact into a test of span length.
- Reporting found alone. A setting can find the answer and still send far more tokens to the model.
- Mixing Chroma's token-level recall with your recall@k. The two measure different things.
Common questions
How many questions do I need to test chunk size?
Ten to fifteen real questions is the range this plan uses. With ten, each question moves a rate by 10 points; with fifteen, by about 7. More questions help, but writing good ones is the bottleneck, so start with ten and add more when the results are close.
Can I use the same questions to choose a setting and to report it?
Not as a clean test. After you choose a setting with these questions, they no longer show how it performs on new ones. Write a few new questions after you choose, and check that the setting still holds before you build on it.
What is MRR in retrieval?
Mean reciprocal rank averages, over all questions, 1 divided by the rank of the first chunk that overlaps the answer at all. A question with no such chunk scores 0, and that zero stays in the average. MRR is not capped by found@k: an answer that sits second when k is 1 is not found, yet it still scores one half.
Is found@k the same as recall@k?
No. Found is a per-question test: a top-k chunk overlaps at least half of the answer. Recall@k is the share of relevant documents retrieved by rank k, and Success@k asks whether a relevant result is in the top k. Chroma's recall counts tokens. The strategy guide's recall@k question gives the full ir-measures wording.
Do I need a retriever to test chunking?
Yes. Chunk size changes what a retriever returns, so the test has to include one. BM25 is a fair baseline, and BEIR's authors call it "a robust baseline." Use the same retriever and k for every setting, then repeat the check with your embedding model.
Can a 30-minute test replace a full evaluation?
No. It is a first check that narrows the settings worth building. A full evaluation also checks the model's answers, uses more questions and includes the embedding model you will ship.
Sources and further reading
- Chroma, "Evaluating Chunking Strategies for Retrieval," technical report, July 2024
- Anthropic, "Introducing Contextual Retrieval," engineering blog, 19 September 2024
- Microsoft Learn, "Chunk documents," last updated 2 July 2026
- NVIDIA Technical Blog, "Finding the Best Chunking Strategy for Accurate AI Responses," 18 June 2025
- Pinecone, "Chunking Strategies for LLM Applications," 28 June 2025
- Thakur et al., "BEIR," arXiv 2104.08663, accepted at NeurIPS 2021
- ir-measures, "Measures" reference, Terrier team, undated
- Apache Lucene 9.9.0, BM25Similarity API documentation
- Elastic, "Similarity settings" reference, undated