apprun.icuGuides

Guide · Documents & retrieval

How to test chunk size before you build a RAG app

Pick 10 to 15 real questions, mark the passage that answers each, run a few chunking settings through the same retriever and compare how often answers come back whole and how many tokens each setting sends.

By apprun.icuUpdated Read 8 minIncludes a free browser toolSources 9 cited
On this page
  1. The short answer
  2. The 30-minute plan for one document
  3. Write the questions first, then mark the answers
  4. What each number means in this test
  5. Read the result without over-reading it
  6. What lexical retrieval can and cannot tell you
  7. What this test cannot tell you, and common mistakes
  8. Common questions
  9. Sources and further reading
Questions with marked answers are chunked, retrieved top k, and then scored Questions10 to 15 realwith gold passages Chunkone settingat a time Retrievetop k chunkssame retriever Read the resultfound, intactMRR, tokens
Each setting is run the same way, and you read the result only after you have checked the misses.

The 30-minute plan for one document

This plan tests one document. You can run the chunking and scoring in the Chunking Strategy Comparator, which applies seven strategy settings to the same questions. The strategy definitions and published numbers are in the strategy guide. If your source is a PDF, extract the text and check its reading order first with the PDF to Knowledge Pack.

The 30-minute test, step by step. The minutes are a guide, not a guarantee.
StepMinutesWhat you produce
1. Write 10 to 15 questions from real use0 to 8A list written before you look at any chunk
2. Mark the answer passage for each question8 to 14The shortest span that answers it
3. Choose three to five settings14 to 18Settings that differ on one axis at a time
4. Run each with the same retriever and k18 to 22One result per setting
5. Read the missed questions22 to 27The reason each miss happened
6. Write the decision27 to 30The setting to build with, and what to retest

Steps 1 and 2 shape the result most.

Write the questions first, then mark the answers

Take questions from support tickets, search logs or the questions you would ask the document yourself. Write them before you look at any chunk. Once you have seen where the cuts fall, the questions drift toward the cuts you already like, and the test then confirms your own choice.

The gold passage is the shortest span that answers the question. A long span is hard for any small chunk to contain whole, so an intact score partly measures answer length. Keep answers to a sentence or two where you can.

Ten questions is a practical floor for this plan: below that, one question moves results a lot. Questions built from section headings are an easy shortcut, but they may favour cuts that follow headings. Mix them with questions a reader would actually type.

What each number means in this test

Found, intact and tokens returned use the top k chunks for each question. Split is judged over every chunk of the strategy, because a cut is a property of the chunking, and MRR uses the full ranking. Mid-sentence cuts describe the whole strategy, not one question.

Measures used in this test, with the definition this site uses
MeasureDefinition in this testRead it as
Found@kShare of questions where a top-k chunk overlaps at least half of the answer's charactersDid the retriever surface the answer at all?
Intact@kShare of questions where one top-k chunk contains the whole answerDid the cut keep the answer in one piece?
SplitShare of questions where no single chunk holds the answer, but two or more together cover it, judged over every chunkA boundary cut through the answer
MRRMean of 1 divided by the rank of the first chunk that overlaps the answer at all, with 0 when none doesHigher means the match sits nearer the top
Tokens returnedMean, over questions, of the tokens in the top-k chunksWhat you would send the model for that question
Mid-sentence chunksShare of chunks that start or end inside a sentence, for the whole strategyQuality of the chunk boundaries

The MRR convention used here

Sources define reciprocal rank as a score "based on the reciprocal of the rank of the highest-scoring relevance document," but they give no formula and no rule for a question with no match. This site's convention, which the tool implements, is this: the tool ranks the chunks that share at least one indexed term with the question, best first, and MRR takes 1 divided by the rank of the first chunk that overlaps the answer at all. Overlap here means any shared characters, not the 50 percent test used for found@k. A question with no such chunk scores 0, and that zero stays in the average. A chunk that overlaps the answer but shares no indexed term with the question is never ranked, so it contributes nothing.

Read the result without over-reading it

Compare settings only after the run, and after you have read the misses. With 10 questions, each question moves a rate by 10 percentage points; with 15, by about 7. A two-question gap is not a finding. Chroma's 472-query evaluation still reported standard deviations of about 30 points on recall, computed across all queries and corpora, which shows how much results vary from question to question.

How to read common patterns in the result
What you seeWhat it usually meansWhat to try next
Found is low for every settingThe questions, marked answers or wording may not match the documentCheck three missed questions by hand; reword only if a real user would ask that way
Split is high under fixed-size, low under sentence or heading-awareBoundaries are cutting answersTry a larger size or a boundary-respecting cut, and compare tokens returned
Intact is high, and tokens returned are high tooLarge chunks or overlap keep answers whole at the cost of repeated textTry a smaller size or less overlap, then compare again
Found is equal, but MRR differsThe answer is returned but ranks lower in some settingsCheck which chunks rank above the match for those questions
Heading-aware wins only on heading questionsThe test rewards the structure it was built aroundAdd questions that do not reuse section titles
Two settings differ by one or two questionsNoise at this sample sizeAdd questions before you choose
The best setting changes when k changesThe result depends on how much context you sendFix k at the value you will use in production

Run the same check after any change to the document, because a new section can move cut points for every setting. The Chunking Strategy Comparator keeps the retriever and k fixed across settings, so a change in the result points at the chunking.

What lexical retrieval can and cannot tell you

This test uses a lexical retriever. BM25 scores chunks by the words they share with the question. Its defaults in Lucene 9.9.0 and Elasticsearch are k1 = 1.2 and b = 0.75, which are library defaults rather than values tuned to your documents. The tool's BM25 and TF-IDF also match word stems, drop common English stop words and ignore one-letter terms. Anthropic describes BM25 as one that "uses lexical matching to find precise word or phrase matches," which makes it strong on identifiers. Its example is an error code, TS-999, that an embedding search could miss. The BM25 versus embeddings guide covers that trade-off in more depth.

Because the retriever stays fixed, the test isolates what the chunks do: where cuts fall, how large the pieces are, whether a heading gives context, and how many tokens come back. Those are the effects it can show.

It cannot show how a paraphrased question fares, because lexical matching scores shared words. It cannot choose an embedding model, and a setting that wins under BM25 may lose under embeddings. Anthropic found that embeddings plus BM25 beat embeddings alone in its experiments, so a production system may need both. Run the same questions through your embedding model before you commit.

What this test cannot tell you, and common mistakes

The test does not judge the model's answer. Found and intact describe retrieval only. It also cannot prove that a setting is best. NVIDIA's best token size changed between datasets, and Microsoft describes its own figures as starting points.

  • Writing questions after seeing the chunks. The test then rewards the cuts you already prefer.
  • Copying a chunk size from a tutorial without checking the library version you install.
  • Using long answer spans, which turns intact into a test of span length.
  • Reporting found alone. A setting can find the answer and still send far more tokens to the model.
  • Mixing Chroma's token-level recall with your recall@k. The two measure different things.

Common questions

How many questions do I need to test chunk size?

Ten to fifteen real questions is the range this plan uses. With ten, each question moves a rate by 10 points; with fifteen, by about 7. More questions help, but writing good ones is the bottleneck, so start with ten and add more when the results are close.

Can I use the same questions to choose a setting and to report it?

Not as a clean test. After you choose a setting with these questions, they no longer show how it performs on new ones. Write a few new questions after you choose, and check that the setting still holds before you build on it.

What is MRR in retrieval?

Mean reciprocal rank averages, over all questions, 1 divided by the rank of the first chunk that overlaps the answer at all. A question with no such chunk scores 0, and that zero stays in the average. MRR is not capped by found@k: an answer that sits second when k is 1 is not found, yet it still scores one half.

Is found@k the same as recall@k?

No. Found is a per-question test: a top-k chunk overlaps at least half of the answer. Recall@k is the share of relevant documents retrieved by rank k, and Success@k asks whether a relevant result is in the top k. Chroma's recall counts tokens. The strategy guide's recall@k question gives the full ir-measures wording.

Do I need a retriever to test chunking?

Yes. Chunk size changes what a retriever returns, so the test has to include one. BM25 is a fair baseline, and BEIR's authors call it "a robust baseline." Use the same retriever and k for every setting, then repeat the check with your embedding model.

Can a 30-minute test replace a full evaluation?

No. It is a first check that narrows the settings worth building. A full evaluation also checks the model's answers, uses more questions and includes the embedding model you will ship.

Sources and further reading

Share this guide