apprun.icuGuides

Guide · Documents & retrieval

RAG chunking strategies compared: fixed, sentence, recursive and heading-aware

Chunking decides where a document is cut before retrieval. Fixed-size, sentence, paragraph, recursive, heading-aware and semantic cuts each keep different answers whole, and published results do not agree on one winner.

By apprun.icuUpdated Read 9 minIncludes a free browser toolSources 11 cited
On this page
  1. The short answer
  2. Six strategies and where each one cuts
  3. One paragraph, three cuts
  4. What Chroma's evaluation found, and its limits
  5. What Anthropic reported for contextual retrieval, and what it does not show
  6. Why overlap is a trade-off
  7. Why no strategy wins everywhere, so test on your documents
  8. What this comparison cannot tell you
  9. Common questions
  10. Sources and further reading

Six strategies and where each one cuts

A chunk is a piece of a document that a retrieval system stores and returns. Retrieval-augmented generation (RAG) searches chunks for the ones that match a question and passes them to a language model. The cut rule decides which sentences share a chunk, and so which answers get split.

The last column is a starting point for testing, not a ranking.

Six chunking strategies by cut rule, strength, failure mode and when to test it
StrategyHow it cutsStrengthsFailure modeTest it when
Fixed-sizeA set number of tokens or charactersSimple, predictable, easy to compareCuts through sentences and answersYou need a baseline
SentenceSentence boundaries, grouped up to a sizeEach sentence stays wholeA single sentence may be too short to answer aloneAnswers sit inside one or two sentences
ParagraphBlank linesKeeps the author's grouping of ideasLong paragraphs must still be splitParagraph breaks are clean
RecursiveSeparators in order, from paragraph breaks to spaces, until pieces fitKeeps the largest unit that fitsDefault separators can fall back to word cutsYou want a general default
Heading-awareMarkdown or plain-text headings, then a size limitEach chunk keeps its section contextPlain text has no headingsHeadings are reliable
SemanticTopic shifts, often found with embedding similarityBoundaries follow meaning more than lengthNeeds an embedding model and a thresholdTopics shift often and embedding is affordable

Pinecone calls fixed-size chunking "the best path in most cases" and recommends starting there, but its article shows no experiment to back that up.

LangChain's documentation advises, "For most use cases, start with the RecursiveCharacterTextSplitter." Its default separators are a paragraph break, a line break, a space and an empty string, tried in that order. LlamaIndex warns that its regex "primarily works for English sentences" and that you "may have to tune the breakpoint percentile threshold." In the tool, Semantic cuts where TF-IDF similarity between neighbouring sentence windows is lowest, at the 25th percentile, with no embedding model.

One paragraph goes through a cut rule, becomes chunks, and is tested on your own questions One paragraph30 words Cut rulefixed sizesentenceheading Chunksanswer splitor kept whole Your test10 to 15 questions
One paragraph goes through a cut rule into chunks, and the chunks are then tested against your own questions.

One paragraph, three cuts

This example is invented. It counts words instead of tokens so you can check each cut by eye. The paragraph is 30 words, and the answer to "How long do I have to ask for a refund?" is "within 30 days of purchase."

Refunds are available within 30 days of purchase. Send the order number to support and we reply within two business days. Annual plans are refunded pro rata for unused months.

Fixed-size, six words, no overlap. The answer is split across chunks 1 and 2.

[1] Refunds are available within 30 days
[2] of purchase. Send the order number
[3] to support and we reply within
[4] two business days. Annual plans are
[5] refunded pro rata for unused months.

Sentence cut. The answer sits inside chunk 1, so it stays whole.

[1] Refunds are available within 30 days of purchase.
[2] Send the order number to support and we reply within two business days.
[3] Annual plans are refunded pro rata for unused months.

Recursive, 12 words, default separators. The paragraph has no line breaks, so the splitter falls back to spaces. The answer is inside chunk 1, but chunk 2 starts mid-sentence.

[1] Refunds are available within 30 days of purchase. Send the order number
[2] to support and we reply within two business days. Annual plans are
[3] refunded pro rata for unused months.

Now take a second answer, "within 30 days of purchase. Send the order number," which crosses a sentence boundary. The sentence cut splits it, while the 12-word recursive cut keeps it whole. One example proves little, because the outcome depends on where the boundary falls. The tool sizes chunks in estimated tokens, not words, so it will not reproduce these word counts. To see how each setting cuts your own text, paste your document into the Chunking Strategy Comparator.

What Chroma's evaluation found, and its limits

Chroma's July 2024 report, "Evaluating Chunking Strategies for Retrieval," tested several chunkers on five corpora with 472 generated queries. Its recall counts tokens, so it is not recall@k, the share of relevant documents retrieved in the top k. Its IoU (intersection over union) penalizes both missed and extra tokens.

Recall = |t_e ∩ t_r| / |t_e|
IoU    = |t_e ∩ t_r| / (|t_e| + |t_r| − |t_e ∩ t_r|)

t_e = tokens in the relevant excerpts
t_r = tokens in the retrieved chunks
Chroma's results with text-embedding-3-large and five retrieved chunks, as percentages (mean ± standard deviation over all queries and corpora). n/r means not reported in the fetched text.
SettingMean chunk tokensRecallPrecisionIoU
Recursive, size 200, no overlapabout 13788.1 ± 30.17.0 ± 5.66.9 ± 5.6
Recursive, size 400, no overlapabout 27689.5 ± 29.73.6 ± 3.23.6 ± 3.2
ClusterSemanticChunker, max 200about 10387.3 ± 29.88.0 ± 6.08.0 ± 6.0
ClusterSemanticChunker, max 400about 18291.3 ± 25.4n/r4.5 ± 3.4
Kamradt semantic defaultabout 66083.6 ± 36.8n/r1.5 ± 1.6
LLM chunker (GPT-4o)about 24091.9 ± 26.5n/r3.9 ± 3.2

In the rows that report them, smaller chunks had higher precision and IoU, while recall stayed between 83.6 and 91.9 in every row. The report says the recursive splitter at 200 with no overlap "performs well": while it "does not achieve the best result," it is "consistently high performing." It also warns that "default settings for certain popular chunking strategies can lead to relatively poor performance." One semantic row is Chroma's own ClusterSemanticChunker, so weigh it with the report's authorship in mind.

The limits are large. The embedding models date from 2024, retrieval was embedding-based rather than BM25, and the queries and excerpts were generated by an LLM. LLMs "tend to generate a specific style of question," and the method cannot find relevant excerpts the generator did not produce. The report itself calls its "relatively small size, and limited diversity" a limitation.

What Anthropic reported for contextual retrieval, and what it does not show

Contextual retrieval changes what each chunk contains, not where it is cut. In Anthropic's engineering post of 19 September 2024, an LLM writes a context of usually 50 to 100 tokens for each chunk. That context is prepended before embedding and before the BM25 index is built. The post measures failure as "1 minus recall@20."

Anthropic's reported top-20 chunk failure rate, averaged across knowledge domains (lower is better)
ConfigurationFailure rateReduction
Baseline5.7%none
Contextual embeddings3.7%35%
Contextual embeddings and contextual BM252.9%49%
Plus reranking1.9%67%

These are Anthropic's own internal results. The headline is that the method can "reduce the number of failed retrievals by 49%" before any reranking. The experiments covered codebases, fiction, arXiv papers and science papers, used Gemini Text 004, and the final row adds a Cohere reranker. The post's charts and appendix are not in the text used here, so no BM25-only figure is available. The post's only explicit size guidance is that chunks are "usually no more than a few hundred tokens."

Why overlap is a trade-off

Overlap repeats the end of one chunk at the start of the next, so an answer cut at a boundary can appear whole in one of them. LangChain's parameter notes say overlapping chunks "helps to mitigate loss of information when context is divided between chunks." The cost is repetition: the same words are stored twice and can reach the model twice.

Chroma's data show both effects. With all-MiniLM-L6-v2, 250-token chunks with 125 tokens of overlap had recall of 82.4 ± 36.2, against 77.1 ± 39.3 with none. Elsewhere, the report notes that "reducing chunk overlap improves IoU scores." NVIDIA tested 10, 15 and 20 percent on one dataset and picked 15 percent, a result it does not present as a full grid search. Treat overlap as a value to test. At 1,024 tokens, 15 percent overlap repeats about 154 tokens at each boundary.

Why no strategy wins everywhere, so test on your documents

No strategy wins everywhere. NVIDIA's page-level chunks, which need page boundaries, had the highest average accuracy (0.648), but the post says page-level chunking is "not always the absolute best for every dataset." Its best token size varied: 1,024 tokens for FinanceBench (0.579, against 0.506 at 2,048) and 512 for Earnings (0.681, against 0.663 at 1,024).

Microsoft gives 512 tokens with 25 percent overlap in one place and 200 words or 600 characters with 10 to 15 percent in another, and it calls these starting points. It also notes that chunking "is only required if the source documents are too large" for a model's input.

To test your own documents, run the same questions through each setting with the same retriever. Pinecone's guide recommends the same approach: build one index per chunk size, or one index with namespaces, and run the same queries against each. The Chunking Strategy Comparator does this for seven settings with one lexical retriever, so chunking is the only variable. The chunk size test guide shows a 30-minute method.

What this comparison cannot tell you

This comparison shows structural effects under one retriever. It does not say which embedding model to buy, and hybrid retrieval or reranking can change the ranking. BEIR, a benchmark of 18 datasets, says that "re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs."

BM25 is still a fair test bed, and BEIR's authors call it "a robust baseline." A winner under one retriever is only a candidate, so re-run the best settings with the embedding model you plan to ship.

Common mistakes

  • Trusting a copied default. The LangChain base splitter read for this guide defaults to 4,000 characters with 200 of overlap. Check your installed version.
  • Comparing Chroma's recall with your recall@k. Chroma's recall counts tokens, so the two measure different things.
  • Reading recall alone. The setting with the best recall may also return the most tokens to the model.

Common questions

What chunk size should I use for RAG?

No size is correct for every corpus. Microsoft suggests starting with 512 tokens and 25 percent overlap, and Pinecone suggests exploring smaller sizes such as 128 or 256 tokens. Chroma described its recursive setting at 200 tokens as "consistently high performing," though not the best result. Use these as starting points, then test against your own questions.

How much chunk overlap is enough?

The sources give starting points, not optimal values. Microsoft uses 25 percent in one example and 10 to 15 percent elsewhere. NVIDIA found 15 percent best in one tested setting. Test zero, 10, 15 and 20 percent on your own data, and watch how many tokens each setting returns.

Is semantic chunking better than fixed-size chunking?

Not in general. Chroma's Kamradt default setting scored "slightly below average across all metrics." Chroma's ClusterSemanticChunker at a 400-token maximum had recall of 91.3, against 87.9 for TokenText, a fixed token window of 800 tokens with 400 overlap. The sizes differ, so the comparison is not clean. Test both on the same questions.

What does recall@k measure?

In the ir-measures documentation, Recall@k is "The fraction of relevant documents for a query that have been retrieved by rank k." Some tasks instead ask whether any relevant result appears in the top k, which ir-measures calls Success@k.

What is the recursive character text splitter?

It is a LangChain splitter that tries separators in order, from paragraph breaks down to spaces, and keeps the largest unit that fits the size. Its size is counted in characters by default. Check the defaults in the version you install, because they can change.

Sources and further reading

Share this guide