Six strategies and where each one cuts
A chunk is a piece of a document that a retrieval system stores and returns. Retrieval-augmented generation (RAG) searches chunks for the ones that match a question and passes them to a language model. The cut rule decides which sentences share a chunk, and so which answers get split.
The last column is a starting point for testing, not a ranking.
| Strategy | How it cuts | Strengths | Failure mode | Test it when |
|---|---|---|---|---|
| Fixed-size | A set number of tokens or characters | Simple, predictable, easy to compare | Cuts through sentences and answers | You need a baseline |
| Sentence | Sentence boundaries, grouped up to a size | Each sentence stays whole | A single sentence may be too short to answer alone | Answers sit inside one or two sentences |
| Paragraph | Blank lines | Keeps the author's grouping of ideas | Long paragraphs must still be split | Paragraph breaks are clean |
| Recursive | Separators in order, from paragraph breaks to spaces, until pieces fit | Keeps the largest unit that fits | Default separators can fall back to word cuts | You want a general default |
| Heading-aware | Markdown or plain-text headings, then a size limit | Each chunk keeps its section context | Plain text has no headings | Headings are reliable |
| Semantic | Topic shifts, often found with embedding similarity | Boundaries follow meaning more than length | Needs an embedding model and a threshold | Topics shift often and embedding is affordable |
Pinecone calls fixed-size chunking "the best path in most cases" and recommends starting there, but its article shows no experiment to back that up.
LangChain's documentation advises, "For most use cases, start with the RecursiveCharacterTextSplitter." Its default separators are a paragraph break, a line break, a space and an empty string, tried in that order. LlamaIndex warns that its regex "primarily works for English sentences" and that you "may have to tune the breakpoint percentile threshold." In the tool, Semantic cuts where TF-IDF similarity between neighbouring sentence windows is lowest, at the 25th percentile, with no embedding model.
One paragraph, three cuts
This example is invented. It counts words instead of tokens so you can check each cut by eye. The paragraph is 30 words, and the answer to "How long do I have to ask for a refund?" is "within 30 days of purchase."
Refunds are available within 30 days of purchase. Send the order number to support and we reply within two business days. Annual plans are refunded pro rata for unused months.
Fixed-size, six words, no overlap. The answer is split across chunks 1 and 2.
[1] Refunds are available within 30 days
[2] of purchase. Send the order number
[3] to support and we reply within
[4] two business days. Annual plans are
[5] refunded pro rata for unused months.
Sentence cut. The answer sits inside chunk 1, so it stays whole.
[1] Refunds are available within 30 days of purchase.
[2] Send the order number to support and we reply within two business days.
[3] Annual plans are refunded pro rata for unused months.
Recursive, 12 words, default separators. The paragraph has no line breaks, so the splitter falls back to spaces. The answer is inside chunk 1, but chunk 2 starts mid-sentence.
[1] Refunds are available within 30 days of purchase. Send the order number
[2] to support and we reply within two business days. Annual plans are
[3] refunded pro rata for unused months.
Now take a second answer, "within 30 days of purchase. Send the order number," which crosses a sentence boundary. The sentence cut splits it, while the 12-word recursive cut keeps it whole. One example proves little, because the outcome depends on where the boundary falls. The tool sizes chunks in estimated tokens, not words, so it will not reproduce these word counts. To see how each setting cuts your own text, paste your document into the Chunking Strategy Comparator.
What Chroma's evaluation found, and its limits
Chroma's July 2024 report, "Evaluating Chunking Strategies for Retrieval," tested several chunkers on five corpora with 472 generated queries. Its recall counts tokens, so it is not recall@k, the share of relevant documents retrieved in the top k. Its IoU (intersection over union) penalizes both missed and extra tokens.
Recall = |t_e ∩ t_r| / |t_e|
IoU = |t_e ∩ t_r| / (|t_e| + |t_r| − |t_e ∩ t_r|)
t_e = tokens in the relevant excerpts
t_r = tokens in the retrieved chunks
| Setting | Mean chunk tokens | Recall | Precision | IoU |
|---|---|---|---|---|
| Recursive, size 200, no overlap | about 137 | 88.1 ± 30.1 | 7.0 ± 5.6 | 6.9 ± 5.6 |
| Recursive, size 400, no overlap | about 276 | 89.5 ± 29.7 | 3.6 ± 3.2 | 3.6 ± 3.2 |
| ClusterSemanticChunker, max 200 | about 103 | 87.3 ± 29.8 | 8.0 ± 6.0 | 8.0 ± 6.0 |
| ClusterSemanticChunker, max 400 | about 182 | 91.3 ± 25.4 | n/r | 4.5 ± 3.4 |
| Kamradt semantic default | about 660 | 83.6 ± 36.8 | n/r | 1.5 ± 1.6 |
| LLM chunker (GPT-4o) | about 240 | 91.9 ± 26.5 | n/r | 3.9 ± 3.2 |
In the rows that report them, smaller chunks had higher precision and IoU, while recall stayed between 83.6 and 91.9 in every row. The report says the recursive splitter at 200 with no overlap "performs well": while it "does not achieve the best result," it is "consistently high performing." It also warns that "default settings for certain popular chunking strategies can lead to relatively poor performance." One semantic row is Chroma's own ClusterSemanticChunker, so weigh it with the report's authorship in mind.
The limits are large. The embedding models date from 2024, retrieval was embedding-based rather than BM25, and the queries and excerpts were generated by an LLM. LLMs "tend to generate a specific style of question," and the method cannot find relevant excerpts the generator did not produce. The report itself calls its "relatively small size, and limited diversity" a limitation.
What Anthropic reported for contextual retrieval, and what it does not show
Contextual retrieval changes what each chunk contains, not where it is cut. In Anthropic's engineering post of 19 September 2024, an LLM writes a context of usually 50 to 100 tokens for each chunk. That context is prepended before embedding and before the BM25 index is built. The post measures failure as "1 minus recall@20."
| Configuration | Failure rate | Reduction |
|---|---|---|
| Baseline | 5.7% | none |
| Contextual embeddings | 3.7% | 35% |
| Contextual embeddings and contextual BM25 | 2.9% | 49% |
| Plus reranking | 1.9% | 67% |
These are Anthropic's own internal results. The headline is that the method can "reduce the number of failed retrievals by 49%" before any reranking. The experiments covered codebases, fiction, arXiv papers and science papers, used Gemini Text 004, and the final row adds a Cohere reranker. The post's charts and appendix are not in the text used here, so no BM25-only figure is available. The post's only explicit size guidance is that chunks are "usually no more than a few hundred tokens."
Why overlap is a trade-off
Overlap repeats the end of one chunk at the start of the next, so an answer cut at a boundary can appear whole in one of them. LangChain's parameter notes say overlapping chunks "helps to mitigate loss of information when context is divided between chunks." The cost is repetition: the same words are stored twice and can reach the model twice.
Chroma's data show both effects. With all-MiniLM-L6-v2, 250-token chunks with 125 tokens of overlap had recall of 82.4 ± 36.2, against 77.1 ± 39.3 with none. Elsewhere, the report notes that "reducing chunk overlap improves IoU scores." NVIDIA tested 10, 15 and 20 percent on one dataset and picked 15 percent, a result it does not present as a full grid search. Treat overlap as a value to test. At 1,024 tokens, 15 percent overlap repeats about 154 tokens at each boundary.
Why no strategy wins everywhere, so test on your documents
No strategy wins everywhere. NVIDIA's page-level chunks, which need page boundaries, had the highest average accuracy (0.648), but the post says page-level chunking is "not always the absolute best for every dataset." Its best token size varied: 1,024 tokens for FinanceBench (0.579, against 0.506 at 2,048) and 512 for Earnings (0.681, against 0.663 at 1,024).
Microsoft gives 512 tokens with 25 percent overlap in one place and 200 words or 600 characters with 10 to 15 percent in another, and it calls these starting points. It also notes that chunking "is only required if the source documents are too large" for a model's input.
To test your own documents, run the same questions through each setting with the same retriever. Pinecone's guide recommends the same approach: build one index per chunk size, or one index with namespaces, and run the same queries against each. The Chunking Strategy Comparator does this for seven settings with one lexical retriever, so chunking is the only variable. The chunk size test guide shows a 30-minute method.
What this comparison cannot tell you
This comparison shows structural effects under one retriever. It does not say which embedding model to buy, and hybrid retrieval or reranking can change the ranking. BEIR, a benchmark of 18 datasets, says that "re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs."
BM25 is still a fair test bed, and BEIR's authors call it "a robust baseline." A winner under one retriever is only a candidate, so re-run the best settings with the embedding model you plan to ship.
Common mistakes
- Trusting a copied default. The LangChain base splitter read for this guide defaults to 4,000 characters with 200 of overlap. Check your installed version.
- Comparing Chroma's recall with your recall@k. Chroma's recall counts tokens, so the two measure different things.
- Reading recall alone. The setting with the best recall may also return the most tokens to the model.
Common questions
What chunk size should I use for RAG?
No size is correct for every corpus. Microsoft suggests starting with 512 tokens and 25 percent overlap, and Pinecone suggests exploring smaller sizes such as 128 or 256 tokens. Chroma described its recursive setting at 200 tokens as "consistently high performing," though not the best result. Use these as starting points, then test against your own questions.
How much chunk overlap is enough?
The sources give starting points, not optimal values. Microsoft uses 25 percent in one example and 10 to 15 percent elsewhere. NVIDIA found 15 percent best in one tested setting. Test zero, 10, 15 and 20 percent on your own data, and watch how many tokens each setting returns.
Is semantic chunking better than fixed-size chunking?
Not in general. Chroma's Kamradt default setting scored "slightly below average across all metrics." Chroma's ClusterSemanticChunker at a 400-token maximum had recall of 91.3, against 87.9 for TokenText, a fixed token window of 800 tokens with 400 overlap. The sizes differ, so the comparison is not clean. Test both on the same questions.
What does recall@k measure?
In the ir-measures documentation, Recall@k is "The fraction of relevant documents for a query that have been retrieved by rank k." Some tasks instead ask whether any relevant result appears in the top k, which ir-measures calls Success@k.
What is the recursive character text splitter?
It is a LangChain splitter that tries separators in order, from paragraph breaks down to spaces, and keeps the largest unit that fits the size. Its size is counted in characters by default. Check the defaults in the version you install, because they can change.
Sources and further reading
- Chroma, "Evaluating Chunking Strategies for Retrieval," technical report, July 2024
- Anthropic, "Introducing Contextual Retrieval," engineering blog, 19 September 2024
- LangChain, "Text splitters" overview, undated (read 17 October 2026)
- LangChain, "Recursive character text splitter," undated (read 17 October 2026)
- LangChain, text-splitters source, master branch (read 17 October 2026)
- Microsoft Learn, "Chunk documents," last updated 2 July 2026
- NVIDIA Technical Blog, "Finding the Best Chunking Strategy for Accurate AI Responses," 18 June 2025
- Pinecone, "Chunking Strategies for LLM Applications," 28 June 2025
- LlamaIndex, "Node parser modules," developer documentation, undated
- Thakur et al., "BEIR," arXiv 2104.08663, accepted at NeurIPS 2021
- ir-measures, "Measures" reference, Terrier team, undated