Chunking Documentation for RAG Pipelines

Chunking documentation for RAG pipelines is not a solved problem with one right answer. It is a matching exercise: the right way to split a document depends on what that document looks like structurally and what kind of questions people actually ask against it, and the industry has spent the past two years generating enough benchmark data to make that matching exercise tractable rather than guesswork.

Retrieval-augmented generation has gone from a niche architecture pattern to standard infrastructure fast. Menlo Ventures found that 51% of enterprises were running RAG in production in 2024, up from 31% the year before, and the market around it is projected to grow several times over from 2025 to 2030, a 38.4% compound annual growth rate, which amounts to a lot of teams shipping retrieval systems. That is a lot of teams shipping retrieval systems. And yet chunking, the step where source documents get cut into pieces before embedding, remains the part of the pipeline that gets the least scrutiny. Teams will spend weeks A/B testing embedding models and rerankers, then split their documents with whatever default came in the tutorial they followed.

That is a mistake the data does not support. A Vectara study presented at NAACL 2025 tested 25 different chunking configurations across 48 embedding models and found that chunking configuration influenced retrieval quality as much as, or more than, the choice of embedding model itself. Separately, fast.io has reported that getting chunk size wrong by even one size bracket can cost 15 to 30 percentage points of context precision. A PMC Bioengineering study from 2025 ran four otherwise identical RAG pipelines, same model, same data, same prompt, and varied only the chunking strategy. Accuracy ranged from 50% to 87%. Same everything else. That range alone should settle the argument that chunking is a minor implementation detail.

This piece walks through the strategies that actually show up in benchmarks and production systems, in roughly the order a team should consider them: a defensible default, a structure-aware upgrade, a way to resolve the tension between precision and context, the semantic chunking approach and what its numbers really mean, LLM-augmented methods for when the stakes justify the cost, and what happens when documentation includes tables and diagrams rather than plain prose. Overlap gets its own section at the end, because it is the setting almost nobody tunes with evidence.

What chunking does inside a RAG pipeline

Chunking is the act of splitting a source document into smaller segments before those segments get embedded and stored for retrieval. That sounds almost too simple to warrant explanation. But consider what the embedding model actually does with each chunk: it compresses the entire piece of text into a single vector. If a chunk contains two unrelated ideas, the resulting vector represents neither one well; it is in some blurry middle ground that matches poorly against queries about either topic. If the chunk gets cut off mid-sentence, the embedding captures a fragment of a thought instead of the thought itself.

Every downstream step, vector search, reranking, generation, inherits whatever the chunking step handed it. There is no later stage that can recover information that got split apart badly at the start. A reranker can reorder five retrieved chunks by relevance, but it cannot stitch together a table header from one chunk and its data rows from another. Chunking errors do not stay contained. They compound.

Teams also tend to conflate two different things when they talk about "retrieval quality." One is retrieval recall: how much of the actually relevant text the retriever manages to surface. The other is end-to-end accuracy: whether the LLM, given what got retrieved, produces the correct answer. These are not the same measurement, and optimizing for one can actively hurt the other. A chunk that scores beautifully on recall can still be a bad chunk if it is too thin on context for the model to reason with. This distinction becomes important later, particularly in the semantic chunking section, where the recall numbers look great and the accuracy numbers do not.

Query pattern matters just as much as document type, and it is the variable most teams skip past. Factoid queries, the single-fact lookups like "what's the default timeout value," tend to perform best against chunks in the 256 to 512 token range. Multi-hop analytical queries, the kind that require connecting two or three pieces of information across a document, do better with larger chunks, somewhere around 512 to 1,024 tokens. So before picking a chunking strategy, it helps to know: what do people mostly ask this system? That, combined with what the source documents structurally look like, is the actual decision surface the rest of this piece is mapping.

Chunking cannot fix bad source data. Atlan has pointed out that a large share of production RAG failures trace back to stale, ungoverned, or semantically thin source content, not boundary placement. If the underlying documentation is out of date or thin on detail, no chunking strategy, however sophisticated, will produce good answers from it.

Recursive character splitting as the defensible starting point

As of 2026, the closest thing the field has to a consensus default is recursive character splitting at 512 tokens with 50 to 100 tokens of overlap. Enough independent benchmarks recommend it as the starting point that it earns the label "defensible," meaning nobody can fault a team for starting there.

The mechanism is straightforward once you see it. The algorithm tries a list of separators in priority order: paragraph breaks first, then sentence breaks, then word boundaries, and only as a last resort individual characters. It looks for the most meaningful break point it can find within the target chunk size, so it avoids cutting mid-word or mid-sentence whenever a better boundary exists nearby. It is not smart in any deep sense. It just respects the shape of prose.

The results back it up. A FloTorch benchmark from February 2026, run against 50 academic papers, found recursive splitting at 512 tokens scoring 69% end-to-end answer accuracy, which beat every more expensive alternative tested in that study. Chroma Research separately measured retrieval recall in the 85.4% to 89.5% range for recursive splitting, with the best performance, 88.1% to 89.5%, landing around 400 tokens. Microsoft's Azure documentation recommends 512 tokens with 25% overlap (128 tokens), measured in BERT tokens rather than raw character counts. Arize AI has found that chunk sizes between 300 and 500 tokens, paired with retrieving the top 4 results (K=4), gives the best balance of speed and quality for most applications.

Why does something this simple keep beating fancier methods? Partly cost: recursive splitting makes zero model calls at chunking time, so it is nearly free to run at any scale. Partly quality: it preserves semantic coherence better than naive fixed-size splitting, which just counts characters and cuts wherever the count lands. And partly convenience: it ships as a drop-in function in LangChain and LlamaIndex, so there is no custom infrastructure to build.

It has real limits, though. Recursive splitting has no awareness of document structure. It does not know a heading from a paragraph, and it will happily merge two adjacent, unrelated paragraphs into one chunk if they both fit inside the token budget. It also does not understand tables or code blocks, it will split a function definition wherever the character count runs out. That gap is exactly what the next section addresses. For now, the practical instruction is simple: start with recursive splitting, run it against your own corpus, and only reach for something more complex once you can point to a specific failure mode this default produces on your specific documents.

When document structure should drive chunk boundaries instead

Document-aware, or structure-aware, chunking flips the approach. Instead of counting tokens and looking for the nearest sentence break, it first parses the document's structural elements, headers, sections, paragraphs, tables, code blocks, list items, and uses those elements as the chunk boundaries themselves.

A table stays together as one unit rather than getting split between its header row and its data. A code block never gets cut in the middle of a function. An H2 heading becomes a reliable, predictable place to start a new chunk. This is a different philosophy from recursive splitting: instead of asking "where's the nearest good break point within my token budget," it asks "what did the document's author already tell me about its own structure, and can I just use that."

This approach is the right default for a specific and fairly common category of content: Markdown documentation, PDFs where the structure is machine-detectable, HTML pages, Jupyter notebooks, technical manuals, and data dictionaries. Notice what that list adds up to. It is essentially the format most developer-facing documentation already takes.

The reason this matters so much for documentation specifically comes down to a particular failure mode. A chunk containing a table's header row but none of its data returns from a retrieval query looking confident and relevant, and then answers nothing useful once the LLM tries to work with it. Same story with a code example that gets separated from the paragraph explaining what it does. Structure-aware chunking avoids this entire class of failure, and it does so without a single model call, since it is just parsing markup that is already there.

For structured content types, structure-aware chunking offers strong retrieval effectiveness relative to implementation cost. It beats recursive splitting on precision without adding meaningful complexity, at least for content that is already well-formatted.

That is the catch, though. This method depends entirely on the source document actually having detectable structure. A PDF that was scanned and OCR'd, or converted sloppily from some other format, may not expose reliable headers or section breaks for the parser to find. In that case structure-aware chunking has nothing to grab onto, and it degrades toward whatever fallback logic the implementation uses.

Documentation is also a living, constantly updated thing. Documentation that changes weekly or monthly creates a re-chunking problem: every edit potentially shifts chunk boundaries. Structure-aware chunking does not eliminate that problem, but it does guarantee that when a section gets rewritten, the resulting chunk boundary lands at a semantically meaningful point, an actual heading or paragraph break, rather than an arbitrary character count that happened to fall in a different place after the edit.

The precision-context trade-off and how hierarchical chunking resolves it

Every chunking decision runs into the same tension eventually. Small chunks match queries with precision, because a tightly scoped chunk about one specific fact will score highly against a query about that exact fact. Large chunks give the LLM enough surrounding context to actually construct a correct, well-reasoned answer. A single chunk size cannot fully satisfy both goals at once, because what makes retrieval precise is exactly what makes generation context-poor.

Hierarchical chunking, sometimes called parent-child chunking, resolves this by refusing to pick one size. It builds two layers instead. Small "child" chunks, typically 128 to 256 tokens, get used for the actual retrieval step, since their narrow scope makes them match queries precisely. Larger "parent" chunks, typically 512 to 1,024 tokens, sit behind those child chunks and get handed to the LLM once a child chunk is selected. Retrieval stays sharp. Generation gets the fuller picture.

This has become, by most accounts including analysis from Atlan, the most widely adopted pattern in production RAG systems through 2025 and into 2026. A concrete implementation appears in the H-RAG system described at SemEval-2026: documents get segmented into small overlapping child chunks, with both the child chunks and the parent documents indexed together in a hybrid vector store.

The cost is structural, not computational. Running two index layers, one for children, one for parents, means maintaining two things instead of one, and it means every time a source document changes, both layers need updating in sync. For documentation that changes often, every time a source document changes, both layers need updating in sync, creating ongoing engineering work. It is not a hypothetical inconvenience, it is ongoing engineering work every time content gets revised.

So when does a team know it is time to move to this pattern? The symptom is fairly distinctive: retrieval is working, the system is finding the right chunk, but the generated answers come back incomplete or shallow anyway. That is the signature of a chunk size tuned for precision at the expense of context, and it is exactly the problem hierarchical chunking is built to solve.

Semantic chunking: what the recall numbers hide

Semantic chunking works by embedding individual sentences, measuring the cosine similarity between each sentence and the one before it, and inserting a new chunk boundary wherever that similarity drops below some threshold. The idea is elegant: chunks end up representing coherent topics rather than arbitrary token counts, because the boundary gets drawn exactly where the subject matter shifts.

Here is where the data gets genuinely confusing if you only look at one number. Chroma Research found that its LLMSemanticChunker achieved 0.919 recall and its ClusterSemanticChunker hit 0.913, the best retrieval recall scores of any method in that study. On the surface, that reads like semantic chunking wins outright. But FloTorch's 2026 benchmark found semantic chunking scoring only 54% on end-to-end answer accuracy, compared to recursive splitting's 69%. Same general category of technique, wildly different verdicts.

These two findings are not actually in conflict. They are measuring different failure points along the same pipeline. FloTorch's semantic chunking configuration produced fragments averaging just 43 tokens each. Those fragments were tiny and topically pure. That is exactly why they retrieved so cleanly, matching queries with surgical precision. But 43 tokens is not enough material for an LLM to construct a full answer from. High recall, wrong answer. The chunk did its retrieval job and then failed at the only thing that actually matters to the person asking the question.

The Vectara study from NAACL 2025 backs this up at a larger scale: on realistic document sets, fixed-size chunking consistently outperformed semantic chunking across document retrieval, evidence retrieval, and answer generation, and the extra computational cost of semantic chunking was not justified by the results.

There is a practical fix, and fast.io recommends it directly: set a minimum chunk size floor, something around 200 tokens, and merge any fragment that falls below that floor into an adjacent one before indexing. That single guardrail prevents the 43-token collapse that sank FloTorch's numbers.

Cost is a separate consideration entirely. Semantic chunking requires computing an embedding for every sentence during ingestion, which is meaningfully more expensive than recursive splitting's zero-model-call approach. That expense is worth paying for high-value, heterogeneous documents where topic density genuinely varies unpredictably from paragraph to paragraph, think a compiled knowledge base pulled from many different sources. It is a poor trade for uniformly structured documentation where the topic density is already fairly consistent.

The broader lesson sits in that recall gap. A roughly nine-point recall advantage sounds decisive until you remember that recall alone does not answer the question anyone actually cares about, whether the system gave a correct answer.

LLM-augmented approaches: contextual retrieval and adaptive chunking

Contextual retrieval takes a different approach to the same underlying problem: instead of trying to draw better boundaries, it enriches each chunk with information about where it sits in the larger document. An LLM reads the full source document and generates a short header, typically 50 to 100 tokens, explaining what role that particular chunk plays within the document as a whole. That header gets prepended to the chunk before embedding.

The gains reported are meaningful. Engineering teams have reported improvements in retrieval precision across varied datasets using this technique, according to the paper at arXiv:2601.05265. Anthropic's own published work on Contextual Retrieval reportedly reduced top-20 retrieval failures by up to 67% when paired with a reranking step. That is a substantial jump.

It comes at a real cost, though: one LLM call per chunk during ingestion. For a corpus with a few hundred documents, that is a manageable, one-time expense. For a corpus with tens of thousands of documents that gets re-indexed weekly, the ingestion bill and the pipeline latency both grow in a way that becomes hard to justify.

A related but distinct approach is adaptive or LLM-driven chunking, exemplified by LumberChunker, described by Duarte and colleagues at EMNLP 2024. Rather than adding context after the fact, it prompts an LLM directly to identify where the natural content transitions fall in a long piece of narrative text, then draws chunk boundaries there. The paper reports higher QA accuracy than fixed-size or simpler splitting methods on the texts it was tested against.

A domain-specific data point makes the potential upside vivid. A PMC Bioengineering study found that adaptive chunking aligned to logical topic boundaries achieved 87% accuracy against a 13% baseline for fixed-size chunking, a gap statistically confirmed at p = 0.001. That is a massive spread. That number comes from clinical and post-operative query data specifically, and there is no basis for assuming a gap that large would show up in, say, API documentation or a product FAQ.

So where do LLM-augmented methods actually earn their cost? Small, high-value corpora that rarely change, where retrieval precision genuinely carries mission-critical weight. Domain-specific content, clinical, legal, compliance, where getting a topic boundary wrong has real consequences. What they are not suited for is large-scale documentation that updates on a weekly or daily cadence, where the ingestion cost and the re-chunking burden on every update compound into something operationally unsustainable.

Late chunking is a middle ground. It embeds the entire document first and then derives chunks from token-level embeddings afterward, which avoids the LLM call that contextual retrieval requires and is correspondingly cheaper. But according to a paper from April 2025 (arXiv:2504.19754), contextual retrieval preserves semantic coherence more effectively than late chunking does. Late chunking is more efficient to run, but it tends to trade away some relevance and completeness to get there, and its practical ceiling is set by whatever context window the embedding model supports.

When documentation contains tables, diagrams, and multi-page structure

Every method discussed so far assumes the document is, at bottom, text. That assumption breaks down fast once documentation includes multi-page tables, embedded figures, procedural steps illustrated with screenshots, or information that depends on context spanning across a page boundary. Traditional text-based chunking, of any flavor, simply cannot represent an image or reliably keep a table's full structure intact across pages.

Multimodal document chunking addresses this by using Large Multimodal Models to process PDF documents in configurable batches of pages, aiming to preserve both semantic coherence and structural integrity across that batch, an approach described in the paper at arXiv:2506.16035. What this handles that pure text chunking cannot: a table that spans three pages and needs to stay conceptually whole, a diagram alongside the paragraph that references it, a numbered procedure where a screenshot illustrates step 4 and needs to travel with the text describing step 4.

On the embedding side, Cohere's Embed 4 model, released in April 2025, was built with this problem in mind. It accepts interleaved text and images directly, processes raw PDF pages without requiring a separate parsing step beforehand, and supports a context window of 128,000 tokens, enough to cover roughly a 200-page document in a single embedding call, according to reporting from BigData Boutique.

This is not a strategy every documentation pipeline needs. Most text-heavy documentation, API references, conceptual guides, FAQs, gets nothing extra from multimodal handling and would just be paying more for infrastructure it does not use. It becomes the right answer specifically when the source material has non-trivial visual or tabular content that a text-only pipeline would otherwise discard or, worse, silently corrupt by splitting it in the wrong place.

The signal that a team has crossed into this territory is fairly recognizable: retrieval finds the right section of the document, the section really does contain the answer, and the LLM still cannot produce it, because the answer lives inside a table or a figure that the text chunk never captured in the first place.

Overlap: the setting teams configure by assumption rather than by evidence

Almost every chunking strategy discussed above involves a second number sitting quietly next to chunk size: overlap. And overlap is, by a wide margin, the setting most teams configure on instinct rather than on any actual test against their own data.

The logic behind overlap is sound in principle. If a chunk boundary happens to fall in the middle of an idea, some meaning gets lost right at the seam. Overlap, repeating the last 50 to 100 tokens of one chunk at the start of the next, is meant to paper over that seam so no idea gets fully orphaned at a boundary. Microsoft's Azure guidance suggests 25% overlap relative to chunk size; other sources converge on the same 50 to 100 token range for a 512-token chunk size, which itself is a little odd if you think about it, since a fixed overlap size interacts very differently with a 256-token chunk than it does with a 1,024-token one.

What is notably absent across the literature surveyed here is a rigorous, isolated study of overlap as its own variable. The benchmarks that report strong numbers, FloTorch's 69% accuracy figure, Chroma's recall scores, do not isolate overlap as an independent variable at some reasonable value while varying chunk size or chunking method. That is methodologically sound for answering the question those studies were built to answer. But it also means the 50-to-100-token convention has become something closer to received wisdom than an empirically isolated best setting.

That should not be read as an argument against overlap. Cutting a document into non-overlapping chunks with zero redundancy risks losing an idea entirely at every seam, and that is a real failure mode. The overlap value most teams ship with is inherited from tutorials and default library settings, not from a controlled test against their own documents and their own query patterns. Given how much the earlier sections established about chunk size and structure mattering enormously to outcomes, treating overlap as the one variable safe to leave on autopilot deserves at least a second look.