RAG Chunking: Split Documents for Better Answers

Before a retrieval-augmented generation system can answer from your documents, it has to cut them into pieces it can search. How you make those cuts, known as RAG chunking, has a large effect on answer quality. Chunks that are too small lose the context that gives them meaning. Chunks that are too large mix several topics, dilute the match with the question and use up the model's context window. There is no single correct chunk size, but there is a sensible process for choosing one. This guide explains the main chunking approaches, how to handle different document types and how to test your choices against real questions.

Why RAG chunking matters

A RAG system turns each chunk into an embedding and, when a question arrives, retrieves the chunks whose embeddings are closest to the question's. If the right information is split across two chunks, neither may match well. If a chunk covers five topics, it may match many questions loosely and none precisely.

Pinecone's guide to chunking strategies puts it simply: chunks that are too small or too large can lead to imprecise search results or missed opportunities to surface relevant content. It offers a useful rule of thumb: if a chunk of text makes sense to a person without the surrounding context, it will make sense to the language model as well (Pinecone: Chunking strategies).

Fixed-size chunking

The simplest approach splits text into chunks of a set number of tokens or characters, often with some overlap between neighbouring chunks so that a sentence cut at a boundary appears in both. Pinecone describes fixed-size chunking as the most common and straightforward approach and recommends starting there in most cases, iterating only once you find it insufficient.

Fixed-size chunking is predictable and easy to implement. Its weakness is that it ignores structure: a chunk boundary can fall in the middle of a table, a list or a key sentence.

Structure-aware and recursive splitting

Better results often come from respecting a document's own structure. LangChain's text splitter documentation groups approaches into text structure-based, length-based and document structure-based splitting, and recommends starting with its RecursiveCharacterTextSplitter for most use cases because it balances keeping context intact with managing chunk size (LangChain: Text splitters). Pinecone explains that this splitter tries to split text using separators in a given order, such as paragraphs before sentences.

For structured formats, use the structure directly:

• Markdown and HTML: split on headings, and keep the heading path with each chunk.
• PDFs: extract text with layout awareness, and handle headers, footers and tables separately.
• Tables: keep rows together and repeat column headers, or convert rows into short sentences.
• Code: split by function or class rather than by character count.

Semantic chunking

A newer approach groups sentences by meaning, starting a new chunk when the topic shifts. Pinecone describes semantic chunking as an experimental technique. It can produce more coherent chunks for long, loosely structured text, at the cost of more processing and less predictable chunk sizes. Try it only after simpler methods have been tested.

Add context to every chunk

A chunk pulled out of a long document can lose crucial context, such as which product, policy version or customer it refers to. Two habits help:

• Store metadata with each chunk: source document, section heading, date, owner and access permissions.
• Prefix the chunk text with a short header, such as the document title and section path, before embedding it.

Metadata also lets you filter results, for example to the latest policy version or to documents a user may see. Our guide to private RAG explains why permissions should be applied at retrieval time.

Worked example: chunking plans per document type

The table below is a hypothetical chunking plan for a company building an internal assistant over several document types. The sizes are placeholders to test, not recommendations.

Document typeStructure availableChunking approachMetadata keptPolicy handbook (Markdown)Headings and listsSplit by heading; recursive split for long sectionsPolicy name, version, section pathProduct manuals (PDF)Headings, tables, page numbersLayout-aware extraction; tables kept wholeProduct, model, pageSupport ticketsShort threadsOne chunk per resolved ticket summaryProduct, date, resolution categoryContractsNumbered clausesOne chunk per clause with clause numberCustomer, contract date, clause IDMeeting notesLoose textRecursive split with overlapDate, team, attendees removed for privacy

The point is that different sources deserve different treatment. One global setting for every document type rarely works well.

Test chunk sizes with real questions

Choose chunking by evidence, not by guesswork. Pinecone suggests testing chunk sizes on a representative dataset, for example by indexing the same content with different sizes and comparing results. A practical process:

1. Collect a set of real questions with known correct source passages.
2. Index the documents with two or three chunking settings.
3. For each setting, check how often the correct passage appears in the top results.
4. Review failures to see whether chunks were too small, too large or badly split.
5. Pick the setting that performs best, then retest when documents or models change.

This kind of retrieval test is cheap compared with tuning prompts endlessly to compensate for poor chunks.

Keep the original text close

Whatever you embed, keep a link from each chunk back to the exact place in the source document. When the assistant answers, show that link so people can read the passage in context and check it. This builds trust, makes errors easy to report and helps your team see when a chunk boundary has cut off something important. It also gives reviewers a quick way to judge whether a poor answer came from retrieval or from the model.

Plan for updates

Documents change. When a policy is revised, its old chunks must be removed or marked as superseded, or the system may answer from outdated text. Store a document ID and version with every chunk, re-chunk documents when they change and delete chunks whose source no longer exists. Schedule a regular check that the index matches the source library.

Chunking and resilience

Good chunking also helps when parts of your system fail. If your main model or retrieval service is unavailable, a fallback with a smaller context window works better when chunks are compact and well labelled. Our guide to LLM failover covers how to keep RAG applications answering during outages.

Frequently Asked Questions

What is RAG chunking?

It is the process of splitting documents into smaller pieces that a retrieval-augmented generation system can embed, search and pass to a language model as context.

What chunk size should I use?

There is no universal answer. Start with a simple approach, then test two or three sizes against real questions and choose the one that retrieves the right passages most often.

Should chunks overlap?

Overlap can help when important sentences fall at a boundary, because they then appear in both neighbouring chunks. Test whether it improves results for your content.

How should tables be chunked?

Keep each table, or each row with its headers, together. Splitting tables by character count usually separates values from the columns that explain them.

Build retrieval that finds the right passage

We design RAG systems with document-specific chunking, useful metadata and retrieval tests that show what works. See our AI agent orchestration and RAG service, or talk to us about RAG chunking.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.