You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.
Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.
Size is a tradeoff, not a setting
The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.
Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.
Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.
Semantic splitting is oversold
The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.
A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.
What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.
The mistakes that cost you
A few failure modes show up again and again:
- Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
- Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
- Chasing the perfect chunk size before you have a single real question to test against.
That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.
Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.