RAG

You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.

Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.

Size is a tradeoff, not a setting

The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.

Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.

Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.

Semantic splitting is oversold

The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.

A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.

What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.

The mistakes that cost you

A few failure modes show up again and again:

• Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
• Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
• Chasing the perfect chunk size before you have a single real question to test against.

That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.

Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.

A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.

What the window actually is

Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.

What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.

Lost in the middle

The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.

It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.

So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.

The parts nobody puts on the slide

Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.

This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.

When it earns its keep

Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.

The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.

An embedding is a list of numbers that stands in for a piece of text. Feed a sentence to an embedding model and it hands back a few hundred or a few thousand numbers. That list is the whole trick. Text that means similar things lands on similar lists, and text that means different things lands far apart. Once you accept that one idea, most of the useful applications fall out on their own.

The reason this matters is that computers are good at math and bad at meaning. Keyword search knows that "car" and "car" match. It has no idea that "car" and "automobile" are the same thing, or that "my sedan will not start" is closer to "vehicle trouble" than to "sedan chair." Embeddings give the machine a way to measure closeness in meaning instead of closeness in spelling.

How closeness actually works

Picture every sentence as a point in space. Not the three dimensions you live in, but a space with hundreds of them. You cannot draw that, and you do not need to. What you need is a way to ask how close two points are, and the usual answer is cosine similarity: it measures the angle between two of these lists of numbers and returns a score, roughly from minus one to one. Near one means the two texts point the same direction, which is to say they mean roughly the same thing. Near zero means they are unrelated.

You never look at the raw numbers. Nobody reads an embedding. You compute it, store it, and later ask the database which stored vectors sit nearest to a new one. That single operation, find the nearest points, is the engine under almost everything below.

The jobs embeddings are good at

Search is the obvious one, and the one most people meet first. You embed all your documents once. When a user types a question, you embed the question and pull back the passages whose vectors are nearest. The user searches by meaning, so "how do I cancel my plan" finds the paragraph titled "ending a subscription" even though they share almost no words.

Deduplication is quieter but pays for itself fast. If you have a support inbox, a product catalog, or a pile of scraped articles, near-duplicate items are everywhere and exact matching misses them. Embed everything, look for pairs whose similarity crosses a threshold you pick, and you catch the "same thing, slightly reworded" cases that a string comparison walks right past.

Clustering is what you reach for when you do not know your categories yet. Embed a few thousand customer messages, group the vectors that huddle together, and read a handful from each group. The themes come to you instead of you guessing them in advance. It is a genuinely nice way to find out what people are actually writing about.

And then there is retrieval for RAG, which is the same search step wearing a job title. Before a language model answers, you embed the question, fetch the most relevant chunks of your own documents, and hand them to the model as context. The retrieval half of retrieval-augmented generation is embeddings doing exactly what they do in search. Nothing more exotic than that.

Where it gets you into trouble

Embeddings measure similarity, and similarity is not the same as relevance, correctness, or intent. Two sentences can sit close in vector space and mean opposite things, because "the drug is safe" and "the drug is not safe" share almost every word. Negation, sarcasm, and small but load-bearing details are exactly where cosine similarity gets sloppy. The models improve on this every year, but do not assume the nearest neighbor is the right answer. It is the closest guess.

A few practical notes. The model you embed with matters more than the number of dimensions; a good small model beats a mediocre large one. You have to embed your query and your documents with the same model, or the comparison is meaningless. And thresholds are not universal. A similarity of 0.8 might mean "basically identical" for one model and "vaguely related" for another, so test against your own data before you trust a cutoff.

If you take one thing away, let it be this: embeddings are a ruler for meaning. Not a truth machine, not a reasoning engine, just a very good way to ask what is near what. Most of the time that is precisely the question you had.

When a model is not doing what you want, there are three levers people pull: better prompting, retrieval, or fine-tuning. They are wildly different in cost and effort, and teams reach for the expensive one far too early. Here is how to pick the right lever without burning your budget on the wrong one.

"Memory" is one of the most oversold words in AI right now. Products promise an assistant that remembers you, learns your preferences, picks up where you left off. Then you use it for a week and notice it forgot the thing you told it on Tuesday. The gap between the pitch and the reality comes from a fact people gloss over: a language model has no memory at all.

The model is a function. Text goes in, text comes out, and the moment the response finishes it retains nothing. Everything we call memory is scaffolding we build around that stateless core, deciding what to put back into the input next time. There are only a few honest ways to do it, and each one breaks in its own way.

Stuffing the context window

The crudest approach: keep the whole conversation and paste it back in every turn. For a short chat this is perfect. The model "remembers" because you literally handed it the transcript.

It falls apart on two edges. Context windows are finite, so a long enough conversation eventually will not fit. And even when it fits, more context is not free. You pay for every token on every call, and models get measurably worse at finding the one relevant line when it is buried in tens of thousands of tokens of history. Stuffing is a real technique with a hard ceiling, not a memory system.

Summaries and their slow leak

So you compress. Every so often, ask the model to summarize the conversation so far and carry the summary forward instead of the raw text. This buys a lot of runway, and most chat products do some version of it.

The catch is that summarizing is lossy on purpose, and you do not get to choose what it loses. The detail that seemed irrelevant when the summary was written is exactly the one that matters three turns later. Summaries of summaries drift further each round, like a photocopy of a photocopy. Good for the gist, unreliable for the specific fact.

Retrieval, which is what "memory" usually means now

The approach that actually scales is to stop trying to hold everything in the prompt. Write facts and past exchanges to a store, and when a new message comes in, fetch only the handful of pieces relevant to it and drop those into the context. This is RAG pointed at your own history instead of a document library.

It is the best of the three and still not what people imagine. Retrieval memory only recalls what it thought to save and what the query happens to match. Phrase your question differently than the stored fact was written and the right memory may never surface. It cannot form the kind of connection a person makes across two things noticed months apart, because nothing is reasoning over the whole store, it is just fetching nearest matches. What you get is a good filing clerk, not a mind.

Here is the thing I wish more products said plainly. None of these give a model memory in the human sense, the kind that reshapes understanding over time. They are all variations on one move: choose what text to put in front of a stateless function on the next call. That is a real and useful engineering problem, and you can build genuinely helpful things by solving it well.

So when you design memory, do not chase the fantasy of an assistant that just knows you. Decide, deliberately, what is worth remembering, how it gets written down, and how it gets found again. Keep the recent turns raw, summarize the middle distance, retrieve the long tail, and accept that each layer forgets in a different way. The teams who ship memory that feels good are not the ones with a magic store. They are the ones who were honest about what a model is and engineered around it.

RAG, retrieval-augmented generation, is one of those terms that sounds like it needs a PhD and actually describes something you could explain to a ten-year-old. Here is the plain version, why it matters, and the case, more common than the vendors admit, where you do not need it at all.

The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.

What a vector database actually does

Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.

That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.

When a plain file is genuinely enough

Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.

An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.

The rough line in my head:

• Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
• Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.

Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.

The middle ground people skip

Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.

Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.

My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.

A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.

The machine has no idea what is true

Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.

That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.

This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.

We trained it to guess

Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.

Kinds of hallucination, and what dents them

They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.

You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.

The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.