RAG

Most AI bills are bloated with answers you already paid for. The same question comes in, phrased the same way, and you send it to the model again like you have never seen it before. Caching is how you stop doing that. It is the least glamorous cost lever there is and usually the most effective, and there are three different flavors that people constantly mix up.

Exact-match caching: the free lunch you skipped

The simplest version is a lookup table. Hash the exact input, store the model's output against that hash, and the next time the identical input arrives you return the stored answer instead of calling the model. No tokens spent, no latency, no provider involved. For anything with repeated identical queries, autocomplete suggestions, canned support answers, the same document summarized twice, this is close to free money.

The catch is the word exact. One extra space, a different capital letter, a trailing newline, and the hash changes and you miss the cache. So normalize before you hash. Trim whitespace, settle on a case convention, sort any parameters that do not have a meaningful order. And be honest about your hit rate. If every user query is genuinely unique, an exact cache buys you almost nothing, and you should not pretend otherwise.

Semantic caching: same meaning, different words

This is where it gets interesting and where it gets dangerous. "What is your refund policy?" and "How do I get my money back?" are different strings but the same question. Semantic caching catches that. You embed the incoming query into a vector, compare it against the vectors of things you have already answered, and if one is close enough you return that stored answer.

The whole game is the word "enough." You set a similarity threshold. Set it too loose and you will serve the answer to a neighboring question that is not actually the same, which is worse than a cache miss because it is a confidently wrong answer. "How do I cancel my subscription?" and "How do I pause my subscription?" sit very close in embedding space and mean different things with different consequences. Set the threshold too tight and you catch nothing and paid for the embedding infrastructure for no reason.

My rule: use semantic caching where a near-miss is cheap, like surfacing help articles, and stay away from it where a near-miss is expensive, like anything touching money, account state, or legal wording. Test it on real query logs, not on the three examples that made you want to build it.

Provider prompt caching: caching the prefix, not the answer

The third kind is different in nature. Provider prompt caching does not cache the final answer at all. It caches the model's processing of a long, repeated chunk of your input, typically a big system prompt, a set of tool definitions, or a document you keep asking questions about. The model still runs and still generates a fresh answer, but it skips re-reading the part it already read.

The economics are strong and the two big providers took opposite paths. Anthropic makes you mark the cacheable section explicitly with a cache breakpoint. A cache write costs a bit more than normal input, 1.25 times for the short-lived tier, but a cache read costs one tenth of the input price, a 90 percent discount, with a 1,024-token minimum. OpenAI does it automatically with no code change and reads at roughly half price. Anthropic asks for effort and rewards you more for it. OpenAI asks for nothing and rewards you less. Neither is wrong, they are just different bargains.

Prompt caching shines in exactly the case that would otherwise be brutal: a chat over a long document, or an agent carrying a fat system prompt through many turns. The repeated prefix gets cheap. The part that changes stays full price. Just know these caches are short-lived, minutes for the default tiers, so they help within a session and do nothing across a quiet night.

The pitfall that eats everyone: staleness

Here is the trap that turns a cost win into a support ticket. A cache is a bet that the right answer has not changed since you stored it. When the underlying truth moves and the cache does not, you serve yesterday's answer with today's confidence.

If your model answers over data that changes, prices, inventory, a policy doc, the account balance, your cache needs a way to know when that data moved. The clean move is to build the cache key out of a version of the source, so when the document updates, the key changes and old entries fall away on their own. The lazy move is a time-to-live: expire everything after an hour and accept up to an hour of staleness. TTL is fine for slow-moving content and quietly dangerous for anything a user expects to be current.

Start with exact-match caching, because it is safe and it works. Add prompt caching if you have long repeated prefixes, because the discount is real and the risk is near zero. Reach for semantic caching last, deliberately, and only where being a little bit wrong is genuinely fine. The savings are real. So is the bill from a cache that got too clever.

You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.

Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.

Size is a tradeoff, not a setting

The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.

Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.

Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.

Semantic splitting is oversold

The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.

A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.

What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.

The mistakes that cost you

A few failure modes show up again and again:

• Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
• Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
• Chasing the perfect chunk size before you have a single real question to test against.

That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.

Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.

A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.

What the window actually is

Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.

What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.

Lost in the middle

The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.

It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.

So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.

The parts nobody puts on the slide

Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.

This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.

When it earns its keep

Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.

The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.

Every few months a bigger model lands and the timeline decides intelligence just went up a notch. Meanwhile the people actually shipping features are not sitting around waiting for more parameters. They are fighting a different battle entirely, one that no benchmark score fixes: getting the right information in front of the model at the right moment. That, not size, is where most AI features live or die now.

The model is smart. It just does not know your stuff.

A frontier model has read a staggering slice of the public internet. What it has not read is your customer's last three support tickets, your internal pricing rules, the state of the order the user is asking about, or the document sitting in the tab next to your app. On everything that matters to your actual product, the smartest model on earth starts out ignorant, and no amount of extra parameters changes that. The knowledge it needs lives in your systems, and the job is delivery.

Watch where real failures come from. When an assistant confidently invents a policy, it is usually not because the model was too dumb to reason. It is because nobody put the real policy in front of it, so it filled the gap with something plausible. Swap in a bigger model and you get a more articulate wrong answer. Give a smaller model the right paragraph and it answers correctly. The bottleneck was never the brain. It was the briefing.

More context is not the same as better context

The obvious counter is that context windows are exploding, some models now take a million tokens or more, so just throw everything in and let the model sort it out. This does not work, and it is worth understanding why, because it kills the laziest version of the idea.

Long-context models do not attend evenly across everything you give them. The well-documented "lost in the middle" effect shows models reliably use what sits at the start and end of a long input while quietly glossing over the stuff buried in the middle. Stuff a huge context full and you can watch quality sag, not climb, as the signal gets diluted by noise the model has to wade through. There is a reason people now talk about "context rot," the way a model's grip loosens as the window fills with marginally relevant material. Dumping is not the answer. Curation is.

So the skill is not fetching more. It is fetching less, better. Retrieval that surfaces the three passages that matter instead of the thirty that might. A memory layer that remembers the two facts about this user that change the answer, and forgets the noise. Tool calls that pull a live value at the moment it is needed instead of a stale snapshot baked into a prompt an hour ago. Every one of those is a context problem wearing a different hat.

Why this is where the frontier actually is

Look at where serious effort is going and the shift is obvious. RAG, agents, tool use, memory systems, the whole apparatus is engineering to assemble the right context on the fly. None of it makes the base model smarter. All of it makes the model better-informed at the instant it answers, and that turns out to matter far more for whether your feature works.

This is also why two teams using the identical model ship wildly different products. The model is a commodity they both rent from the same API. The difference is entirely in what each team feeds it: how they chunk and rank their documents, when they call which tool, what they choose to remember and what they let go. That plumbing is the actual product. The model is the easy part, because you can buy it off a menu.

I am not claiming bigger models are pointless. A stronger model does more with a messy briefing and is more forgiving of a sloppy retrieval step, and that is real. But the returns from a better model are shrinking while the returns from better context are wide open, because most teams have barely started on the context side. If your AI feature is underperforming, the honest first question is almost never "do I need a bigger model." It is "did I actually give it what it needed to answer." Nine times out of ten, you did not, and that is a problem you can fix this week without waiting for anyone's next release.

An embedding is a list of numbers that stands in for a piece of text. Feed a sentence to an embedding model and it hands back a few hundred or a few thousand numbers. That list is the whole trick. Text that means similar things lands on similar lists, and text that means different things lands far apart. Once you accept that one idea, most of the useful applications fall out on their own.

The reason this matters is that computers are good at math and bad at meaning. Keyword search knows that "car" and "car" match. It has no idea that "car" and "automobile" are the same thing, or that "my sedan will not start" is closer to "vehicle trouble" than to "sedan chair." Embeddings give the machine a way to measure closeness in meaning instead of closeness in spelling.

How closeness actually works

Picture every sentence as a point in space. Not the three dimensions you live in, but a space with hundreds of them. You cannot draw that, and you do not need to. What you need is a way to ask how close two points are, and the usual answer is cosine similarity: it measures the angle between two of these lists of numbers and returns a score, roughly from minus one to one. Near one means the two texts point the same direction, which is to say they mean roughly the same thing. Near zero means they are unrelated.

You never look at the raw numbers. Nobody reads an embedding. You compute it, store it, and later ask the database which stored vectors sit nearest to a new one. That single operation, find the nearest points, is the engine under almost everything below.

The jobs embeddings are good at

Search is the obvious one, and the one most people meet first. You embed all your documents once. When a user types a question, you embed the question and pull back the passages whose vectors are nearest. The user searches by meaning, so "how do I cancel my plan" finds the paragraph titled "ending a subscription" even though they share almost no words.

Deduplication is quieter but pays for itself fast. If you have a support inbox, a product catalog, or a pile of scraped articles, near-duplicate items are everywhere and exact matching misses them. Embed everything, look for pairs whose similarity crosses a threshold you pick, and you catch the "same thing, slightly reworded" cases that a string comparison walks right past.

Clustering is what you reach for when you do not know your categories yet. Embed a few thousand customer messages, group the vectors that huddle together, and read a handful from each group. The themes come to you instead of you guessing them in advance. It is a genuinely nice way to find out what people are actually writing about.

And then there is retrieval for RAG, which is the same search step wearing a job title. Before a language model answers, you embed the question, fetch the most relevant chunks of your own documents, and hand them to the model as context. The retrieval half of retrieval-augmented generation is embeddings doing exactly what they do in search. Nothing more exotic than that.

Where it gets you into trouble

Embeddings measure similarity, and similarity is not the same as relevance, correctness, or intent. Two sentences can sit close in vector space and mean opposite things, because "the drug is safe" and "the drug is not safe" share almost every word. Negation, sarcasm, and small but load-bearing details are exactly where cosine similarity gets sloppy. The models improve on this every year, but do not assume the nearest neighbor is the right answer. It is the closest guess.

A few practical notes. The model you embed with matters more than the number of dimensions; a good small model beats a mediocre large one. You have to embed your query and your documents with the same model, or the comparison is meaningless. And thresholds are not universal. A similarity of 0.8 might mean "basically identical" for one model and "vaguely related" for another, so test against your own data before you trust a cutoff.

If you take one thing away, let it be this: embeddings are a ruler for meaning. Not a truth machine, not a reasoning engine, just a very good way to ask what is near what. Most of the time that is precisely the question you had.

When a model is not doing what you want, there are three levers people pull: better prompting, retrieval, or fine-tuning. They are wildly different in cost and effort, and teams reach for the expensive one far too early. Here is how to pick the right lever without burning your budget on the wrong one.

"Memory" is one of the most oversold words in AI right now. Products promise an assistant that remembers you, learns your preferences, picks up where you left off. Then you use it for a week and notice it forgot the thing you told it on Tuesday. The gap between the pitch and the reality comes from a fact people gloss over: a language model has no memory at all.

The model is a function. Text goes in, text comes out, and the moment the response finishes it retains nothing. Everything we call memory is scaffolding we build around that stateless core, deciding what to put back into the input next time. There are only a few honest ways to do it, and each one breaks in its own way.

Stuffing the context window

The crudest approach: keep the whole conversation and paste it back in every turn. For a short chat this is perfect. The model "remembers" because you literally handed it the transcript.

It falls apart on two edges. Context windows are finite, so a long enough conversation eventually will not fit. And even when it fits, more context is not free. You pay for every token on every call, and models get measurably worse at finding the one relevant line when it is buried in tens of thousands of tokens of history. Stuffing is a real technique with a hard ceiling, not a memory system.

Summaries and their slow leak

So you compress. Every so often, ask the model to summarize the conversation so far and carry the summary forward instead of the raw text. This buys a lot of runway, and most chat products do some version of it.

The catch is that summarizing is lossy on purpose, and you do not get to choose what it loses. The detail that seemed irrelevant when the summary was written is exactly the one that matters three turns later. Summaries of summaries drift further each round, like a photocopy of a photocopy. Good for the gist, unreliable for the specific fact.

Retrieval, which is what "memory" usually means now

The approach that actually scales is to stop trying to hold everything in the prompt. Write facts and past exchanges to a store, and when a new message comes in, fetch only the handful of pieces relevant to it and drop those into the context. This is RAG pointed at your own history instead of a document library.

It is the best of the three and still not what people imagine. Retrieval memory only recalls what it thought to save and what the query happens to match. Phrase your question differently than the stored fact was written and the right memory may never surface. It cannot form the kind of connection a person makes across two things noticed months apart, because nothing is reasoning over the whole store, it is just fetching nearest matches. What you get is a good filing clerk, not a mind.

Here is the thing I wish more products said plainly. None of these give a model memory in the human sense, the kind that reshapes understanding over time. They are all variations on one move: choose what text to put in front of a stateless function on the next call. That is a real and useful engineering problem, and you can build genuinely helpful things by solving it well.

So when you design memory, do not chase the fantasy of an assistant that just knows you. Decide, deliberately, what is worth remembering, how it gets written down, and how it gets found again. Keep the recent turns raw, summarize the middle distance, retrieve the long tail, and accept that each layer forgets in a different way. The teams who ship memory that feels good are not the ones with a magic store. They are the ones who were honest about what a model is and engineered around it.

RAG, retrieval-augmented generation, is one of those terms that sounds like it needs a PhD and actually describes something you could explain to a ten-year-old. Here is the plain version, why it matters, and the case, more common than the vendors admit, where you do not need it at all.

The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.

What a vector database actually does

Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.

That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.

When a plain file is genuinely enough

Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.

An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.

The rough line in my head:

• Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
• Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.

Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.

The middle ground people skip

Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.

Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.

My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.

A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.

The machine has no idea what is true

Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.

That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.

This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.

We trained it to guess

Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.

Kinds of hallucination, and what dents them

They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.

You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.

The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.