RAG, retrieval-augmented generation, is one of those terms that sounds like it needs a PhD and actually describes something you could explain to a ten-year-old. Here is the plain version, why it matters, and the case, more common than the vendors admit, where you do not need it at all.
What it actually is
A language model only knows what it was trained on, plus whatever you paste into the prompt. RAG is just the second part, automated. Before the model answers, you search your own documents for the relevant bits and stuff them into the prompt. The model then answers using that fresh, specific material instead of its stale general memory.
That is the whole idea. Search, then generate. The fancy name hides a simple move: give the model the right page before you ask the question.
Why it helps
- Current and private knowledge. The model can answer about your internal docs, your product, last week meeting, none of which it was trained on.
- Fewer confident lies. Grounded in real retrieved text, the model makes things up less. Not never. Less.
- Citations. You know which document the answer came from, so you can check it.
When you do not need it
If the relevant information fits comfortably in the prompt, just paste it. For a single contract, one policy, a handful of pages, you do not need a retrieval pipeline, a vector database, and a diagram with six boxes. You need copy and paste. RAG earns its complexity when you have too much material to paste and need to find the right slice automatically.
Half the RAG systems in production are an elaborate solution to a problem that a bigger paste would have solved.
The honest trade-offs
RAG adds moving parts. Your retrieval has to actually find the right chunks, and when it does not, the model answers confidently from the wrong ones, which can be worse than no retrieval at all. Good RAG lives or dies on the search quality, not the model. People obsess over the model and neglect the retrieval, then wonder why the answers are off.
The takeaway
Reach for RAG when you have a large, changing, or private body of knowledge the model needs and cannot hold in a prompt. For anything smaller, paste the context and skip the architecture. The goal is a good answer, not an impressive diagram.