AI Guides & Tutorials

• LLMs
• RAG

The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.

What a vector database actually does

Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.

That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.

When a plain file is genuinely enough

Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.

An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.

The rough line in my head:

• Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
• Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.

Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.

The middle ground people skip

Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.

Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.

My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.

• LLMs
• RAG

You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.

Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.

Size is a tradeoff, not a setting

The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.

Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.

Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.

Semantic splitting is oversold

The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.

A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.

What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.

The mistakes that cost you

A few failure modes show up again and again:

• Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
• Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
• Chasing the perfect chunk size before you have a single real question to test against.

That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.

Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.

• LLMs
• RAG

A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.

The machine has no idea what is true

Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.

That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.

This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.

We trained it to guess

Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.

Kinds of hallucination, and what dents them

They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.

You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.

The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.

• LLMs
• Coding

You wire a language model into your app, ask it to "respond in JSON," and it does. It works the first ten times. Then, on a Tuesday, it wraps the JSON in a markdown code fence, or opens with "Sure, here is the data," or trails off with a comment, and your parser throws, and the pager goes off. "Just ask nicely" is not a strategy. It is a demo that has not failed yet.

The reason it fails is structural. A base language model predicts likely text, and there are a thousand plausible ways to wrap a JSON object in friendly words. A tiny fraction of your calls will land on one of those, and a tiny fraction of a large number is a lot of broken responses. You need something stronger than a polite request.

Say what shape you want, exactly

The first move is to stop describing the output in prose and start defining it as a schema. A JSON Schema, or a typed object in your language that compiles to one, spells out the fields, their types, which are required, and what values are allowed. This does two things. It tells the model precisely what to produce, and it gives you something to validate against afterward, which matters more than people expect.

A schema also kills a whole class of quiet bugs. Without one, the model decides on a Tuesday to call the field "phone_number" instead of "phone," or returns the price as the string "nineteen dollars" instead of the number 19. With a strict schema those drifts either cannot happen or get caught immediately instead of three steps downstream.

Use the mode built for this

Most serious providers now offer a real structured-output feature, and you should use it instead of hoping. There are two flavors worth knowing.

• Structured outputs. You pass a schema and the model is constrained so its output conforms. OpenAI's version, launched in August 2024 with a strict flag, reported 100 percent adherence on their internal schema evals, up from under 40 percent for an older model that was merely asked. That gap is the whole argument.
• Function or tool calling. You describe a function with typed arguments, and the model returns a call to it with the arguments filled in. Same underlying idea, framed as calling code, and the right fit when the JSON is meant to trigger an action.

When the constrained mode is available, the model is not writing free text and then hoping it parses. The valid tokens are restricted as it generates, so malformed JSON becomes close to impossible. That is a completely different reliability story from prompt-and-pray, and it is why the failure rate on the good implementations sits at a fraction of a percent rather than a few percent.

Validate anyway, and retry on purpose

Constrained output handles the shape. It does not handle refusals, truncation from hitting a token limit, or values that are the right type but wrong in context, like a date of February 30. So the last piece is a loop you build yourself.

Parse the response. Validate it against the same schema you sent. If it passes, you are done. If it fails, do not just retry blind: feed the error back to the model, say what broke, and ask it to fix that specific thing. Cap the retries at two or three so a stubborn input cannot spin forever, and log the failures, because a cluster of them usually means your schema is asking for something ambiguous rather than the model misbehaving.

One detail that saves real pain: watch for the refusal path. The good structured-output APIs return a separate refusal field when the model declines, precisely so you do not try to parse an apology as data. Check it before you reach for the parser.

The version of this that holds up in production is boring, and boring is the compliment. A schema the model is constrained to follow, a validation step you never skip, and a short retry loop that tells the model what went wrong. Do that and JSON stops being the flaky part of your pipeline. Skip it, and you will meet the markdown code fence at the worst possible hour.