AI Guides & Tutorials
The demo worked because you were the only person hitting it. Then ten users showed up, a provider had a slow afternoon, and your app started returning blank screens and half-written answers. None of that is an AI problem. It is a plumbing problem, and the plumbing for AI apps is a little different because the calls are slow, expensive, and occasionally rejected on purpose.
Here is the honest version of what it takes to make a model-backed feature that does not fall over the first time the network has a bad day.
Rate limits are a promise, not an insult
Every provider caps how many requests and tokens you can send per minute. When you cross the line you get an HTTP 429. People treat that response like a failure, but it is closer to a traffic light. The provider is telling you exactly how to behave, and most of the time the answer is written in the headers. Anthropic and OpenAI both send a Retry-After value on many 429s. If it is there, wait that long. Do not invent your own number when the server already told you the truth.
The mistake I see most often is retrying immediately, in a tight loop, across every worker at once. That does not recover from a rate limit. It deepens it. You have now turned one rejected request into fifty, all landing at the same instant, guaranteeing the next fifty get rejected too. This is the thundering herd, and it is self-inflicted.
Retries: back off, and add jitter
The fix is exponential backoff. Wait one second, then two, then four, then eight, doubling each time up to a ceiling. That spaces your attempts out so the system has room to recover. But backoff alone is not enough, because if a hundred clients all failed at the same moment, they will all wait exactly one second and then all retry at exactly the same moment. You have synchronized your herd instead of scattering it.
So you add jitter: a random amount of slop on each wait. Instead of sleeping exactly two seconds, sleep somewhere between zero and two. This one trick, randomizing the delay, does more for stability than almost anything else you can do, and it is two lines of code.
A few rules that keep retries sane:
- Retry on 429 and 5xx and network timeouts. Do not retry a 400 or a 401, because a malformed request or a bad key will fail identically every time.
- Cap the total. Three to five attempts, then give up and surface an error. Infinite retries just hide the outage from you while your users feel every second of it.
Timeouts, and the streaming trap
Model calls are slow and the tail is long. A response that usually takes three seconds will sometimes take forty. If you have no timeout, a single stuck request can hold a connection open until something upstream kills it, and you find out when your whole pool is exhausted. Set an explicit timeout. Pick a number based on your real latency distribution, not on the median, because the median is not what hurts you.
Streaming changes the shape of this. When you stream tokens, the connection can be alive and healthy while the model has quietly stalled and stopped sending. A total request timeout will not catch that cleanly. What you want is an idle timeout: if no token has arrived in some number of seconds, treat the stream as dead and fall back. Watch the gaps between chunks, not just the clock on the whole call.
Idempotency and degrading on purpose
Here is the part people skip until it bites them. You send a request, the model does the work, and then the response gets lost on the way back to you. Your retry logic kicks in and sends it again. Now you have paid for the same completion twice, and if that call also wrote a row or charged a card, you have a duplicate in your data.
Send an idempotency key with anything that has a side effect. It is a unique string per logical operation. If the provider or your own service sees the same key twice, it returns the first result instead of doing the work again. Providers support this for exactly this reason. Use it, and a lost response becomes a non-event instead of a double charge.
Sometimes the model is just down, or slow enough that waiting is worse than not answering. Decide ahead of time what happens then. Maybe you drop to a smaller, cheaper model. Maybe you return a cached answer that is a little stale. Maybe you show the raw search results without the AI summary on top. The worst option is a spinner that never resolves, because that teaches people your product is broken.
None of this is glamorous and none of it shows up in a launch video. But it is the difference between a feature that survives contact with real traffic and one that only ever worked on your laptop. The models get the headlines. The retry loop with jitter is what keeps the thing standing.
"Memory" is one of the most oversold words in AI right now. Products promise an assistant that remembers you, learns your preferences, picks up where you left off. Then you use it for a week and notice it forgot the thing you told it on Tuesday. The gap between the pitch and the reality comes from a fact people gloss over: a language model has no memory at all.
The model is a function. Text goes in, text comes out, and the moment the response finishes it retains nothing. Everything we call memory is scaffolding we build around that stateless core, deciding what to put back into the input next time. There are only a few honest ways to do it, and each one breaks in its own way.
Stuffing the context window
The crudest approach: keep the whole conversation and paste it back in every turn. For a short chat this is perfect. The model "remembers" because you literally handed it the transcript.
It falls apart on two edges. Context windows are finite, so a long enough conversation eventually will not fit. And even when it fits, more context is not free. You pay for every token on every call, and models get measurably worse at finding the one relevant line when it is buried in tens of thousands of tokens of history. Stuffing is a real technique with a hard ceiling, not a memory system.
Summaries and their slow leak
So you compress. Every so often, ask the model to summarize the conversation so far and carry the summary forward instead of the raw text. This buys a lot of runway, and most chat products do some version of it.
The catch is that summarizing is lossy on purpose, and you do not get to choose what it loses. The detail that seemed irrelevant when the summary was written is exactly the one that matters three turns later. Summaries of summaries drift further each round, like a photocopy of a photocopy. Good for the gist, unreliable for the specific fact.
Retrieval, which is what "memory" usually means now
The approach that actually scales is to stop trying to hold everything in the prompt. Write facts and past exchanges to a store, and when a new message comes in, fetch only the handful of pieces relevant to it and drop those into the context. This is RAG pointed at your own history instead of a document library.
It is the best of the three and still not what people imagine. Retrieval memory only recalls what it thought to save and what the query happens to match. Phrase your question differently than the stored fact was written and the right memory may never surface. It cannot form the kind of connection a person makes across two things noticed months apart, because nothing is reasoning over the whole store, it is just fetching nearest matches. What you get is a good filing clerk, not a mind.
Here is the thing I wish more products said plainly. None of these give a model memory in the human sense, the kind that reshapes understanding over time. They are all variations on one move: choose what text to put in front of a stateless function on the next call. That is a real and useful engineering problem, and you can build genuinely helpful things by solving it well.
So when you design memory, do not chase the fantasy of an assistant that just knows you. Decide, deliberately, what is worth remembering, how it gets written down, and how it gets found again. Keep the recent turns raw, summarize the middle distance, retrieve the long tail, and accept that each layer forgets in a different way. The teams who ship memory that feels good are not the ones with a magic store. They are the ones who were honest about what a model is and engineered around it.
The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.
What a vector database actually does
Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.
That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.
When a plain file is genuinely enough
Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.
An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.
The rough line in my head:
- Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
- Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.
Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.
The middle ground people skip
Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.
Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.
My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.
You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.
Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.
Size is a tradeoff, not a setting
The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.
Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.
Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.
Semantic splitting is oversold
The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.
A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.
What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.
The mistakes that cost you
A few failure modes show up again and again:
- Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
- Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
- Chasing the perfect chunk size before you have a single real question to test against.
That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.
Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.
A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.
The machine has no idea what is true
Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.
That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.
This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.
We trained it to guess
Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.
Kinds of hallucination, and what dents them
They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.
You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.
The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.