LLMs

RAG, retrieval-augmented generation, is one of those terms that sounds like it needs a PhD and actually describes something you could explain to a ten-year-old. Here is the plain version, why it matters, and the case, more common than the vendors admit, where you do not need it at all.

Somewhere in every LLM tutorial there is a line that sets temperature to 0.7, and a remarkable number of people copy it, ship it, and never think about it again. Then the output goes weird and they have no idea which knob to turn. Temperature and top-p are the two settings that decide how the model picks each next word, they are genuinely simple once explained, and most people are using them on autopilot. Here is what they actually do.

The model is always guessing, these settings pick how

At every step, the model produces a ranked list of possible next tokens, each with a probability. It does not know the answer, it has a distribution over answers. Temperature and top-p are two different ways of deciding how to draw from that distribution.

Temperature reshapes the whole distribution. Low temperature, near zero, sharpens it so the single most likely token almost always wins, which makes the output predictable and repeatable. Crank it up toward one and beyond and you flatten the distribution, giving unlikely tokens a real shot and making the output more varied and, past a point, more unhinged. At temperature zero the model is effectively deterministic: same prompt, same answer, every time.

Top-p, also called nucleus sampling, works differently. Instead of reshaping probabilities, it draws a cutoff. Top-p of 0.9 means "consider only the smallest set of tokens whose probabilities add up to 90 percent, and ignore the long tail." The clever part is that it adapts to the model's confidence. When the model is sure, one token already holds most of the probability, so top-p samples from just a token or two. When the model is unsure and fifty tokens each hold a little, top-p keeps all fifty in play. It widens and narrows on its own.

Where people go wrong

The first mistake is turning both knobs at once. Temperature and top-p both constrain the same sampling step, and stacking them makes the effect of either impossible to reason about. The standard advice, which I agree with, is to move one and leave the other at its neutral setting. Pick temperature or pick top-p. Do not choreograph both and then wonder why the behavior is unpredictable.

The second mistake is believing high temperature equals creativity. It does not. High temperature buys you randomness, and randomness is not the same thing as good ideas. Past a certain point you are not getting a more imaginative model, you are getting a model that picks worse words more often. There is a real ceiling where interesting tips over into incoherent, and it arrives sooner than people expect. If you want surprising-but-good, a modest bump plus a better prompt beats cranking the dial to the roof.

The third mistake is silent and nasty: tuning these values on one backend and deploying on another. The OpenAI API, vLLM, and llama.cpp do not necessarily apply samplers in the same order, so the same numbers can produce different text on different stacks. Always tune against the exact backend you will actually run.

Sane defaults to start from

For anything where there is a correct answer, set temperature to 0. Code generation, data extraction, classification, factual questions, anything you would run twice and want the same result. Randomness here is pure downside: it introduces inconsistency across identical inputs and buys you nothing. This is the setting most people should be using far more than they do, because most real work has a right answer and they leave it at 0.7 out of habit.

For creative writing, brainstorming, generating varied options, raise it. Somewhere around 0.7 to 1.0 is a reasonable playground, and this is genuinely where a warmer setting earns its place: you want the model to wander a little, and repeating the exact same story every time defeats the point.

The default that ships with most APIs, around 0.7, is a compromise aimed at open-ended chat. It is a fine middle for conversation and a poor fit for either extreme. If you are extracting invoice fields at 0.7, you are inviting the occasional wrong number for literally no benefit.

The whole thing takes five minutes to understand and saves you a lot of confused debugging. When output feels too samey, you know which way to push. When it feels unreliable and you want the same answer twice, you know to pull temperature to the floor. That is the entire skill: not memorizing magic numbers, but knowing what the two knobs move so you can reach for the right one instead of copying 0.7 from a tutorial and hoping.

I write with AI most days and I am not going to pretend I do not. It drafts, it rephrases, it gets me past the blank page. So take what follows as a note from someone who likes the tool, not someone who wants it banned. There is a cost to handing it all your writing, and it is quiet enough that you can rack up a lot of it before you notice.

Writing is thinking, and you can skip the thinking

The uncomfortable truth about writing is that most of the work is not the words. It is the figuring out. You think you understand an idea until you try to put it in a sentence and discover the hole in the middle of it. The struggle to phrase something is the struggle to actually know it. That is not a flaw in writing, it is the entire point.

When you let a model produce the sentence, you get the artifact without the process. The paragraph looks like understanding. It reads like you thought it through. But the mental work writing usually forces, the part that turns a vague sense into a real position, quietly did not happen. You skipped the gym and kept the mirror.

This is not a hunch. A 2025 study out of MIT wired people up while they wrote essays with an LLM, a search engine, or nothing, and the LLM group showed the lowest engagement, what the researchers called cognitive debt. A separate study of 319 knowledge workers found that the more people trusted the AI, the less critical thinking they reported doing. The effect is measurable, and it points the way you would fear.

Everyone starts to sound the same

There is a second cost, harder to measure and easy to feel. These models write in a house style: smooth, balanced, agreeable, faintly corporate. Lean on it and your writing drifts toward that average. The odd phrasing that was actually yours gets sanded off. Multiply that across everyone using the same handful of models and you get a strange flattening, a web where a lot of prose has the same tidy cadence and no fingerprints.

Voice is not decoration. It is the trace of a specific person having a specific thought. When you outsource the sentence, you outsource the fingerprint, and the reader feels the absence even when they cannot name it. Half of why anyone reads a particular writer is to hear how that person, and no one else, would put it.

Where I actually draw the line

I am not arguing for writing everything by hand out of principle. That would be its own kind of pose. The line I try to hold is about what the writing is for.

If the goal is to move information from A to B, a status update, a boilerplate email, a summary nobody will reread, let the model do it and get your afternoon back. The thinking there is not worth protecting. But if the writing is where you work out what you believe, an argument, a design you are still unsure of, anything you will have to defend later, write the first pass yourself. Struggle through the bad draft. That draft is you learning the subject, and the model cannot do that part for you. It can only hide that you skipped it.

The skill you stop using is the skill you lose. Not dramatically, not all at once, just a slow softening you do not clock until the day you sit down to write something that matters and find the muscle is not there. Keep writing the things worth thinking about. Let the machine have the rest.

June and early July gave us a model release almost every day, which is exactly why you should not try to follow all of them. Most were incremental. A few will still matter next summer. Here is the month sorted the only way that is useful: by how long it will stay relevant.

The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.

What a vector database actually does

Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.

That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.

When a plain file is genuinely enough

Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.

An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.

The rough line in my head:

• Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
• Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.

Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.

The middle ground people skip

Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.

Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.

My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.

A token is the thing you are actually paying for, and almost nobody thinks in tokens. They think in words, or messages, or vibes. Then the invoice arrives and the math does not line up. Here is the plain version of what a token is, how your text gets chopped into them, and why the model bills this way instead of the way you would expect.

Not a word, not a letter, something in between

When you send text to a model, the first thing that happens is that a piece of software called a tokenizer breaks your text into pieces. A token is one of those pieces. Sometimes a token is a whole word. Often it is a fragment. The word "sleeping" might become "sleep" plus "ing". Common words like "the" are usually a single token, while a rare or long word gets split into two or three.

The rough rule people quote is that one token is about four characters of English, or roughly three quarters of a word. OpenAI's own guidance says about 750 words works out to around 1000 tokens. That heuristic is fine for a quick estimate and wrong the moment your text stops looking like a plain paragraph. Code, punctuation, JSON, and tables all push the count up, because symbols and indentation do not compress into neat word-sized chunks. So does anything not in English. The tokenizers were trained mostly on English, so "strawberry" costs one or two tokens while the same word in Chinese or Arabic can cost several times more. Same meaning, bigger bill.

Why chop words up at all? Because the model has a fixed vocabulary, usually somewhere between 50,000 and 100,000 tokens. If every distinct word needed its own entry, the vocabulary would explode and the model would choke on any word it had never seen. Subword tokens are the compromise. The model learns a set of common fragments and reassembles anything, including words that did not exist when it was trained, out of pieces it already knows.

Why the meter runs on tokens

Providers charge per token because tokens are what the model literally processes, one step at a time. Every token in your prompt has to be read before the model writes anything, and every token it writes is another unit of compute. There is no natural billing unit called a "word" inside the machine. There are only tokens going in and tokens coming out.

That is also why input and output are usually priced differently, with output costing more. Generating a token is more expensive than reading one, because the model does a full forward pass for each new token it produces. When you see two prices on a pricing page, that is what the gap is about.

Where the bill bites

The practical consequences follow directly, and they are easy to miss until they cost you money.

• Long context is not free context. If you paste a 40-page document into every request, you pay for all of it on every request, whether or not the model needed most of it to answer. A chat that remembers the whole conversation is re-reading and re-charging that history each turn.
• Verbose system prompts are a recurring tax. That 800-token instruction block you wrote once gets billed on every single call.
• "Make it concise" saves output tokens, which are the expensive ones. Asking for a one-line answer instead of an essay is a real cost lever, not just a style choice.

None of this means you should obsess over trimming tokens like a coupon-clipper. Most single requests cost a fraction of a cent. It matters at volume. A prompt that is 30 percent heavier than it needs to be does not hurt when you run it twice. Run it two million times and it is a line item someone will ask you about.

Count before you commit

The one habit worth building: stop estimating in words when real money is involved. Run your actual prompts, the messy ones with the code blocks and the pasted logs and the non-English names, through a token counter before you ship. Most providers give you one, and the number is almost always higher than your gut said, because your gut counts words and the model counts something stranger.

The word is a unit humans invented for humans. The token is a unit the machine invented for itself. Your bill is written in the machine's units, and once you can see the text the way the tokenizer sees it, the invoice stops being a surprise and starts being something you can actually plan around.

Every few months the internet holds another referendum on artificial general intelligence. Are we two years away, ten years away, or is the whole idea a mirage? People with real credentials line up on both sides and yell past each other. It is a fun argument. It is also, for almost everyone actually using these tools, a waste of a good afternoon.

Nobody agrees what the word means

Start with the obvious problem: AGI has no agreed definition. Ask ten researchers and you get ten thresholds. Human-level performance on most economically valuable work. The ability to learn any task a person can. Something that can do original science. A system that beats experts across every benchmark we can invent. These are not variations on one idea. They are different ideas wearing the same three letters.

An argument where the central term means something different to each participant is not a disagreement about the world. It is a disagreement about vocabulary, and those never resolve, because there is no fact that could settle them. You can watch a debate run for an hour and realize at the end that one person was talking about job displacement and the other was talking about consciousness, and they both thought they were talking about the same thing.

When a target keeps moving, hitting it stops meaning anything. Systems now do things that would have counted as AGI to a researcher in 2015: hold a coherent conversation, pass the bar exam, write working code from a description. We did not throw a party. We shrugged and moved the line. That reflex tells you the concept is doing rhetorical work, not descriptive work.

The questions that actually pay rent

Here is what the AGI argument crowds out. Is this tool useful for the thing I need done? Is it reliable enough that I do not have to check every output by hand? Can I afford to run it at the scale I need? Those three questions decide whether AI helps you this quarter, and none of them require anyone to agree on when or whether the machines wake up.

They are also answerable. You can measure whether a coding assistant saves your team time. You can count how often a support bot hands back a wrong answer. You can read a pricing page and do arithmetic. These are boring, concrete, checkable things, which is exactly why they get less airtime than the cosmic stuff. Nobody writes a viral thread about "we ran it for three weeks and it was fine, mostly."

The gap between capability and reliability is where all the real action sits, and AGI talk skips right over it. A model that can ace a graduate exam can also state a made-up statute with total confidence. Both facts are true at once. The exciting frontier and the boring failure live in the same box, and if you are deploying anything, the boring failure is the part that ends up in your incident report.

Skeptical is not the same as cynical

I want to be careful here, because dismissing AGI talk can slide into dismissing AI, and that is not the point. These tools are genuinely useful. I use them every day and would be annoyed to lose them. The point is narrower: whether they eventually become general in some philosophical sense has almost no bearing on whether they are worth your money and attention right now.

Treat the far-future question as a hobby, not a strategy. It is fine to find it fascinating. Read the essays, argue at dinner, place your bets. Just do not let it stand in for the work of evaluating what is actually in front of you, because the two have almost nothing to do with each other. A team that spends its meetings debating timelines is a team not testing the tool on its own data.

The honest position is unsatisfying and I will say it anyway: I do not know when or whether AGI arrives, and neither does anyone selling you a strong opinion about it. What I do know is whether the model in front of me got the invoice numbers right, and that is the question that pays my bills. When someone wants to argue about superintelligence, I am happy to listen. Then I go back to checking whether the thing works, because that is the only argument with an answer I can use.

A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.

The machine has no idea what is true

Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.

That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.

This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.

We trained it to guess

Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.

Kinds of hallucination, and what dents them

They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.

You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.

The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.