LLMs

Someone in every AI project eventually says it: "We are spending too much on the API. Let us just host our own model." It sounds obvious. You are renting when you could own. But the spreadsheet almost never says what people expect, and the version both sides show you leaves out the line items that decide the whole thing.

The number that looks damning

Per-token API pricing feels expensive when you watch it add up. A busy product can run a real monthly bill, and the temptation is to compare that against a GPU rental and declare victory. A single H100 rents for somewhere between $1.50 and $10 an hour depending on provider and commitment, call it $1,100 to $7,300 a month running steadily. Next to a fat API invoice, renting the GPU can look cheaper.

It usually is not, because the GPU is the part of self-hosting you can see. The rest hides.

The costs that do not show up on the GPU bill

Running your own inference is not renting a graphics card. It is standing up a service. That means someone who knows how to serve models, batch requests, handle load, patch the stack, and get paged at 3am when it falls over. MLOps engineers are not cheap, and their time is the real cost. Reasonable estimates put the fully loaded cost of self-hosting at three to five times the raw GPU price once you count the people and the plumbing.

Then there is utilization, the quiet killer. The API charges you per token, so you pay for exactly what you use. Your rented GPU charges you per hour whether it is busy or idle. Traffic is spiky. If your GPU sits at 15 percent utilization most of the day, you are paying full price for an empty machine, and your effective cost per token climbs above what the API charged.

Where the break-even actually sits

Concrete numbers help. Against premium APIs, self-hosting a comparable open model tends to break even somewhere around 5 to 10 million tokens a month, and only if you already have the ops capacity. Against the cheap tier, the mini and small models that cost almost nothing per token, the line moves way out: you might need hundreds of millions of tokens a month before owning beats renting.

One worked example from a 2026 analysis: 50 million tokens a day through a small hosted model came to about $2,250 a month on the API, versus roughly $5,175 self-hosted on four mid-range GPUs. More than double, to run the exact same workload yourself.

So the honest brackets look like this:

• Under about 50M tokens a month: the API wins almost every time. Do not self-host to save money, you will not.
• 50M to 500M against frontier-priced APIs, with real MLOps staff on hand: now it is a genuine decision.
• Sustained heavy volume against comparable open models: self-hosting can cut cost several fold, and at that point you probably already know who you are.

There are reasons to self-host that have nothing to do with cost. Data that legally cannot leave your walls. Latency you cannot get from a shared endpoint. A model you have fine-tuned and want to own outright. Those are real, and they can justify the bill on their own. Just do not dress them up as savings, because for most teams below serious scale, self-hosting is the more expensive choice wearing a thrifty costume. If you are moving to it, move for control or privacy, name that out loud, and go in knowing the GPU line was the cheap part.

For years the deal was simple: Microsoft paid, OpenAI built, and Copilot ran on someone else models. At Build 2026, Microsoft quietly changed the arrangement by shipping seven models of its own. The most important AI news of the month was not a benchmark. It was a message.

Ask a language model a question it has no business answering and watch what happens. It answers. Fluently, in complete sentences, with the same even tone it uses for the things it actually knows. The confidence does not move. That gap, between how sure the model sounds and how sure it should be, is the single most dangerous thing about the current generation of these tools.

People call this hallucination and treat it like a bug that a bigger model will fix. It is not a bug. It is baked into how the models are trained.

We taught them to guess

Think about how a model is graded. On most benchmarks a correct answer scores a point, a wrong answer scores zero, and "I do not know" also scores zero. Line those incentives up and the math is brutal: if abstaining pays the same as being wrong, you should always guess. A model that guesses on every uncertain question will beat an honest one that admits its gaps.

So that is the model we built. OpenAI researchers have made the point directly: under binary grading, systems are rewarded for guessing and penalized for saying they are unsure. We optimized for good test-takers, and good test-takers do not leave answers blank. The confident wrong answer is not the model failing. It is the model doing exactly what we rewarded.

Confidence is not knowledge

Here is the part that trips up smart people. We are wired to read fluency as competence. Someone who speaks in clear, structured, well-organized sentences usually knows their subject, because for humans that fluency was expensive to fake. Models make it free. A model produces the same polished prose whether it is reciting a fact or inventing one, and our instinct to trust the smooth talker fires anyway.

That is why "always check the output" is weaker advice than it sounds. Checking is exactly the effort the fluent answer discourages. The better the prose, the less you feel the urge to verify, and the more the one wrong sentence in twenty slips through.

Designing for a witness who never doubts

I do not think the answer is to distrust these tools. I use them every day and they are genuinely good. The answer is to stop treating a model like a source and start treating it like a very fast, very well-read assistant who is constitutionally incapable of saying "I am not sure." Once you accept that, the design choices get clearer.

Put the model where being wrong is cheap and visible: drafting, brainstorming, summarizing text you can see, writing code you are about to run and test. Keep it away from places where a confident fabrication is expensive and hard to catch, like a medical dose, a legal citation, a number that flows straight into a decision with no human in the loop. Retrieval helps, because grounding an answer in a document you can inspect turns "trust me" into "here is where it came from." And when you can, prefer systems that surface uncertainty at all, even a rough confidence score, over ones that render everything in the same calm voice.

There is real research now on training models to abstain, to earn credit for admitting a gap instead of being punished for it. I hope it works. Until it ships, the burden sits with us, and the honest move is to build workflows that assume the machine will never tell you when it is out of its depth. It will keep talking. The question worth sitting with is why we find that so much more comforting than a system that occasionally, usefully, went quiet.

The open-versus-closed argument gets treated like a team you join, complete with jerseys. It is not. It is a trade-off you make one job at a time, and the same person can sensibly land on opposite answers for two projects in the same week. What follows is a decision guide, not a leaderboard, because the leaderboard changes monthly and the trade-off underneath it does not.

What each side actually gives you

Open-weights models, the Llamas and DeepSeeks and Qwens of the world, hand you the actual model. You can download it, run it on your own hardware or a rented box, look at what it does, fine-tune it on your data, and keep the whole thing behind your own firewall. Nobody meters your calls. Nobody can change the model out from under you or deprecate it next quarter. If your data cannot legally or comfortably leave your walls, this is often the only real option.

Closed models, reached through an API, hand you a result. You send text, you get text back, and someone else owns the running of it. In exchange for giving up control you get the current frontier of quality, no infrastructure to babysit, and a model that quietly improves without you lifting a finger. For a lot of teams that is the entire pitch and it is a good one: you want the answer, not a second job running GPUs.

The costs hide in different places

People compare these on the wrong axis. They look at the per-token price of a closed API, see a number bigger than zero, and conclude that self-hosting an open model is cheaper. Sometimes. The closed price includes the hardware, the scaling, the uptime, and the salaries of people who keep it running. Open weights are free to download and very much not free to operate. You are now buying or renting GPUs, and someone on your team owns keeping the thing up at 3 a.m.

The honest version is about volume and steadiness. Spiky, low, or unpredictable traffic almost always favors the closed API, because you pay only for what you use and nothing when you are idle. Heavy, steady, round-the-clock traffic is where owning the hardware can win, because a machine you have already paid for does not care how many calls you push through it. The crossover is real, but it sits much further out than the sticker-price comparison suggests, and it moves every time GPU rental prices or token prices shift.

How to actually choose

Skip the identity and answer a few blunt questions about the specific job.

• Where is the data allowed to go? If it legally cannot leave your infrastructure, that decides it before any quality debate starts. Open weights, self-hosted, done.
• Do you need the absolute top of the quality range? For the hardest reasoning, the newest capabilities, the widest language coverage, the closed frontier models still tend to lead, and the gap is often worth paying for. For a well-scoped task, a mid-size open model may clear the bar with room to spare and cost far less to run.
• Does the model changing under you break things? If you have tuned prompts against exact behavior and cannot afford a silent update, a weights file you pin and control has an edge a hosted endpoint cannot match.
• Do you have people to run it? Serving a model in production is real, ongoing engineering. If that team does not exist, the API is not a compromise, it is the sane choice.

The gap keeps closing, the choice does not

The genuinely good news is that open weights have gotten shockingly close to the closed frontier on many everyday tasks. For summarizing, extraction, classification, ordinary chat, drafting, the practical difference is often small enough not to matter, and it keeps shrinking. That is a real shift and worth being cheerful about.

It does not, however, dissolve the trade-off. Control and privacy and the ops burden that comes with them sit on one side. Convenience and frontier quality and someone else's pager sit on the other. That tension is structural. It will still be here when today's model names are forgotten. The people who get the most out of AI are not the ones who picked a side and defended it. They are the ones who ask, for this specific job, which set of headaches they would rather have, and then pick accordingly.

If you are new to this and trying to pick your first AI model, the internet will hand you forty benchmark charts and a headache. Ignore them. The right first model has almost nothing to do with leaderboard trivia and almost everything to do with what you are actually going to do. Here is the short, honest version.

RAG, retrieval-augmented generation, is one of those terms that sounds like it needs a PhD and actually describes something you could explain to a ten-year-old. Here is the plain version, why it matters, and the case, more common than the vendors admit, where you do not need it at all.

Turn on live captions during a quiet meeting with one clear speaker and you will think the problem is solved. Turn them on during a conference call with three people, a bad mic, and someone eating, and you will remember it is not. Speech-to-text has made a genuine leap in the last two years, and it has also hit a wall that no amount of marketing copy can talk past. Here is where the line actually sits, based on what these systems do outside the demo.

What works, and it is a lot

Clean audio in a common language is close to done. Whisper Large v3, the model most of the field is measured against, lands around a 2.7 percent word error rate on the clean LibriSpeech benchmark. In human terms that is roughly one wrong word in forty, which is better than most people type. If you are transcribing a well-recorded podcast, a single presenter with a decent microphone, or dictation in a quiet room, modern models will not be your problem.

Multilingual coverage is the other real win. Whisper handles ninety-nine languages, and while quality varies, the major ones are genuinely usable. A few years ago, transcription outside English was a research project. Now it is a checkbox. Batch processing, where the system has the whole recording and can take its time, is also strong. Deepgram's Nova-3 reports around 5.26 percent word error on batch audio, and for the common job of turning a recorded call into searchable text overnight, that is more than good enough.

Where it still falls apart

Now the wall. The headline numbers come from clean audio, and the real world is filthy.

One contact-center study told the whole story with three numbers from a single system: 92 percent accuracy on clean headset audio, dropping to 78 percent in a conference room, and collapsing to 65 percent on mobile calls with background noise. Same model, same language, same day. The only thing that changed was the room, and the room is exactly what you do not control. Background noise, crosstalk, and cheap microphones are not edge cases. For most real deployments they are the median case.

Accents remain a stubborn source of error, and it is a fairness problem as much as a technical one. Models trained mostly on standard American and British English degrade on strong regional accents and on second-language speakers, which means the people most likely to get a garbled transcript are often the ones already underserved by technology. This is getting better. It is not fixed.

Real-time is the other hard frontier. A model that scores beautifully in batch mode has to give up accuracy to run live, because it can no longer wait for the end of a sentence to decide what the beginning meant. Deepgram's own numbers show the gap, roughly 6.84 percent word error streaming against 5.26 percent batch. That spread is the price of being live, and for voice agents it is the difference between smooth and infuriating.

And then there is punctuation and speaker labeling, the unglamorous last mile. Getting the words right is one thing. Knowing where the sentence ends, who said it, and where the question mark goes is another, and it is where a technically accurate transcript still reads like a wall of mush.

What to actually pick

Match the tool to the job rather than the leaderboard.

• Offline transcription of recorded files, any budget: Whisper Large v3 is the sensible default, and if speed matters more than the last fraction of accuracy, Distil-Whisper runs about six times faster and stays within a point of it.
• Real-time voice agents and live captions: a streaming-first system like Deepgram Nova-3 will serve you better than forcing a batch model to go live.
• Noisy or multi-speaker audio: budget for cleanup and diarization as separate steps, and test on your actual recordings, not the vendor's samples.

The mistake I see most is trusting the benchmark number and skipping the test on your own audio. That gap between 92 and 65 percent is not hiding in a spec sheet. It is hiding in your particular room, your particular microphones, your particular speakers. Record ten real samples from the environment you actually care about, run them through two or three systems, and read the transcripts yourself. The winner is rarely the one with the prettiest published number, and you will know within an afternoon.

Somewhere in every LLM tutorial there is a line that sets temperature to 0.7, and a remarkable number of people copy it, ship it, and never think about it again. Then the output goes weird and they have no idea which knob to turn. Temperature and top-p are the two settings that decide how the model picks each next word, they are genuinely simple once explained, and most people are using them on autopilot. Here is what they actually do.

The model is always guessing, these settings pick how

At every step, the model produces a ranked list of possible next tokens, each with a probability. It does not know the answer, it has a distribution over answers. Temperature and top-p are two different ways of deciding how to draw from that distribution.

Temperature reshapes the whole distribution. Low temperature, near zero, sharpens it so the single most likely token almost always wins, which makes the output predictable and repeatable. Crank it up toward one and beyond and you flatten the distribution, giving unlikely tokens a real shot and making the output more varied and, past a point, more unhinged. At temperature zero the model is effectively deterministic: same prompt, same answer, every time.

Top-p, also called nucleus sampling, works differently. Instead of reshaping probabilities, it draws a cutoff. Top-p of 0.9 means "consider only the smallest set of tokens whose probabilities add up to 90 percent, and ignore the long tail." The clever part is that it adapts to the model's confidence. When the model is sure, one token already holds most of the probability, so top-p samples from just a token or two. When the model is unsure and fifty tokens each hold a little, top-p keeps all fifty in play. It widens and narrows on its own.

Where people go wrong

The first mistake is turning both knobs at once. Temperature and top-p both constrain the same sampling step, and stacking them makes the effect of either impossible to reason about. The standard advice, which I agree with, is to move one and leave the other at its neutral setting. Pick temperature or pick top-p. Do not choreograph both and then wonder why the behavior is unpredictable.

The second mistake is believing high temperature equals creativity. It does not. High temperature buys you randomness, and randomness is not the same thing as good ideas. Past a certain point you are not getting a more imaginative model, you are getting a model that picks worse words more often. There is a real ceiling where interesting tips over into incoherent, and it arrives sooner than people expect. If you want surprising-but-good, a modest bump plus a better prompt beats cranking the dial to the roof.

The third mistake is silent and nasty: tuning these values on one backend and deploying on another. The OpenAI API, vLLM, and llama.cpp do not necessarily apply samplers in the same order, so the same numbers can produce different text on different stacks. Always tune against the exact backend you will actually run.

Sane defaults to start from

For anything where there is a correct answer, set temperature to 0. Code generation, data extraction, classification, factual questions, anything you would run twice and want the same result. Randomness here is pure downside: it introduces inconsistency across identical inputs and buys you nothing. This is the setting most people should be using far more than they do, because most real work has a right answer and they leave it at 0.7 out of habit.

For creative writing, brainstorming, generating varied options, raise it. Somewhere around 0.7 to 1.0 is a reasonable playground, and this is genuinely where a warmer setting earns its place: you want the model to wander a little, and repeating the exact same story every time defeats the point.

The default that ships with most APIs, around 0.7, is a compromise aimed at open-ended chat. It is a fine middle for conversation and a poor fit for either extreme. If you are extracting invoice fields at 0.7, you are inviting the occasional wrong number for literally no benefit.

The whole thing takes five minutes to understand and saves you a lot of confused debugging. When output feels too samey, you know which way to push. When it feels unreliable and you want the same answer twice, you know to pull temperature to the floor. That is the entire skill: not memorizing magic numbers, but knowing what the two knobs move so you can reach for the right one instead of copying 0.7 from a tutorial and hoping.

I write with AI most days and I am not going to pretend I do not. It drafts, it rephrases, it gets me past the blank page. So take what follows as a note from someone who likes the tool, not someone who wants it banned. There is a cost to handing it all your writing, and it is quiet enough that you can rack up a lot of it before you notice.

Writing is thinking, and you can skip the thinking

The uncomfortable truth about writing is that most of the work is not the words. It is the figuring out. You think you understand an idea until you try to put it in a sentence and discover the hole in the middle of it. The struggle to phrase something is the struggle to actually know it. That is not a flaw in writing, it is the entire point.

When you let a model produce the sentence, you get the artifact without the process. The paragraph looks like understanding. It reads like you thought it through. But the mental work writing usually forces, the part that turns a vague sense into a real position, quietly did not happen. You skipped the gym and kept the mirror.

This is not a hunch. A 2025 study out of MIT wired people up while they wrote essays with an LLM, a search engine, or nothing, and the LLM group showed the lowest engagement, what the researchers called cognitive debt. A separate study of 319 knowledge workers found that the more people trusted the AI, the less critical thinking they reported doing. The effect is measurable, and it points the way you would fear.

Everyone starts to sound the same

There is a second cost, harder to measure and easy to feel. These models write in a house style: smooth, balanced, agreeable, faintly corporate. Lean on it and your writing drifts toward that average. The odd phrasing that was actually yours gets sanded off. Multiply that across everyone using the same handful of models and you get a strange flattening, a web where a lot of prose has the same tidy cadence and no fingerprints.

Voice is not decoration. It is the trace of a specific person having a specific thought. When you outsource the sentence, you outsource the fingerprint, and the reader feels the absence even when they cannot name it. Half of why anyone reads a particular writer is to hear how that person, and no one else, would put it.

Where I actually draw the line

I am not arguing for writing everything by hand out of principle. That would be its own kind of pose. The line I try to hold is about what the writing is for.

If the goal is to move information from A to B, a status update, a boilerplate email, a summary nobody will reread, let the model do it and get your afternoon back. The thinking there is not worth protecting. But if the writing is where you work out what you believe, an argument, a design you are still unsure of, anything you will have to defend later, write the first pass yourself. Struggle through the bad draft. That draft is you learning the subject, and the model cannot do that part for you. It can only hide that you skipped it.

The skill you stop using is the skill you lose. Not dramatically, not all at once, just a slow softening you do not clock until the day you sit down to write something that matters and find the muscle is not there. Keep writing the things worth thinking about. Let the machine have the rest.

June and early July gave us a model release almost every day, which is exactly why you should not try to follow all of them. Most were incremental. A few will still matter next summer. Here is the month sorted the only way that is useful: by how long it will stay relevant.

The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.

What a vector database actually does

Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.

That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.

When a plain file is genuinely enough

Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.

An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.

The rough line in my head:

• Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
• Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.

Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.

The middle ground people skip

Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.

Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.

My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.

A token is the thing you are actually paying for, and almost nobody thinks in tokens. They think in words, or messages, or vibes. Then the invoice arrives and the math does not line up. Here is the plain version of what a token is, how your text gets chopped into them, and why the model bills this way instead of the way you would expect.

Not a word, not a letter, something in between

When you send text to a model, the first thing that happens is that a piece of software called a tokenizer breaks your text into pieces. A token is one of those pieces. Sometimes a token is a whole word. Often it is a fragment. The word "sleeping" might become "sleep" plus "ing". Common words like "the" are usually a single token, while a rare or long word gets split into two or three.

The rough rule people quote is that one token is about four characters of English, or roughly three quarters of a word. OpenAI's own guidance says about 750 words works out to around 1000 tokens. That heuristic is fine for a quick estimate and wrong the moment your text stops looking like a plain paragraph. Code, punctuation, JSON, and tables all push the count up, because symbols and indentation do not compress into neat word-sized chunks. So does anything not in English. The tokenizers were trained mostly on English, so "strawberry" costs one or two tokens while the same word in Chinese or Arabic can cost several times more. Same meaning, bigger bill.

Why chop words up at all? Because the model has a fixed vocabulary, usually somewhere between 50,000 and 100,000 tokens. If every distinct word needed its own entry, the vocabulary would explode and the model would choke on any word it had never seen. Subword tokens are the compromise. The model learns a set of common fragments and reassembles anything, including words that did not exist when it was trained, out of pieces it already knows.

Why the meter runs on tokens

Providers charge per token because tokens are what the model literally processes, one step at a time. Every token in your prompt has to be read before the model writes anything, and every token it writes is another unit of compute. There is no natural billing unit called a "word" inside the machine. There are only tokens going in and tokens coming out.

That is also why input and output are usually priced differently, with output costing more. Generating a token is more expensive than reading one, because the model does a full forward pass for each new token it produces. When you see two prices on a pricing page, that is what the gap is about.

Where the bill bites

The practical consequences follow directly, and they are easy to miss until they cost you money.

• Long context is not free context. If you paste a 40-page document into every request, you pay for all of it on every request, whether or not the model needed most of it to answer. A chat that remembers the whole conversation is re-reading and re-charging that history each turn.
• Verbose system prompts are a recurring tax. That 800-token instruction block you wrote once gets billed on every single call.
• "Make it concise" saves output tokens, which are the expensive ones. Asking for a one-line answer instead of an essay is a real cost lever, not just a style choice.

None of this means you should obsess over trimming tokens like a coupon-clipper. Most single requests cost a fraction of a cent. It matters at volume. A prompt that is 30 percent heavier than it needs to be does not hurt when you run it twice. Run it two million times and it is a line item someone will ask you about.

Count before you commit

The one habit worth building: stop estimating in words when real money is involved. Run your actual prompts, the messy ones with the code blocks and the pasted logs and the non-English names, through a token counter before you ship. Most providers give you one, and the number is almost always higher than your gut said, because your gut counts words and the model counts something stranger.

The word is a unit humans invented for humans. The token is a unit the machine invented for itself. Your bill is written in the machine's units, and once you can see the text the way the tokenizer sees it, the invoice stops being a surprise and starts being something you can actually plan around.

The first time I used an AI assistant to fix a gnarly bug, I felt two things at once: relief that it was solved in four minutes, and a small, nagging sense that I had been robbed. Not by the tool. By myself. Two weeks later the same class of bug showed up in a different file, and I stared at it like a stranger. The AI had solved the problem. I had learned nothing.

This is the part of the AI conversation that gets flattened into slogans. One camp says friction is waste and anything that removes it is pure gain. The other says we are raising a generation that cannot think. Both are too neat. The honest position is that some struggle is where understanding gets built, and some struggle is just pain with no payoff, and the whole skill now is telling them apart.

Not all friction is the same

There is a concept from learning research called desirable difficulty. The idea is that certain kinds of effort, retrieving something from memory instead of rereading it, working a problem before seeing the solution, make the knowledge stick harder precisely because they were hard. The difficulty is not a tax on learning. It is the mechanism of learning. When you struggle to recall how a data structure works and then get it right, you are not being inefficient. You are laying down the thing that lets you recall it next time without help.

Now compare that to the friction of fighting a badly documented API for an afternoon, or hand-formatting a table, or writing the same boilerplate for the tenth time. That friction teaches you nothing you did not already know. It is the pain of a task, not the effort of learning. Handing it to a machine is an unambiguous win, and anyone romanticizing it has confused suffering with growth.

The trouble is that these two kinds of friction feel identical in the moment. Both are annoying. Both make you want the answer now. And the AI is equally happy to dissolve either one. So the tool that saves you from pointless boilerplate is the same tool, with the same one-keystroke ease, that saves you from the productive struggle that would have made you better. It does not know the difference, and if you are not paying attention, neither will you.

The muscle you do not know you are losing

Here is what worries me, and I say this as someone who loves these tools and uses them all day. The loss is invisible while it happens. You do not get a warning that says your debugging intuition is atrophying. You just notice, months later, that you reach for the assistant a beat earlier each time, that the threshold of difficulty at which you give up and ask has quietly dropped. The struggle you skip does not announce itself. It shows up as an absence, later, when the tool is not there or is confidently wrong and you have lost the ability to check it.

There is a real professional risk in this. The senior engineer who is valuable is valuable because of ten thousand small struggles that built judgment. If the next cohort skips those struggles, they arrive at senior titles with junior instincts, and they will not know it, because the code shipped and the tests passed the whole way up.

A rule I actually use

I have started asking one question before I hand something off: will I need to understand this again? If the answer is yes, if this is a domain I work in, a system I own, a skill I want, I do the struggle first and let the AI check my work or fill the gaps. If the answer is no, if this is a one-time chore in a corner of the world I will never revisit, I hand it over without a shred of guilt and move on with my afternoon.

This is not a call to suffer for its own sake. Suffering for its own sake is a religion, not an engineering practice. It is a call to be deliberate about which difficulties you outsource, because the machine will take all of them if you let it, and a few of them were the point.

I still fixed that bug in four minutes. I would do it again. But now, on the problems I actually care about getting good at, I make myself sit in the not-knowing a little longer before I ask. It is slower. It is also the only part of the work that is still making me better, and I am not ready to give that up for a faster Tuesday.

Every few months the internet holds another referendum on artificial general intelligence. Are we two years away, ten years away, or is the whole idea a mirage? People with real credentials line up on both sides and yell past each other. It is a fun argument. It is also, for almost everyone actually using these tools, a waste of a good afternoon.

Nobody agrees what the word means

Start with the obvious problem: AGI has no agreed definition. Ask ten researchers and you get ten thresholds. Human-level performance on most economically valuable work. The ability to learn any task a person can. Something that can do original science. A system that beats experts across every benchmark we can invent. These are not variations on one idea. They are different ideas wearing the same three letters.

An argument where the central term means something different to each participant is not a disagreement about the world. It is a disagreement about vocabulary, and those never resolve, because there is no fact that could settle them. You can watch a debate run for an hour and realize at the end that one person was talking about job displacement and the other was talking about consciousness, and they both thought they were talking about the same thing.

When a target keeps moving, hitting it stops meaning anything. Systems now do things that would have counted as AGI to a researcher in 2015: hold a coherent conversation, pass the bar exam, write working code from a description. We did not throw a party. We shrugged and moved the line. That reflex tells you the concept is doing rhetorical work, not descriptive work.

The questions that actually pay rent

Here is what the AGI argument crowds out. Is this tool useful for the thing I need done? Is it reliable enough that I do not have to check every output by hand? Can I afford to run it at the scale I need? Those three questions decide whether AI helps you this quarter, and none of them require anyone to agree on when or whether the machines wake up.

They are also answerable. You can measure whether a coding assistant saves your team time. You can count how often a support bot hands back a wrong answer. You can read a pricing page and do arithmetic. These are boring, concrete, checkable things, which is exactly why they get less airtime than the cosmic stuff. Nobody writes a viral thread about "we ran it for three weeks and it was fine, mostly."

The gap between capability and reliability is where all the real action sits, and AGI talk skips right over it. A model that can ace a graduate exam can also state a made-up statute with total confidence. Both facts are true at once. The exciting frontier and the boring failure live in the same box, and if you are deploying anything, the boring failure is the part that ends up in your incident report.

Skeptical is not the same as cynical

I want to be careful here, because dismissing AGI talk can slide into dismissing AI, and that is not the point. These tools are genuinely useful. I use them every day and would be annoyed to lose them. The point is narrower: whether they eventually become general in some philosophical sense has almost no bearing on whether they are worth your money and attention right now.

Treat the far-future question as a hobby, not a strategy. It is fine to find it fascinating. Read the essays, argue at dinner, place your bets. Just do not let it stand in for the work of evaluating what is actually in front of you, because the two have almost nothing to do with each other. A team that spends its meetings debating timelines is a team not testing the tool on its own data.

The honest position is unsatisfying and I will say it anyway: I do not know when or whether AGI arrives, and neither does anyone selling you a strong opinion about it. What I do know is whether the model in front of me got the invoice numbers right, and that is the question that pays my bills. When someone wants to argue about superintelligence, I am happy to listen. Then I go back to checking whether the thing works, because that is the only argument with an answer I can use.

A language model tells you, with total confidence, that a book exists that does not, cites a court case nobody filed, and invents a function your library never had. People call this hallucination and treat it like a glitch, a bug the next update will squash. It is not a glitch. It is the natural output of how these systems work, and once you see the mechanism, the confident wrongness stops being mysterious and starts being predictable.

The machine has no idea what is true

Strip a language model down and it does one thing: given the text so far, it predicts the next token, then the next, then the next. It was trained to make that prediction as plausible as possible against a mountain of human text. Nowhere in that process is there a step where the model checks a fact against the world. It has no database of truth to consult. It has a very good sense of what words tend to follow other words.

That is the whole engine, and it explains the behavior. When you ask for a real citation, the model produces a string of tokens that looks exactly like a citation, because it has seen thousands of them and knows the shape cold. Author, year, title, a plausible page number. Whether that particular paper exists is a question the model was never built to answer. It is generating something citation-flavored, and most of the time reality happens to line up. When it does not, you get a hallucination that reads just as smoothly as a true one, because the model is equally fluent either way. Fluency is what it optimizes. Accuracy is a thing that sometimes rides along.

This is also why the tone never wavers. A person who is unsure hedges, slows down, says "I think." The model has no separate confidence signal wired to its output style. It generates the most likely continuation, and the most likely continuation of a question is a direct, assured answer, whether or not the content is right. Confident by construction, not by conviction.

We trained it to guess

Here is the part that stings, from OpenAI's own 2025 research on the topic. Models hallucinate partly because the way we grade them rewards it. Most benchmarks score a question right or wrong. Say "I do not know" and you score zero. Take a confident guess and you have some chance of being right, which scores better on average. Over millions of training and evaluation signals, that math teaches the model the same lesson it would teach a student facing a test with no penalty for wrong answers: when unsure, bluff. The calibration is off on purpose, because the incentive was off. An honest "I am not certain" gets punished by the scoreboard, so the model learns not to say it.

Kinds of hallucination, and what dents them

They are not all the same. There is the invented fact, a person or event that does not exist. There is the wrong detail inside a real answer, a correct summary with one bogus number. There is the fabricated source, the citation or link that leads nowhere. And there is the instruction failure, where the model contradicts something you told it three sentences ago because keeping the whole context straight is itself imperfect. Different causes, different fixes.

You cannot eliminate hallucination, so stop trying to and start reducing it. Grounding is the biggest lever: give the model the actual source text and ask it to answer from that, which is the core reason retrieval-augmented generation exists. It is far harder to invent a citation when the real documents are sitting in the prompt. Ask for sources you can check, and then check them, because an unverifiable claim from a model is a claim, not a fact. For anything that matters, add a verification pass, a second model or a rule or a human confirming the output against something real. And give the model permission to bail. If your prompt makes "I do not know" an acceptable answer, you undo a little of the guessing it was trained into.

The honest summary is that a language model is a fluent guesser with no built-in sense of truth, graded for years by tests that paid it to guess. Treat every confident answer as a strong draft that has not been fact-checked, because from the model's side, that is exactly what it is.

You run a prompt, get a good answer, run the exact same prompt again, and get something different. Sometimes it is a rephrasing. Sometimes it is a different conclusion. If you come from normal software, where the same input gives the same output forever, this feels like the thing is broken. It is not broken. It is doing exactly what it was built to do, and understanding why will change how you use these tools.

The model does not pick a word, it picks from a distribution

At every step, a language model does not decide on the next word. It produces a probability for every possible next token. The word "blue" might get 60 percent, "green" 20 percent, "warm" 5 percent, and a long tail of everything else splitting the rest. Then a sampler reaches into that distribution and draws one.

Temperature is the knob that reshapes those odds before the draw. Turn it up and the flat, unlikely options get more weight, so the output wanders and surprises you. Turn it down and probability piles onto the front-runners, so the model plays it safe. This is a feature. You want a brainstorming assistant to roam and a data-extraction call to stay boring. The variation you are seeing at normal temperature is the model exploring, on purpose.

So the obvious move is to set temperature to 0. Now it always takes the single most likely token, greedy decoding, no dice roll. Same prompt, same answer, forever. Right?

Temperature 0 is not the guarantee you think it is

In practice, temperature 0 gets you close but not all the way. People assumed for years that the leftover wobble was just floating-point noise, some vague hand-wave about GPUs being messy. That answer was never satisfying, and in 2025 researchers at Thinking Machines laid out the actual culprit, and it is more interesting than randomness.

The real cause is batch invariance, or rather the lack of it. When you send a request to a hosted model, you are not alone. The serving system bundles your request together with whatever other requests arrived at the same moment and runs them through the GPU as one batch. That batch is different every time, because it depends on who else is calling the API at that instant, something you have zero control over.

Here is why that matters. Floating-point addition is not associative. Adding a set of numbers in one order can give a very slightly different result than adding them in another order, down in the last bits. When the batch size changes, the GPU kernels split and sum their work in a different order, so the model's internal numbers drift by a hair. Almost always that hair is invisible. But every so often two candidate tokens are nearly tied, the drift nudges one above the other, and the model picks a different word. From there the two answers diverge, because each new token conditions on the last one.

Sit with that for a second. The nondeterminism you see is not the model being random. It is your request sharing a GPU with strangers. The cause is numerical, not magical, and it is fixable. Batch-invariant kernels that pin the reduction order now ship in serving engines like vLLM and SGLang, and with them you can get genuinely reproducible output. Most hosted APIs do not turn that on by default, because it costs a bit of throughput, so the wobble stays.

How to actually think about it

Stop treating a single model output as a fact and start treating it as a sample. One run tells you what the model tends to say. It does not tell you what it always says. That reframe fixes a lot of bad habits.

If you need the same answer every time, do not lean on the model's internals to give it to you. Cache the result keyed on the input, and serve the cached copy. That is deterministic by construction and it is cheaper. If you are extracting structured data, run the call, but validate the shape and constrain the output with a schema so a reworded answer cannot break your parser. If you are evaluating a prompt, never judge it on one run. Sample it ten times and look at the spread, because the spread is the real behavior and a single lucky run is a trap that will embarrass you in production.

And when a demo shows you one perfect output, remember what you are looking at. You are seeing one draw from a distribution, chosen and probably cherry-picked. The honest question is not whether it can produce that answer. It is how often, and what the other draws look like. Ask for a few more and you will learn more in thirty seconds than the polished screenshot will ever tell you.