LLMs

AI coding tools are genuinely great and genuinely dangerous, often in the same suggestion. They will write in thirty seconds something that would have taken you twenty minutes, and it will contain a subtle bug you would never have written yourself. Here is the workflow I actually use to get the speed without shipping the mistakes.

For a long time the comforting story about AI went like this: sure, it can crunch numbers and play chess, but it will never do the human things, the creative things, the intuitive things. That was the line, repeated confidently, for years. I want to gently point out that the line keeps moving, and it is moving in a direction that deserves more honesty than it usually gets.

You have a good retriever and a good model, and the answers still come back vague or wrong. Nine times out of ten the problem is not the model. It is how you cut the documents. Chunking is the least glamorous part of a RAG pipeline and the part that quietly decides whether the whole thing works.

Here is the plain version. Before you can retrieve text, you have to split it into pieces small enough to embed and to fit in a prompt. Each piece becomes a searchable unit. Cut in the wrong place and you hand the model half a thought.

Size is a tradeoff, not a setting

The number everyone asks for first is chunk size. A reasonable default for most documents is 400 to 512 tokens with 10 to 20 percent overlap. That is not a law, it is a starting point that happens to work.

Smaller chunks (128 to 256 tokens) shine on narrow factual questions, because the matching text is dense and the noise around it is low. Larger chunks (512 to 1024 tokens) do better on analytical questions where the answer is spread across a few paragraphs and the model needs the surrounding argument to make sense. If your users ask both kinds of question, and they usually do, pick something in the middle and stop agonizing.

Overlap exists for one reason: sentences do not respect your boundaries. Give each chunk 50 to 100 tokens of its neighbor so a fact that lands on a seam still appears whole somewhere. Too much overlap and you pay to store and search near-duplicate text. Twenty percent is plenty.

Semantic splitting is oversold

The seductive idea is to split on meaning: run an embedding over each sentence, watch for the topic to shift, and cut there. It sounds obviously better than counting characters. In practice it often is not.

A recent benchmark across fifty academic papers put plain recursive splitting at 512 tokens in first place, around 69 percent accuracy, while semantic chunking landed near 54 percent, partly because it produced tiny fragments averaging 43 tokens. Fragments that small have lost the context that made them useful. Semantic chunking can buy you a couple of points of recall on the right corpus, and it costs more compute to build. I reach for it late, after the boring approach is already in place, not first.

What moves the needle more than clever splitting is respecting structure. Split on the document's own seams: headings, sections, list boundaries, the Markdown or HTML skeleton. A recursive splitter that tries paragraph breaks, then sentences, then characters, in that order, keeps most thoughts intact for free.

The mistakes that cost you

A few failure modes show up again and again:

• Stripping the context that says what a chunk is about. A chunk reading "revenue fell 12 percent" is useless if nobody knows it is from the 2023 annual report, EU segment. Prepend the section title, document name, or a one-line summary before you embed it.
• Cutting tables and code down the middle, so half the rows or half the function land in different chunks.
• Chasing the perfect chunk size before you have a single real question to test against.

That last one is the big one. You cannot tune chunking in the abstract. Build a small set of real questions with known answers, change one variable, and measure retrieval. Everything else is superstition.

Chunking rewards attention, not brilliance. Match the document's structure, keep enough context that a stranger could read one chunk and know what it is about, overlap a little. And then go do the unglamorous thing almost nobody does: write down twenty real questions, run them, and let the numbers pick your chunk size for you. The pipeline you can measure is the one you can fix.

A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.

What the window actually is

Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.

What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.

Lost in the middle

The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.

It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.

So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.

The parts nobody puts on the slide

Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.

This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.

When it earns its keep

Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.

The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.

An embedding is a list of numbers that stands in for a piece of text. Feed a sentence to an embedding model and it hands back a few hundred or a few thousand numbers. That list is the whole trick. Text that means similar things lands on similar lists, and text that means different things lands far apart. Once you accept that one idea, most of the useful applications fall out on their own.

The reason this matters is that computers are good at math and bad at meaning. Keyword search knows that "car" and "car" match. It has no idea that "car" and "automobile" are the same thing, or that "my sedan will not start" is closer to "vehicle trouble" than to "sedan chair." Embeddings give the machine a way to measure closeness in meaning instead of closeness in spelling.

How closeness actually works

Picture every sentence as a point in space. Not the three dimensions you live in, but a space with hundreds of them. You cannot draw that, and you do not need to. What you need is a way to ask how close two points are, and the usual answer is cosine similarity: it measures the angle between two of these lists of numbers and returns a score, roughly from minus one to one. Near one means the two texts point the same direction, which is to say they mean roughly the same thing. Near zero means they are unrelated.

You never look at the raw numbers. Nobody reads an embedding. You compute it, store it, and later ask the database which stored vectors sit nearest to a new one. That single operation, find the nearest points, is the engine under almost everything below.

The jobs embeddings are good at

Search is the obvious one, and the one most people meet first. You embed all your documents once. When a user types a question, you embed the question and pull back the passages whose vectors are nearest. The user searches by meaning, so "how do I cancel my plan" finds the paragraph titled "ending a subscription" even though they share almost no words.

Deduplication is quieter but pays for itself fast. If you have a support inbox, a product catalog, or a pile of scraped articles, near-duplicate items are everywhere and exact matching misses them. Embed everything, look for pairs whose similarity crosses a threshold you pick, and you catch the "same thing, slightly reworded" cases that a string comparison walks right past.

Clustering is what you reach for when you do not know your categories yet. Embed a few thousand customer messages, group the vectors that huddle together, and read a handful from each group. The themes come to you instead of you guessing them in advance. It is a genuinely nice way to find out what people are actually writing about.

And then there is retrieval for RAG, which is the same search step wearing a job title. Before a language model answers, you embed the question, fetch the most relevant chunks of your own documents, and hand them to the model as context. The retrieval half of retrieval-augmented generation is embeddings doing exactly what they do in search. Nothing more exotic than that.

Where it gets you into trouble

Embeddings measure similarity, and similarity is not the same as relevance, correctness, or intent. Two sentences can sit close in vector space and mean opposite things, because "the drug is safe" and "the drug is not safe" share almost every word. Negation, sarcasm, and small but load-bearing details are exactly where cosine similarity gets sloppy. The models improve on this every year, but do not assume the nearest neighbor is the right answer. It is the closest guess.

A few practical notes. The model you embed with matters more than the number of dimensions; a good small model beats a mediocre large one. You have to embed your query and your documents with the same model, or the comparison is meaningless. And thresholds are not universal. A similarity of 0.8 might mean "basically identical" for one model and "vaguely related" for another, so test against your own data before you trust a cutoff.

If you take one thing away, let it be this: embeddings are a ruler for meaning. Not a truth machine, not a reasoning engine, just a very good way to ask what is near what. Most of the time that is precisely the question you had.

When a model is not doing what you want, there are three levers people pull: better prompting, retrieval, or fine-tuning. They are wildly different in cost and effort, and teams reach for the expensive one far too early. Here is how to pick the right lever without burning your budget on the wrong one.

Every model has a default voice, and it is the same voice: smooth, agreeable, faintly corporate, allergic to a strong opinion. It is the tone of a brand apologizing. If you want AI to help you write without sounding like everyone else who uses AI, you have to actively drag it away from that default. Here is how.

You wire a language model into your app, ask it to "respond in JSON," and it does. It works the first ten times. Then, on a Tuesday, it wraps the JSON in a markdown code fence, or opens with "Sure, here is the data," or trails off with a comment, and your parser throws, and the pager goes off. "Just ask nicely" is not a strategy. It is a demo that has not failed yet.

The reason it fails is structural. A base language model predicts likely text, and there are a thousand plausible ways to wrap a JSON object in friendly words. A tiny fraction of your calls will land on one of those, and a tiny fraction of a large number is a lot of broken responses. You need something stronger than a polite request.

Say what shape you want, exactly

The first move is to stop describing the output in prose and start defining it as a schema. A JSON Schema, or a typed object in your language that compiles to one, spells out the fields, their types, which are required, and what values are allowed. This does two things. It tells the model precisely what to produce, and it gives you something to validate against afterward, which matters more than people expect.

A schema also kills a whole class of quiet bugs. Without one, the model decides on a Tuesday to call the field "phone_number" instead of "phone," or returns the price as the string "nineteen dollars" instead of the number 19. With a strict schema those drifts either cannot happen or get caught immediately instead of three steps downstream.

Use the mode built for this

Most serious providers now offer a real structured-output feature, and you should use it instead of hoping. There are two flavors worth knowing.

• Structured outputs. You pass a schema and the model is constrained so its output conforms. OpenAI's version, launched in August 2024 with a strict flag, reported 100 percent adherence on their internal schema evals, up from under 40 percent for an older model that was merely asked. That gap is the whole argument.
• Function or tool calling. You describe a function with typed arguments, and the model returns a call to it with the arguments filled in. Same underlying idea, framed as calling code, and the right fit when the JSON is meant to trigger an action.

When the constrained mode is available, the model is not writing free text and then hoping it parses. The valid tokens are restricted as it generates, so malformed JSON becomes close to impossible. That is a completely different reliability story from prompt-and-pray, and it is why the failure rate on the good implementations sits at a fraction of a percent rather than a few percent.

Validate anyway, and retry on purpose

Constrained output handles the shape. It does not handle refusals, truncation from hitting a token limit, or values that are the right type but wrong in context, like a date of February 30. So the last piece is a loop you build yourself.

Parse the response. Validate it against the same schema you sent. If it passes, you are done. If it fails, do not just retry blind: feed the error back to the model, say what broke, and ask it to fix that specific thing. Cap the retries at two or three so a stubborn input cannot spin forever, and log the failures, because a cluster of them usually means your schema is asking for something ambiguous rather than the model misbehaving.

One detail that saves real pain: watch for the refusal path. The good structured-output APIs return a separate refusal field when the model declines, precisely so you do not try to parse an apology as data. Check it before you reach for the parser.

The version of this that holds up in production is boring, and boring is the compliment. A schema the model is constrained to follow, a validation step you never skip, and a short retry loop that tells the model what went wrong. Do that and JSON stops being the flaky part of your pipeline. Skip it, and you will meet the markdown code fence at the worst possible hour.

"Memory" is one of the most oversold words in AI right now. Products promise an assistant that remembers you, learns your preferences, picks up where you left off. Then you use it for a week and notice it forgot the thing you told it on Tuesday. The gap between the pitch and the reality comes from a fact people gloss over: a language model has no memory at all.

The model is a function. Text goes in, text comes out, and the moment the response finishes it retains nothing. Everything we call memory is scaffolding we build around that stateless core, deciding what to put back into the input next time. There are only a few honest ways to do it, and each one breaks in its own way.

Stuffing the context window

The crudest approach: keep the whole conversation and paste it back in every turn. For a short chat this is perfect. The model "remembers" because you literally handed it the transcript.

It falls apart on two edges. Context windows are finite, so a long enough conversation eventually will not fit. And even when it fits, more context is not free. You pay for every token on every call, and models get measurably worse at finding the one relevant line when it is buried in tens of thousands of tokens of history. Stuffing is a real technique with a hard ceiling, not a memory system.

Summaries and their slow leak

So you compress. Every so often, ask the model to summarize the conversation so far and carry the summary forward instead of the raw text. This buys a lot of runway, and most chat products do some version of it.

The catch is that summarizing is lossy on purpose, and you do not get to choose what it loses. The detail that seemed irrelevant when the summary was written is exactly the one that matters three turns later. Summaries of summaries drift further each round, like a photocopy of a photocopy. Good for the gist, unreliable for the specific fact.

Retrieval, which is what "memory" usually means now

The approach that actually scales is to stop trying to hold everything in the prompt. Write facts and past exchanges to a store, and when a new message comes in, fetch only the handful of pieces relevant to it and drop those into the context. This is RAG pointed at your own history instead of a document library.

It is the best of the three and still not what people imagine. Retrieval memory only recalls what it thought to save and what the query happens to match. Phrase your question differently than the stored fact was written and the right memory may never surface. It cannot form the kind of connection a person makes across two things noticed months apart, because nothing is reasoning over the whole store, it is just fetching nearest matches. What you get is a good filing clerk, not a mind.

Here is the thing I wish more products said plainly. None of these give a model memory in the human sense, the kind that reshapes understanding over time. They are all variations on one move: choose what text to put in front of a stateless function on the next call. That is a real and useful engineering problem, and you can build genuinely helpful things by solving it well.

So when you design memory, do not chase the fantasy of an assistant that just knows you. Decide, deliberately, what is worth remembering, how it gets written down, and how it gets found again. Keep the recent turns raw, summarize the middle distance, retrieve the long tail, and accept that each layer forgets in a different way. The teams who ship memory that feels good are not the ones with a magic store. They are the ones who were honest about what a model is and engineered around it.

Every month brings a new open model, and most are footnotes. GLM-5.2 is not. Z.ai released a 753-billion-parameter model, with a one-million-token context window, under a plain MIT license. That last part is what makes it matter.

Most AI bills are not big because the work is hard. They are big because of lazy defaults: the most expensive model on every task, the biggest context every time, and no thought about which jobs actually need the premium option. Here is how to spend far less without your results getting worse.

The single most dangerous thing about a good AI model is how convincing it sounds when it is wrong. It does not hedge, it does not sweat, it just states the confident falsehood in the same tone as the truth. Here is how to catch that before it costs you something, without turning every answer into a research project.

Every model launch comes with a chart where the new model is tallest. The charts are technically true and practically useless, because they are marketing wearing a lab coat. Here is how to read a benchmark like a skeptic, so a leaderboard never again talks you into the wrong model for your actual work.

Most prompt engineering advice is either obvious or superstition. You do not need a 2,000-word mega-prompt or a secret phrase that unlocks the model true power. You need to say what you want the way you would say it to a sharp, literal-minded colleague who has no context and will take you at your word.

The demo worked because you were the only person hitting it. Then ten users showed up, a provider had a slow afternoon, and your app started returning blank screens and half-written answers. None of that is an AI problem. It is a plumbing problem, and the plumbing for AI apps is a little different because the calls are slow, expensive, and occasionally rejected on purpose.

Here is the honest version of what it takes to make a model-backed feature that does not fall over the first time the network has a bad day.

Rate limits are a promise, not an insult

Every provider caps how many requests and tokens you can send per minute. When you cross the line you get an HTTP 429. People treat that response like a failure, but it is closer to a traffic light. The provider is telling you exactly how to behave, and most of the time the answer is written in the headers. Anthropic and OpenAI both send a Retry-After value on many 429s. If it is there, wait that long. Do not invent your own number when the server already told you the truth.

The mistake I see most often is retrying immediately, in a tight loop, across every worker at once. That does not recover from a rate limit. It deepens it. You have now turned one rejected request into fifty, all landing at the same instant, guaranteeing the next fifty get rejected too. This is the thundering herd, and it is self-inflicted.

Retries: back off, and add jitter

The fix is exponential backoff. Wait one second, then two, then four, then eight, doubling each time up to a ceiling. That spaces your attempts out so the system has room to recover. But backoff alone is not enough, because if a hundred clients all failed at the same moment, they will all wait exactly one second and then all retry at exactly the same moment. You have synchronized your herd instead of scattering it.

So you add jitter: a random amount of slop on each wait. Instead of sleeping exactly two seconds, sleep somewhere between zero and two. This one trick, randomizing the delay, does more for stability than almost anything else you can do, and it is two lines of code.

A few rules that keep retries sane:

• Retry on 429 and 5xx and network timeouts. Do not retry a 400 or a 401, because a malformed request or a bad key will fail identically every time.
• Cap the total. Three to five attempts, then give up and surface an error. Infinite retries just hide the outage from you while your users feel every second of it.

Timeouts, and the streaming trap

Model calls are slow and the tail is long. A response that usually takes three seconds will sometimes take forty. If you have no timeout, a single stuck request can hold a connection open until something upstream kills it, and you find out when your whole pool is exhausted. Set an explicit timeout. Pick a number based on your real latency distribution, not on the median, because the median is not what hurts you.

Streaming changes the shape of this. When you stream tokens, the connection can be alive and healthy while the model has quietly stalled and stopped sending. A total request timeout will not catch that cleanly. What you want is an idle timeout: if no token has arrived in some number of seconds, treat the stream as dead and fall back. Watch the gaps between chunks, not just the clock on the whole call.

Idempotency and degrading on purpose

Here is the part people skip until it bites them. You send a request, the model does the work, and then the response gets lost on the way back to you. Your retry logic kicks in and sends it again. Now you have paid for the same completion twice, and if that call also wrote a row or charged a card, you have a duplicate in your data.

Send an idempotency key with anything that has a side effect. It is a unique string per logical operation. If the provider or your own service sees the same key twice, it returns the first result instead of doing the work again. Providers support this for exactly this reason. Use it, and a lost response becomes a non-event instead of a double charge.

Sometimes the model is just down, or slow enough that waiting is worse than not answering. Decide ahead of time what happens then. Maybe you drop to a smaller, cheaper model. Maybe you return a cached answer that is a little stale. Maybe you show the raw search results without the AI summary on top. The worst option is a spinner that never resolves, because that teaches people your product is broken.

None of this is glamorous and none of it shows up in a launch video. But it is the difference between a feature that survives contact with real traffic and one that only ever worked on your laptop. The models get the headlines. The retry loop with jitter is what keeps the thing standing.

Someone in every AI project eventually says it: "We are spending too much on the API. Let us just host our own model." It sounds obvious. You are renting when you could own. But the spreadsheet almost never says what people expect, and the version both sides show you leaves out the line items that decide the whole thing.

The number that looks damning

Per-token API pricing feels expensive when you watch it add up. A busy product can run a real monthly bill, and the temptation is to compare that against a GPU rental and declare victory. A single H100 rents for somewhere between $1.50 and $10 an hour depending on provider and commitment, call it $1,100 to $7,300 a month running steadily. Next to a fat API invoice, renting the GPU can look cheaper.

It usually is not, because the GPU is the part of self-hosting you can see. The rest hides.

The costs that do not show up on the GPU bill

Running your own inference is not renting a graphics card. It is standing up a service. That means someone who knows how to serve models, batch requests, handle load, patch the stack, and get paged at 3am when it falls over. MLOps engineers are not cheap, and their time is the real cost. Reasonable estimates put the fully loaded cost of self-hosting at three to five times the raw GPU price once you count the people and the plumbing.

Then there is utilization, the quiet killer. The API charges you per token, so you pay for exactly what you use. Your rented GPU charges you per hour whether it is busy or idle. Traffic is spiky. If your GPU sits at 15 percent utilization most of the day, you are paying full price for an empty machine, and your effective cost per token climbs above what the API charged.

Where the break-even actually sits

Concrete numbers help. Against premium APIs, self-hosting a comparable open model tends to break even somewhere around 5 to 10 million tokens a month, and only if you already have the ops capacity. Against the cheap tier, the mini and small models that cost almost nothing per token, the line moves way out: you might need hundreds of millions of tokens a month before owning beats renting.

One worked example from a 2026 analysis: 50 million tokens a day through a small hosted model came to about $2,250 a month on the API, versus roughly $5,175 self-hosted on four mid-range GPUs. More than double, to run the exact same workload yourself.

So the honest brackets look like this:

• Under about 50M tokens a month: the API wins almost every time. Do not self-host to save money, you will not.
• 50M to 500M against frontier-priced APIs, with real MLOps staff on hand: now it is a genuine decision.
• Sustained heavy volume against comparable open models: self-hosting can cut cost several fold, and at that point you probably already know who you are.

There are reasons to self-host that have nothing to do with cost. Data that legally cannot leave your walls. Latency you cannot get from a shared endpoint. A model you have fine-tuned and want to own outright. Those are real, and they can justify the bill on their own. Just do not dress them up as savings, because for most teams below serious scale, self-hosting is the more expensive choice wearing a thrifty costume. If you are moving to it, move for control or privacy, name that out loud, and go in knowing the GPU line was the cheap part.

For years the deal was simple: Microsoft paid, OpenAI built, and Copilot ran on someone else models. At Build 2026, Microsoft quietly changed the arrangement by shipping seven models of its own. The most important AI news of the month was not a benchmark. It was a message.

Ask a language model a question it has no business answering and watch what happens. It answers. Fluently, in complete sentences, with the same even tone it uses for the things it actually knows. The confidence does not move. That gap, between how sure the model sounds and how sure it should be, is the single most dangerous thing about the current generation of these tools.

People call this hallucination and treat it like a bug that a bigger model will fix. It is not a bug. It is baked into how the models are trained.

We taught them to guess

Think about how a model is graded. On most benchmarks a correct answer scores a point, a wrong answer scores zero, and "I do not know" also scores zero. Line those incentives up and the math is brutal: if abstaining pays the same as being wrong, you should always guess. A model that guesses on every uncertain question will beat an honest one that admits its gaps.

So that is the model we built. OpenAI researchers have made the point directly: under binary grading, systems are rewarded for guessing and penalized for saying they are unsure. We optimized for good test-takers, and good test-takers do not leave answers blank. The confident wrong answer is not the model failing. It is the model doing exactly what we rewarded.

Confidence is not knowledge

Here is the part that trips up smart people. We are wired to read fluency as competence. Someone who speaks in clear, structured, well-organized sentences usually knows their subject, because for humans that fluency was expensive to fake. Models make it free. A model produces the same polished prose whether it is reciting a fact or inventing one, and our instinct to trust the smooth talker fires anyway.

That is why "always check the output" is weaker advice than it sounds. Checking is exactly the effort the fluent answer discourages. The better the prose, the less you feel the urge to verify, and the more the one wrong sentence in twenty slips through.

Designing for a witness who never doubts

I do not think the answer is to distrust these tools. I use them every day and they are genuinely good. The answer is to stop treating a model like a source and start treating it like a very fast, very well-read assistant who is constitutionally incapable of saying "I am not sure." Once you accept that, the design choices get clearer.

Put the model where being wrong is cheap and visible: drafting, brainstorming, summarizing text you can see, writing code you are about to run and test. Keep it away from places where a confident fabrication is expensive and hard to catch, like a medical dose, a legal citation, a number that flows straight into a decision with no human in the loop. Retrieval helps, because grounding an answer in a document you can inspect turns "trust me" into "here is where it came from." And when you can, prefer systems that surface uncertainty at all, even a rough confidence score, over ones that render everything in the same calm voice.

There is real research now on training models to abstain, to earn credit for admitting a gap instead of being punished for it. I hope it works. Until it ships, the burden sits with us, and the honest move is to build workflows that assume the machine will never tell you when it is out of its depth. It will keep talking. The question worth sitting with is why we find that so much more comforting than a system that occasionally, usefully, went quiet.

The open-versus-closed argument gets treated like a team you join, complete with jerseys. It is not. It is a trade-off you make one job at a time, and the same person can sensibly land on opposite answers for two projects in the same week. What follows is a decision guide, not a leaderboard, because the leaderboard changes monthly and the trade-off underneath it does not.

What each side actually gives you

Open-weights models, the Llamas and DeepSeeks and Qwens of the world, hand you the actual model. You can download it, run it on your own hardware or a rented box, look at what it does, fine-tune it on your data, and keep the whole thing behind your own firewall. Nobody meters your calls. Nobody can change the model out from under you or deprecate it next quarter. If your data cannot legally or comfortably leave your walls, this is often the only real option.

Closed models, reached through an API, hand you a result. You send text, you get text back, and someone else owns the running of it. In exchange for giving up control you get the current frontier of quality, no infrastructure to babysit, and a model that quietly improves without you lifting a finger. For a lot of teams that is the entire pitch and it is a good one: you want the answer, not a second job running GPUs.

The costs hide in different places

People compare these on the wrong axis. They look at the per-token price of a closed API, see a number bigger than zero, and conclude that self-hosting an open model is cheaper. Sometimes. The closed price includes the hardware, the scaling, the uptime, and the salaries of people who keep it running. Open weights are free to download and very much not free to operate. You are now buying or renting GPUs, and someone on your team owns keeping the thing up at 3 a.m.

The honest version is about volume and steadiness. Spiky, low, or unpredictable traffic almost always favors the closed API, because you pay only for what you use and nothing when you are idle. Heavy, steady, round-the-clock traffic is where owning the hardware can win, because a machine you have already paid for does not care how many calls you push through it. The crossover is real, but it sits much further out than the sticker-price comparison suggests, and it moves every time GPU rental prices or token prices shift.

How to actually choose

Skip the identity and answer a few blunt questions about the specific job.

• Where is the data allowed to go? If it legally cannot leave your infrastructure, that decides it before any quality debate starts. Open weights, self-hosted, done.
• Do you need the absolute top of the quality range? For the hardest reasoning, the newest capabilities, the widest language coverage, the closed frontier models still tend to lead, and the gap is often worth paying for. For a well-scoped task, a mid-size open model may clear the bar with room to spare and cost far less to run.
• Does the model changing under you break things? If you have tuned prompts against exact behavior and cannot afford a silent update, a weights file you pin and control has an edge a hosted endpoint cannot match.
• Do you have people to run it? Serving a model in production is real, ongoing engineering. If that team does not exist, the API is not a compromise, it is the sane choice.

The gap keeps closing, the choice does not

The genuinely good news is that open weights have gotten shockingly close to the closed frontier on many everyday tasks. For summarizing, extraction, classification, ordinary chat, drafting, the practical difference is often small enough not to matter, and it keeps shrinking. That is a real shift and worth being cheerful about.

It does not, however, dissolve the trade-off. Control and privacy and the ops burden that comes with them sit on one side. Convenience and frontier quality and someone else's pager sit on the other. That tension is structural. It will still be here when today's model names are forgotten. The people who get the most out of AI are not the ones who picked a side and defended it. They are the ones who ask, for this specific job, which set of headaches they would rather have, and then pick accordingly.

If you are new to this and trying to pick your first AI model, the internet will hand you forty benchmark charts and a headache. Ignore them. The right first model has almost nothing to do with leaderboard trivia and almost everything to do with what you are actually going to do. Here is the short, honest version.