AI Reviews
Someone in every AI project eventually says it: "We are spending too much on the API. Let us just host our own model." It sounds obvious. You are renting when you could own. But the spreadsheet almost never says what people expect, and the version both sides show you leaves out the line items that decide the whole thing.
The number that looks damning
Per-token API pricing feels expensive when you watch it add up. A busy product can run a real monthly bill, and the temptation is to compare that against a GPU rental and declare victory. A single H100 rents for somewhere between $1.50 and $10 an hour depending on provider and commitment, call it $1,100 to $7,300 a month running steadily. Next to a fat API invoice, renting the GPU can look cheaper.
It usually is not, because the GPU is the part of self-hosting you can see. The rest hides.
The costs that do not show up on the GPU bill
Running your own inference is not renting a graphics card. It is standing up a service. That means someone who knows how to serve models, batch requests, handle load, patch the stack, and get paged at 3am when it falls over. MLOps engineers are not cheap, and their time is the real cost. Reasonable estimates put the fully loaded cost of self-hosting at three to five times the raw GPU price once you count the people and the plumbing.
Then there is utilization, the quiet killer. The API charges you per token, so you pay for exactly what you use. Your rented GPU charges you per hour whether it is busy or idle. Traffic is spiky. If your GPU sits at 15 percent utilization most of the day, you are paying full price for an empty machine, and your effective cost per token climbs above what the API charged.
Where the break-even actually sits
Concrete numbers help. Against premium APIs, self-hosting a comparable open model tends to break even somewhere around 5 to 10 million tokens a month, and only if you already have the ops capacity. Against the cheap tier, the mini and small models that cost almost nothing per token, the line moves way out: you might need hundreds of millions of tokens a month before owning beats renting.
One worked example from a 2026 analysis: 50 million tokens a day through a small hosted model came to about $2,250 a month on the API, versus roughly $5,175 self-hosted on four mid-range GPUs. More than double, to run the exact same workload yourself.
So the honest brackets look like this:
- Under about 50M tokens a month: the API wins almost every time. Do not self-host to save money, you will not.
- 50M to 500M against frontier-priced APIs, with real MLOps staff on hand: now it is a genuine decision.
- Sustained heavy volume against comparable open models: self-hosting can cut cost several fold, and at that point you probably already know who you are.
There are reasons to self-host that have nothing to do with cost. Data that legally cannot leave your walls. Latency you cannot get from a shared endpoint. A model you have fine-tuned and want to own outright. Those are real, and they can justify the bill on their own. Just do not dress them up as savings, because for most teams below serious scale, self-hosting is the more expensive choice wearing a thrifty costume. If you are moving to it, move for control or privacy, name that out loud, and go in knowing the GPU line was the cheap part.
The reflex is to reach for the biggest model available and call it a day. It usually works, which is exactly why it is a trap. The frontier model is the safe default, not the right answer, and for a surprising share of real tasks a small model running on your own hardware does the job faster, cheaper, and without shipping your data anywhere. The interesting question is not whether small models can win. They can. It is how to know when yours will.
Where small actually wins
Four advantages, and they are concrete. Latency: a small model on a local GPU answers in tens of milliseconds with no network hop, which is the difference between an interface that feels alive and one that feels like waiting. Cost: the gap is not subtle. Analysts have put the swing at more than thirty times per token when you move a high-volume task off a frontier API onto a small model you host. At a million calls a day, that stops being a rounding error and starts being a line item someone asks about. Privacy: if the model runs on your machine, the data never leaves it, which for medical, legal, or internal work is not a nice-to-have but the whole ballgame. Focus: on a narrow, well-defined task, a small model tuned for it often matches or beats a giant generalist, because it is not spending capacity on the ten thousand things you will never ask.
That last point is the one people underrate. Microsoft's Phi-4, at 14 billion parameters, lands near models five to fifteen times its size on math and coding benchmarks. The old assumption that bigger simply means better broke somewhere in 2024, and it has not been true as a blanket rule since. A tightly scoped classifier, extractor, or router is precisely the shape of task where a small model shines.
How to actually test it
Do not trust the benchmark leaderboard and do not trust your gut. Both are measuring the wrong thing. The benchmark tests a generic task; you care about your task. Run the bake-off yourself, and it is less work than it sounds.
Build a small evaluation set from your real inputs. A hundred examples that look like production traffic beats ten thousand synthetic ones. For each, know what a good answer is, whether that is an exact label, an acceptable range, or a human judgment you write down in advance. Then run both models, the big one and the small candidate, over the identical set. Now you can compare the thing that matters: not "which is smarter" but "which is good enough for this," alongside cost per call and time per call.
Two habits make this honest. Use the big model as your reference for what "correct" looks like, but grade the small one against the task, not against the big one's exact wording. And look at the failures individually, not just the aggregate score. Small models tend to fail in clusters, on one input type or one edge case, and often you can fix a whole cluster with a better prompt or a few examples rather than reaching back for the giant.
When not to bother
I will save you some time. If the task is genuinely open-ended reasoning, long multi-step chains, hard code across a big context, nuanced writing that has to be right the first time, the frontier model still earns its price, and forcing a small model onto it is false economy. If your volume is low, a few hundred calls a day, the cost argument evaporates and the engineering time to host and maintain a local model is not worth recovering pennies. And if you do not have an evaluation set, you cannot tell whether the swap worked, so build that first or do not swap at all.
The right mental model is a portfolio, not a champion. Route the narrow, high-volume, latency-sensitive, or privacy-bound work to a small local model, and keep the frontier model for the hard cases where its extra capability actually shows up in your results. The teams getting real leverage out of this are not the ones who found the single best model. They are the ones who stopped asking that question and started matching the model to the job.
The open-versus-closed argument gets treated like a team you join, complete with jerseys. It is not. It is a trade-off you make one job at a time, and the same person can sensibly land on opposite answers for two projects in the same week. What follows is a decision guide, not a leaderboard, because the leaderboard changes monthly and the trade-off underneath it does not.
What each side actually gives you
Open-weights models, the Llamas and DeepSeeks and Qwens of the world, hand you the actual model. You can download it, run it on your own hardware or a rented box, look at what it does, fine-tune it on your data, and keep the whole thing behind your own firewall. Nobody meters your calls. Nobody can change the model out from under you or deprecate it next quarter. If your data cannot legally or comfortably leave your walls, this is often the only real option.
Closed models, reached through an API, hand you a result. You send text, you get text back, and someone else owns the running of it. In exchange for giving up control you get the current frontier of quality, no infrastructure to babysit, and a model that quietly improves without you lifting a finger. For a lot of teams that is the entire pitch and it is a good one: you want the answer, not a second job running GPUs.
The costs hide in different places
People compare these on the wrong axis. They look at the per-token price of a closed API, see a number bigger than zero, and conclude that self-hosting an open model is cheaper. Sometimes. The closed price includes the hardware, the scaling, the uptime, and the salaries of people who keep it running. Open weights are free to download and very much not free to operate. You are now buying or renting GPUs, and someone on your team owns keeping the thing up at 3 a.m.
The honest version is about volume and steadiness. Spiky, low, or unpredictable traffic almost always favors the closed API, because you pay only for what you use and nothing when you are idle. Heavy, steady, round-the-clock traffic is where owning the hardware can win, because a machine you have already paid for does not care how many calls you push through it. The crossover is real, but it sits much further out than the sticker-price comparison suggests, and it moves every time GPU rental prices or token prices shift.
How to actually choose
Skip the identity and answer a few blunt questions about the specific job.
- Where is the data allowed to go? If it legally cannot leave your infrastructure, that decides it before any quality debate starts. Open weights, self-hosted, done.
- Do you need the absolute top of the quality range? For the hardest reasoning, the newest capabilities, the widest language coverage, the closed frontier models still tend to lead, and the gap is often worth paying for. For a well-scoped task, a mid-size open model may clear the bar with room to spare and cost far less to run.
- Does the model changing under you break things? If you have tuned prompts against exact behavior and cannot afford a silent update, a weights file you pin and control has an edge a hosted endpoint cannot match.
- Do you have people to run it? Serving a model in production is real, ongoing engineering. If that team does not exist, the API is not a compromise, it is the sane choice.
The gap keeps closing, the choice does not
The genuinely good news is that open weights have gotten shockingly close to the closed frontier on many everyday tasks. For summarizing, extraction, classification, ordinary chat, drafting, the practical difference is often small enough not to matter, and it keeps shrinking. That is a real shift and worth being cheerful about.
It does not, however, dissolve the trade-off. Control and privacy and the ops burden that comes with them sit on one side. Convenience and frontier quality and someone else's pager sit on the other. That tension is structural. It will still be here when today's model names are forgotten. The people who get the most out of AI are not the ones who picked a side and defended it. They are the ones who ask, for this specific job, which set of headaches they would rather have, and then pick accordingly.
Coding assistant comparisons usually happen on toy problems with clean answers, which is exactly why they are useless. So I did the opposite. I took one genuinely annoying real bug, the kind that spans a few files and does not announce itself, and pointed three assistants at it: Claude, GPT, and Microsoft new MAI coding model in Copilot. Here is how they actually did.
Read more: Claude, GPT, and MAI on the same real bug: a coding assistant face-off
Everyone reviews the frontier. Almost nobody carefully reviews the budget shelf, which is a shame, because that is where most real work should actually run. I spent time putting the cheap models through the same ordinary tasks I use every day. The winner was not the one I expected, and the losers were instructive.
Read more: The best cheap LLM in 2026 is probably not the one you think