Pricing

Most AI bills are bloated with answers you already paid for. The same question comes in, phrased the same way, and you send it to the model again like you have never seen it before. Caching is how you stop doing that. It is the least glamorous cost lever there is and usually the most effective, and there are three different flavors that people constantly mix up.

Exact-match caching: the free lunch you skipped

The simplest version is a lookup table. Hash the exact input, store the model's output against that hash, and the next time the identical input arrives you return the stored answer instead of calling the model. No tokens spent, no latency, no provider involved. For anything with repeated identical queries, autocomplete suggestions, canned support answers, the same document summarized twice, this is close to free money.

The catch is the word exact. One extra space, a different capital letter, a trailing newline, and the hash changes and you miss the cache. So normalize before you hash. Trim whitespace, settle on a case convention, sort any parameters that do not have a meaningful order. And be honest about your hit rate. If every user query is genuinely unique, an exact cache buys you almost nothing, and you should not pretend otherwise.

Semantic caching: same meaning, different words

This is where it gets interesting and where it gets dangerous. "What is your refund policy?" and "How do I get my money back?" are different strings but the same question. Semantic caching catches that. You embed the incoming query into a vector, compare it against the vectors of things you have already answered, and if one is close enough you return that stored answer.

The whole game is the word "enough." You set a similarity threshold. Set it too loose and you will serve the answer to a neighboring question that is not actually the same, which is worse than a cache miss because it is a confidently wrong answer. "How do I cancel my subscription?" and "How do I pause my subscription?" sit very close in embedding space and mean different things with different consequences. Set the threshold too tight and you catch nothing and paid for the embedding infrastructure for no reason.

My rule: use semantic caching where a near-miss is cheap, like surfacing help articles, and stay away from it where a near-miss is expensive, like anything touching money, account state, or legal wording. Test it on real query logs, not on the three examples that made you want to build it.

Provider prompt caching: caching the prefix, not the answer

The third kind is different in nature. Provider prompt caching does not cache the final answer at all. It caches the model's processing of a long, repeated chunk of your input, typically a big system prompt, a set of tool definitions, or a document you keep asking questions about. The model still runs and still generates a fresh answer, but it skips re-reading the part it already read.

The economics are strong and the two big providers took opposite paths. Anthropic makes you mark the cacheable section explicitly with a cache breakpoint. A cache write costs a bit more than normal input, 1.25 times for the short-lived tier, but a cache read costs one tenth of the input price, a 90 percent discount, with a 1,024-token minimum. OpenAI does it automatically with no code change and reads at roughly half price. Anthropic asks for effort and rewards you more for it. OpenAI asks for nothing and rewards you less. Neither is wrong, they are just different bargains.

Prompt caching shines in exactly the case that would otherwise be brutal: a chat over a long document, or an agent carrying a fat system prompt through many turns. The repeated prefix gets cheap. The part that changes stays full price. Just know these caches are short-lived, minutes for the default tiers, so they help within a session and do nothing across a quiet night.

The pitfall that eats everyone: staleness

Here is the trap that turns a cost win into a support ticket. A cache is a bet that the right answer has not changed since you stored it. When the underlying truth moves and the cache does not, you serve yesterday's answer with today's confidence.

If your model answers over data that changes, prices, inventory, a policy doc, the account balance, your cache needs a way to know when that data moved. The clean move is to build the cache key out of a version of the source, so when the document updates, the key changes and old entries fall away on their own. The lazy move is a time-to-live: expire everything after an hour and accept up to an hour of staleness. TTL is fine for slow-moving content and quietly dangerous for anything a user expects to be current.

Start with exact-match caching, because it is safe and it works. Add prompt caching if you have long repeated prefixes, because the discount is real and the risk is near zero. Reach for semantic caching last, deliberately, and only where being a little bit wrong is genuinely fine. The savings are real. So is the bill from a cache that got too clever.

The AI safety conversation is obsessed with the ceiling: the smartest model, the frontier, the hypothetical superintelligence. I think we are watching the wrong number. The change that will actually reshape the world this decade is not that the best model got smarter. It is that a good-enough model got almost free.

DeepSeek V4 arrived in April with the same trick DeepSeek always pulls: numbers that should cost a fortune, at prices that do not. Top-tier coding scores, a million-token context, an MIT license, and output priced under a dollar per million tokens. The West still has not quite made peace with it.

Most AI bills are not big because the work is hard. They are big because of lazy defaults: the most expensive model on every task, the biggest context every time, and no thought about which jobs actually need the premium option. Here is how to spend far less without your results getting worse.

Kimi K2 belongs to my favorite category of model: the one that makes you recheck the pricing page because the numbers look like a typo. Near-frontier reasoning, notably cheaper than the big names, and it gets to the answer using fewer tokens than its predecessor. Mostly, it holds up. Mostly.

Someone in every AI project eventually says it: "We are spending too much on the API. Let us just host our own model." It sounds obvious. You are renting when you could own. But the spreadsheet almost never says what people expect, and the version both sides show you leaves out the line items that decide the whole thing.

The number that looks damning

Per-token API pricing feels expensive when you watch it add up. A busy product can run a real monthly bill, and the temptation is to compare that against a GPU rental and declare victory. A single H100 rents for somewhere between $1.50 and $10 an hour depending on provider and commitment, call it $1,100 to $7,300 a month running steadily. Next to a fat API invoice, renting the GPU can look cheaper.

It usually is not, because the GPU is the part of self-hosting you can see. The rest hides.

The costs that do not show up on the GPU bill

Running your own inference is not renting a graphics card. It is standing up a service. That means someone who knows how to serve models, batch requests, handle load, patch the stack, and get paged at 3am when it falls over. MLOps engineers are not cheap, and their time is the real cost. Reasonable estimates put the fully loaded cost of self-hosting at three to five times the raw GPU price once you count the people and the plumbing.

Then there is utilization, the quiet killer. The API charges you per token, so you pay for exactly what you use. Your rented GPU charges you per hour whether it is busy or idle. Traffic is spiky. If your GPU sits at 15 percent utilization most of the day, you are paying full price for an empty machine, and your effective cost per token climbs above what the API charged.

Where the break-even actually sits

Concrete numbers help. Against premium APIs, self-hosting a comparable open model tends to break even somewhere around 5 to 10 million tokens a month, and only if you already have the ops capacity. Against the cheap tier, the mini and small models that cost almost nothing per token, the line moves way out: you might need hundreds of millions of tokens a month before owning beats renting.

One worked example from a 2026 analysis: 50 million tokens a day through a small hosted model came to about $2,250 a month on the API, versus roughly $5,175 self-hosted on four mid-range GPUs. More than double, to run the exact same workload yourself.

So the honest brackets look like this:

• Under about 50M tokens a month: the API wins almost every time. Do not self-host to save money, you will not.
• 50M to 500M against frontier-priced APIs, with real MLOps staff on hand: now it is a genuine decision.
• Sustained heavy volume against comparable open models: self-hosting can cut cost several fold, and at that point you probably already know who you are.

There are reasons to self-host that have nothing to do with cost. Data that legally cannot leave your walls. Latency you cannot get from a shared endpoint. A model you have fine-tuned and want to own outright. Those are real, and they can justify the bill on their own. Just do not dress them up as savings, because for most teams below serious scale, self-hosting is the more expensive choice wearing a thrifty costume. If you are moving to it, move for control or privacy, name that out loud, and go in knowing the GPU line was the cheap part.

Everyone reviews the frontier. Almost nobody carefully reviews the budget shelf, which is a shame, because that is where most real work should actually run. I spent time putting the cheap models through the same ordinary tasks I use every day. The winner was not the one I expected, and the losers were instructive.

Quietly, without a single headline capturing it, the cost of using capable AI collapsed this year. Not dropped. Collapsed. The kind of model that cost a small fortune to run at scale two years ago now costs cents per million tokens. If your business plan assumed AI would stay expensive, it is time to redo the math.

A token is the thing you are actually paying for, and almost nobody thinks in tokens. They think in words, or messages, or vibes. Then the invoice arrives and the math does not line up. Here is the plain version of what a token is, how your text gets chopped into them, and why the model bills this way instead of the way you would expect.

Not a word, not a letter, something in between

When you send text to a model, the first thing that happens is that a piece of software called a tokenizer breaks your text into pieces. A token is one of those pieces. Sometimes a token is a whole word. Often it is a fragment. The word "sleeping" might become "sleep" plus "ing". Common words like "the" are usually a single token, while a rare or long word gets split into two or three.

The rough rule people quote is that one token is about four characters of English, or roughly three quarters of a word. OpenAI's own guidance says about 750 words works out to around 1000 tokens. That heuristic is fine for a quick estimate and wrong the moment your text stops looking like a plain paragraph. Code, punctuation, JSON, and tables all push the count up, because symbols and indentation do not compress into neat word-sized chunks. So does anything not in English. The tokenizers were trained mostly on English, so "strawberry" costs one or two tokens while the same word in Chinese or Arabic can cost several times more. Same meaning, bigger bill.

Why chop words up at all? Because the model has a fixed vocabulary, usually somewhere between 50,000 and 100,000 tokens. If every distinct word needed its own entry, the vocabulary would explode and the model would choke on any word it had never seen. Subword tokens are the compromise. The model learns a set of common fragments and reassembles anything, including words that did not exist when it was trained, out of pieces it already knows.

Why the meter runs on tokens

Providers charge per token because tokens are what the model literally processes, one step at a time. Every token in your prompt has to be read before the model writes anything, and every token it writes is another unit of compute. There is no natural billing unit called a "word" inside the machine. There are only tokens going in and tokens coming out.

That is also why input and output are usually priced differently, with output costing more. Generating a token is more expensive than reading one, because the model does a full forward pass for each new token it produces. When you see two prices on a pricing page, that is what the gap is about.

Where the bill bites

The practical consequences follow directly, and they are easy to miss until they cost you money.

• Long context is not free context. If you paste a 40-page document into every request, you pay for all of it on every request, whether or not the model needed most of it to answer. A chat that remembers the whole conversation is re-reading and re-charging that history each turn.
• Verbose system prompts are a recurring tax. That 800-token instruction block you wrote once gets billed on every single call.
• "Make it concise" saves output tokens, which are the expensive ones. Asking for a one-line answer instead of an essay is a real cost lever, not just a style choice.

None of this means you should obsess over trimming tokens like a coupon-clipper. Most single requests cost a fraction of a cent. It matters at volume. A prompt that is 30 percent heavier than it needs to be does not hurt when you run it twice. Run it two million times and it is a line item someone will ask you about.

Count before you commit

The one habit worth building: stop estimating in words when real money is involved. Run your actual prompts, the messy ones with the code blocks and the pasted logs and the non-English names, through a token counter before you ship. Most providers give you one, and the number is almost always higher than your gut said, because your gut counts words and the model counts something stranger.

The word is a unit humans invented for humans. The token is a unit the machine invented for itself. Your bill is written in the machine's units, and once you can see the text the way the tokenizer sees it, the invoice stops being a surprise and starts being something you can actually plan around.