Most AI bills are bloated with answers you already paid for. The same question comes in, phrased the same way, and you send it to the model again like you have never seen it before. Caching is how you stop doing that. It is the least glamorous cost lever there is and usually the most effective, and there are three different flavors that people constantly mix up.
Exact-match caching: the free lunch you skipped
The simplest version is a lookup table. Hash the exact input, store the model's output against that hash, and the next time the identical input arrives you return the stored answer instead of calling the model. No tokens spent, no latency, no provider involved. For anything with repeated identical queries, autocomplete suggestions, canned support answers, the same document summarized twice, this is close to free money.
The catch is the word exact. One extra space, a different capital letter, a trailing newline, and the hash changes and you miss the cache. So normalize before you hash. Trim whitespace, settle on a case convention, sort any parameters that do not have a meaningful order. And be honest about your hit rate. If every user query is genuinely unique, an exact cache buys you almost nothing, and you should not pretend otherwise.
Semantic caching: same meaning, different words
This is where it gets interesting and where it gets dangerous. "What is your refund policy?" and "How do I get my money back?" are different strings but the same question. Semantic caching catches that. You embed the incoming query into a vector, compare it against the vectors of things you have already answered, and if one is close enough you return that stored answer.
The whole game is the word "enough." You set a similarity threshold. Set it too loose and you will serve the answer to a neighboring question that is not actually the same, which is worse than a cache miss because it is a confidently wrong answer. "How do I cancel my subscription?" and "How do I pause my subscription?" sit very close in embedding space and mean different things with different consequences. Set the threshold too tight and you catch nothing and paid for the embedding infrastructure for no reason.
My rule: use semantic caching where a near-miss is cheap, like surfacing help articles, and stay away from it where a near-miss is expensive, like anything touching money, account state, or legal wording. Test it on real query logs, not on the three examples that made you want to build it.
Provider prompt caching: caching the prefix, not the answer
The third kind is different in nature. Provider prompt caching does not cache the final answer at all. It caches the model's processing of a long, repeated chunk of your input, typically a big system prompt, a set of tool definitions, or a document you keep asking questions about. The model still runs and still generates a fresh answer, but it skips re-reading the part it already read.
The economics are strong and the two big providers took opposite paths. Anthropic makes you mark the cacheable section explicitly with a cache breakpoint. A cache write costs a bit more than normal input, 1.25 times for the short-lived tier, but a cache read costs one tenth of the input price, a 90 percent discount, with a 1,024-token minimum. OpenAI does it automatically with no code change and reads at roughly half price. Anthropic asks for effort and rewards you more for it. OpenAI asks for nothing and rewards you less. Neither is wrong, they are just different bargains.
Prompt caching shines in exactly the case that would otherwise be brutal: a chat over a long document, or an agent carrying a fat system prompt through many turns. The repeated prefix gets cheap. The part that changes stays full price. Just know these caches are short-lived, minutes for the default tiers, so they help within a session and do nothing across a quiet night.
The pitfall that eats everyone: staleness
Here is the trap that turns a cost win into a support ticket. A cache is a bet that the right answer has not changed since you stored it. When the underlying truth moves and the cache does not, you serve yesterday's answer with today's confidence.
If your model answers over data that changes, prices, inventory, a policy doc, the account balance, your cache needs a way to know when that data moved. The clean move is to build the cache key out of a version of the source, so when the document updates, the key changes and old entries fall away on their own. The lazy move is a time-to-live: expire everything after an hour and accept up to an hour of staleness. TTL is fine for slow-moving content and quietly dangerous for anything a user expects to be current.
Start with exact-match caching, because it is safe and it works. Add prompt caching if you have long repeated prefixes, because the discount is real and the risk is near zero. Reach for semantic caching last, deliberately, and only where being a little bit wrong is genuinely fine. The savings are real. So is the bill from a cache that got too clever.