You run a prompt, get a good answer, run the exact same prompt again, and get something different. Sometimes it is a rephrasing. Sometimes it is a different conclusion. If you come from normal software, where the same input gives the same output forever, this feels like the thing is broken. It is not broken. It is doing exactly what it was built to do, and understanding why will change how you use these tools.
The model does not pick a word, it picks from a distribution
At every step, a language model does not decide on the next word. It produces a probability for every possible next token. The word "blue" might get 60 percent, "green" 20 percent, "warm" 5 percent, and a long tail of everything else splitting the rest. Then a sampler reaches into that distribution and draws one.
Temperature is the knob that reshapes those odds before the draw. Turn it up and the flat, unlikely options get more weight, so the output wanders and surprises you. Turn it down and probability piles onto the front-runners, so the model plays it safe. This is a feature. You want a brainstorming assistant to roam and a data-extraction call to stay boring. The variation you are seeing at normal temperature is the model exploring, on purpose.
So the obvious move is to set temperature to 0. Now it always takes the single most likely token, greedy decoding, no dice roll. Same prompt, same answer, forever. Right?
Temperature 0 is not the guarantee you think it is
In practice, temperature 0 gets you close but not all the way. People assumed for years that the leftover wobble was just floating-point noise, some vague hand-wave about GPUs being messy. That answer was never satisfying, and in 2025 researchers at Thinking Machines laid out the actual culprit, and it is more interesting than randomness.
The real cause is batch invariance, or rather the lack of it. When you send a request to a hosted model, you are not alone. The serving system bundles your request together with whatever other requests arrived at the same moment and runs them through the GPU as one batch. That batch is different every time, because it depends on who else is calling the API at that instant, something you have zero control over.
Here is why that matters. Floating-point addition is not associative. Adding a set of numbers in one order can give a very slightly different result than adding them in another order, down in the last bits. When the batch size changes, the GPU kernels split and sum their work in a different order, so the model's internal numbers drift by a hair. Almost always that hair is invisible. But every so often two candidate tokens are nearly tied, the drift nudges one above the other, and the model picks a different word. From there the two answers diverge, because each new token conditions on the last one.
Sit with that for a second. The nondeterminism you see is not the model being random. It is your request sharing a GPU with strangers. The cause is numerical, not magical, and it is fixable. Batch-invariant kernels that pin the reduction order now ship in serving engines like vLLM and SGLang, and with them you can get genuinely reproducible output. Most hosted APIs do not turn that on by default, because it costs a bit of throughput, so the wobble stays.
How to actually think about it
Stop treating a single model output as a fact and start treating it as a sample. One run tells you what the model tends to say. It does not tell you what it always says. That reframe fixes a lot of bad habits.
If you need the same answer every time, do not lean on the model's internals to give it to you. Cache the result keyed on the input, and serve the cached copy. That is deterministic by construction and it is cheaper. If you are extracting structured data, run the call, but validate the shape and constrain the output with a schema so a reworded answer cannot break your parser. If you are evaluating a prompt, never judge it on one run. Sample it ten times and look at the spread, because the spread is the real behavior and a single lucky run is a trap that will embarrass you in production.
And when a demo shows you one perfect output, remember what you are looking at. You are seeing one draw from a distribution, chosen and probably cherry-picked. The honest question is not whether it can produce that answer. It is how often, and what the other draws look like. Ask for a few more and you will learn more in thirty seconds than the polished screenshot will ever tell you.