Somewhere in every LLM tutorial there is a line that sets temperature to 0.7, and a remarkable number of people copy it, ship it, and never think about it again. Then the output goes weird and they have no idea which knob to turn. Temperature and top-p are the two settings that decide how the model picks each next word, they are genuinely simple once explained, and most people are using them on autopilot. Here is what they actually do.
The model is always guessing, these settings pick how
At every step, the model produces a ranked list of possible next tokens, each with a probability. It does not know the answer, it has a distribution over answers. Temperature and top-p are two different ways of deciding how to draw from that distribution.
Temperature reshapes the whole distribution. Low temperature, near zero, sharpens it so the single most likely token almost always wins, which makes the output predictable and repeatable. Crank it up toward one and beyond and you flatten the distribution, giving unlikely tokens a real shot and making the output more varied and, past a point, more unhinged. At temperature zero the model is effectively deterministic: same prompt, same answer, every time.
Top-p, also called nucleus sampling, works differently. Instead of reshaping probabilities, it draws a cutoff. Top-p of 0.9 means "consider only the smallest set of tokens whose probabilities add up to 90 percent, and ignore the long tail." The clever part is that it adapts to the model's confidence. When the model is sure, one token already holds most of the probability, so top-p samples from just a token or two. When the model is unsure and fifty tokens each hold a little, top-p keeps all fifty in play. It widens and narrows on its own.
Where people go wrong
The first mistake is turning both knobs at once. Temperature and top-p both constrain the same sampling step, and stacking them makes the effect of either impossible to reason about. The standard advice, which I agree with, is to move one and leave the other at its neutral setting. Pick temperature or pick top-p. Do not choreograph both and then wonder why the behavior is unpredictable.
The second mistake is believing high temperature equals creativity. It does not. High temperature buys you randomness, and randomness is not the same thing as good ideas. Past a certain point you are not getting a more imaginative model, you are getting a model that picks worse words more often. There is a real ceiling where interesting tips over into incoherent, and it arrives sooner than people expect. If you want surprising-but-good, a modest bump plus a better prompt beats cranking the dial to the roof.
The third mistake is silent and nasty: tuning these values on one backend and deploying on another. The OpenAI API, vLLM, and llama.cpp do not necessarily apply samplers in the same order, so the same numbers can produce different text on different stacks. Always tune against the exact backend you will actually run.
Sane defaults to start from
For anything where there is a correct answer, set temperature to 0. Code generation, data extraction, classification, factual questions, anything you would run twice and want the same result. Randomness here is pure downside: it introduces inconsistency across identical inputs and buys you nothing. This is the setting most people should be using far more than they do, because most real work has a right answer and they leave it at 0.7 out of habit.
For creative writing, brainstorming, generating varied options, raise it. Somewhere around 0.7 to 1.0 is a reasonable playground, and this is genuinely where a warmer setting earns its place: you want the model to wander a little, and repeating the exact same story every time defeats the point.
The default that ships with most APIs, around 0.7, is a compromise aimed at open-ended chat. It is a fine middle for conversation and a poor fit for either extreme. If you are extracting invoice fields at 0.7, you are inviting the occasional wrong number for literally no benefit.
The whole thing takes five minutes to understand and saves you a lot of confused debugging. When output feels too samey, you know which way to push. When it feels unreliable and you want the same answer twice, you know to pull temperature to the floor. That is the entire skill: not memorizing magic numbers, but knowing what the two knobs move so you can reach for the right one instead of copying 0.7 from a tutorial and hoping.