Temperature, top-p, top-k: how an LLM picks the next word
TL;DR
The model outputs a probability distribution; sampling decides which token to draw. Use low temperature for consistency and the default for creativity, adjust either temperature or top-p, and remember consistent does not mean correct.
Run the same prompt twice and get two different answers: for a creative chatbot that's a feature, for an invoice extraction system it's a bug. The difference comes from sampling: at each step the model doesn't "pick" the next word; it produces a probability distribution over the whole vocabulary, and an algorithm draws a token from it. Temperature, top-p and top-k are the knobs that control how that draw happens.
How does a model generate text?
- The model computes a score (logit) for every token in the vocabulary.
- Softmax turns the scores into probabilities, e.g. "Paris" 62%, "Lyon" 20%, "Nice" 9%…
- The sampling algorithm draws one token, appends it to the sequence, and repeats.
Three knobs
| Parameter | Mechanism | Low | High |
|---|---|---|---|
| temperature | Divides logits before softmax: makes the distribution "sharper" or "flatter" | Almost always picks the most likely token; stable | Less likely tokens get a chance; more varied |
| top_p (nucleus) | Keeps only the smallest set of tokens whose probabilities sum to ≥ p, then draws from it | Only a few "safe" tokens | Nearly the whole vocabulary |
| top_k | Keeps only the k most likely tokens | Very conservative | Little filtering effect |
Nucleus sampling (top-p) was proposed in "The Curious Case of Neural Text Degeneration" (Holtzman et al., 2019). The paper showed that always choosing the most likely token (greedy, beam search) tends to produce repetitive, bland text, while sampling from the full distribution drifts off-topic because of the "long tail" of low-quality tokens. Cutting off that tail is the balance.
With the Claude API
client.messages.create(
model="claude-sonnet-5-5",
max_tokens=512,
temperature=0.2, # extraction, classification
messages=[{"role": "user", "content": "Extract the name, date and total from this invoice…"}],
)- According to the API docs, Claude's
temperatureranges from 0.0 to 1.0, with a default of 1.0. - Anthropic recommends adjusting only one of temperature or top_p, not both; top_k is for advanced use cases.
- The docs also note that even at temperature 0, results are not fully deterministic.
- Some modes (e.g. extended thinking) restrict changing these parameters; check the docs for the model you use.
Suggestions by task type
| Task | Suggestion |
|---|---|
| Classification, extraction, labelling | Low temperature (0–0.3) + structured output (JSON schema/tool) |
| Writing code, technical answers | Low to medium; leave the default if the model is already tuned for it |
| Brainstorming, naming, ad copy | Default 1.0, run several times and pick |
| Consistent grading (LLM-as-judge) | Low, plus a clear rubric and repeated runs to measure variance |
Risks and caveats
- Temperature doesn't cure hallucinations: temperature 0 only makes the model more consistent, including consistently wrong. Correctness needs source data and permission to say "I don't know".
- Don't rely on determinism in tests: write tests that check structure and criteria, not exact string matches.
- Blind tuning: changing temperature without an eval set is guesswork. Measure on 30–50 real samples before settling.
How to apply it
- Dev: Use low temperature for data-processing pipelines; validate output with a schema rather than trusting repeatability.
- Tech Lead: Keep sampling parameters in versioned config alongside the prompt, so results can be reproduced when investigating bugs.
- PM: For creative features, embrace randomness: generate 3 options for users to choose from instead of 1.
Sources & references
- The Curious Case of Neural Text Degeneration — Holtzman et al. (arXiv)
- Messages API reference — Claude Docs