Temperature is the dial between “always the same answer” and “a different answer every time”.
How it works
A model produces a score (a logit) for every token it could write next. A softmax turns the scores into probabilities. Temperature divides each score first. Hinton and colleagues write the softmax with this T and note that “using a higher value for T produces a softer probability distribution over classes.”
- T below 1 sharpens the distribution. The top token wins more often.
- T = 1 leaves the model’s probabilities as trained.
- T above 1 flattens them. Rare tokens get picked more.
- T near 0 approaches greedy decoding: always take the most likely token.
OpenAI’s API accepts values “between 0 and 2” with a default of 1. Its reference says 0.8 makes output “more random” and 0.2 “more focused and deterministic”. It also recommends “altering this or top_p but not both”, because top-p is a second way to cut randomness.
Open models often ship their own defaults. Hugging Face Transformers reads temperature from a model’s generation_config.json and falls back to 1.0 if it is missing. A serving stack may use either value.
Why it matters
- Repeatability. Extraction, routing and structured output need stable answers. Lower temperature cuts variation between runs.
- Quality. Too high and the model drifts into odd word choices or invented details, which feeds hallucination. Too low and long text can loop: Hugging Face notes that greedy decoding “begins to repeat itself” on longer sequences.
- Evaluation. When you compare models, fix the temperature. Otherwise you are measuring sampling noise. Watching logprobs shows how much room the model had.
Temperature does not change price or speed. A token costs the same at any setting.
On BroadNet Inference
BroadNet’s API reference does not document temperature, its allowed range or its default for any model. The request examples do not set it. Because the API is OpenAI-compatible, SDKs will send the field, but whether each model honours it is not stated. Set it explicitly and run the same prompt several times to see the spread. The OpenAI Python SDK guide shows the client setup.
Common mistakes
- Using temperature 0 and assuming identical output forever. Store answers you need to reproduce.
- Raising temperature to fix a wrong answer. A better prompt or more context fixes accuracy; temperature only changes variety.