BroadRouter
Get started

Temperature

Updated

Temperature is a sampling parameter that controls how random a language model's output is. The model divides its token scores by the temperature before turning them into probabilities: low values make the likeliest token dominate, high values flatten the distribution so less likely tokens are chosen more often.

Also called: sampling temperature, LLM temperature, temperature parameter

Temperature is the dial between “always the same answer” and “a different answer every time”.

How it works

A model produces a score (a logit) for every token it could write next. A softmax turns the scores into probabilities. Temperature divides each score first. Hinton and colleagues write the softmax with this T and note that “using a higher value for T produces a softer probability distribution over classes.”

  • T below 1 sharpens the distribution. The top token wins more often.
  • T = 1 leaves the model’s probabilities as trained.
  • T above 1 flattens them. Rare tokens get picked more.
  • T near 0 approaches greedy decoding: always take the most likely token.

OpenAI’s API accepts values “between 0 and 2” with a default of 1. Its reference says 0.8 makes output “more random” and 0.2 “more focused and deterministic”. It also recommends “altering this or top_p but not both”, because top-p is a second way to cut randomness.

Open models often ship their own defaults. Hugging Face Transformers reads temperature from a model’s generation_config.json and falls back to 1.0 if it is missing. A serving stack may use either value.

Why it matters

  • Repeatability. Extraction, routing and structured output need stable answers. Lower temperature cuts variation between runs.
  • Quality. Too high and the model drifts into odd word choices or invented details, which feeds hallucination. Too low and long text can loop: Hugging Face notes that greedy decoding “begins to repeat itself” on longer sequences.
  • Evaluation. When you compare models, fix the temperature. Otherwise you are measuring sampling noise. Watching logprobs shows how much room the model had.

Temperature does not change price or speed. A token costs the same at any setting.

On BroadNet Inference

BroadNet’s API reference does not document temperature, its allowed range or its default for any model. The request examples do not set it. Because the API is OpenAI-compatible, SDKs will send the field, but whether each model honours it is not stated. Set it explicitly and run the same prompt several times to see the spread. The OpenAI Python SDK guide shows the client setup.

Common mistakes

  • Using temperature 0 and assuming identical output forever. Store answers you need to reproduce.
  • Raising temperature to fix a wrong answer. A better prompt or more context fixes accuracy; temperature only changes variety.

Questions

What temperature should I use?

Low values such as 0 to 0.3 for extraction, classification and code, where you want the same answer each time. Higher values such as 0.7 to 1 for brainstorming and varied wording. Test both on your own prompts.

Does temperature 0 make the output deterministic?

Not guaranteed. It pushes the model toward the top token every time, but providers do not generally promise identical output across runs. Store any answer you need to reproduce.

What is BroadNet's default temperature?

BroadNet's docs do not state a default temperature for any model. Set it explicitly in each request if your output depends on it.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity