BroadRouter

Cached input tokens

Updated

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

Also called: prompt caching, cache hits, cached tokens, context caching, cache read tokens

Most of what an agent sends is the same on every turn. Prompt caching lets the provider skip recomputing that repeated part, and bill it for less.

How it works

When a model reads a prompt, it computes internal state for every token. A provider can keep that state for a short time. If the next request starts with the same tokens, the provider reads the stored state instead of recomputing it. Those tokens are cached input tokens.

Caches match from the start of the prompt. OpenAI’s guide says reuse “requires the entire rendered prefix to match.” One changed character early in the prompt makes everything after it new input.

Providers differ in the details:

  • OpenAI enables caching by default on supported models and reports hits in the usage block. It documents a 1,024-token minimum prefix on its newer models.
  • Anthropic bills cache reads at 0.1 times the base input price for most models, bills cache writes at a premium, and keeps entries for 5 minutes by default.

In the Chat Completions format, hits appear as usage.prompt_tokens_details.cached_tokens. Cache rate is that number divided by prompt_tokens.

Why it matters

For long, repeated prompts, the cache rate decides the bill. An agent that resends a 200,000-token repository on every turn pays the full input price once and the cached price after that, as long as the prefix stays stable. Caching can also shorten time to first token on long prompts, because less work is done before generation starts.

To raise the rate, put stable content first (system prompt, tool definitions, documents) and changing content last. Keep timestamps and request IDs out of the top of the prompt, and keep tool lists in a fixed order. What share of providers’ published prices include a cached-input price, and the typical discount, is in the LLM API price index.

On BroadNet Inference

BroadNet publishes cached input prices for three chat models, per 1M tokens:

Model New input Cached input
DeepSeek V4.1 Flash $0.175 $0.0055
GLM 5.3 $0.78 $0.173
Qwen3.8 27B $0.175 $0.0435

Cached tokens are billed at the lower price automatically, and prompt_tokens_details.cached_tokens shows them when the model reports them. In a stream, request usage with stream_options as shown in the streaming guide.

On a flat-rate plan there is no token meter, so the cache rate does not change the bill. The full walkthrough, with code to log your rate, is in cached input tokens explained.

Questions

Do cached tokens still count toward the context window?

Yes. Caching changes what you pay for those tokens, not whether they count. Anthropic's docs state this directly.

How long does a cache last?

It depends on the provider. Anthropic's default lifetime is 5 minutes, refreshed each time the cache is used, with an optional 1-hour lifetime. Other providers do not always publish a fixed time, so measure your hit rate.

Does caching change the answer?

No. The cache stores work done on the prompt, not the reply. The model still generates a fresh answer.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity