BroadRouter

How to structure prompts so the prefix caches

By the BroadNet Inference team · 4 min read · Published · Updated

Order every request from most stable to least stable: system prompt, tool definitions, reference documents, conversation history, then the new message. Make each layer byte-identical between calls: no timestamps or IDs at the top, tools sorted, JSON serialised deterministically, and history only ever appended. Then measure cached_tokens to confirm the prefix is actually reused.

Cached input tokens are cheap, and on BroadNet Inference DeepSeek V4.1 Flash bills them at $0.0055 per 1M instead of $0.175. The cached tokens explainer covers the price and how to read the usage field. This post is about the design work: how to lay out a prompt so that the cache actually hits, and how to prove it does.

Why does prompt order decide the cache hit rate?

Every major provider caches by prefix. OpenAI’s guide says reuse requires the entire rendered prefix to match. DeepSeek’s docs describe the same rule: a request hits only if it fully matches a stored prefix, and one that diverges partway through does not. Anthropic builds its cache in the order tools, system, messages.

Under the hood the server reuses the KV cache it built for that prefix. So the cache is a straight line from the first token. The first byte that differs from the last call ends the reusable part. A one-line timestamp at the top of a 50,000-token prompt costs you all 50,000 tokens at full price.

What is the right layer order?

Layer Changes Example Rule
1. System prompt Per deploy Role, rules, output format No dates, IDs, user names
2. Tool definitions Per deploy Function schemas Sorted, deterministic JSON
3. Reference material Per session or task Repo files, contracts, product docs Fixed order, stable labels
4. Conversation history Per turn, append-only Earlier user and assistant turns Never edit; only add
5. New message Every call The question, plus the current time if needed Volatile data goes here

Layer 5 is where anything volatile belongs: the current date, the user’s locale, a request ID, retrieved snippets that differ per question.

What breaks the prefix without you noticing?

  1. Dictionary order. Frameworks often build the tools array from a dict or registry. If plugins load in a different order, the bytes change.
  2. JSON formatting. json.dumps(x) and json.dumps(x, indent=2) are different prefixes. So are two serialisers with different key orders.
  3. Template whitespace. A trailing space or a \r\n from one code path and \n from another.
  4. Dynamic tool lists. Removing tools a user cannot use changes the prefix. OpenAI’s guide recommends keeping definitions stable and restricting use with tool_choice or an allowed-tools setting.
  5. Summarising history in place. Rewriting old turns into a summary resets everything after the system prompt. Summarise rarely and in large steps.
  6. Retrieval before instructions. RAG snippets placed between the system prompt and the tools differ per query and break the tools layer too.

How do you build a cache-friendly request in code?

Make each layer a function of stable inputs only, and serialise deterministically:

import json, os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

SYSTEM = open("prompts/system.md").read()  # versioned file, no templating

def stable_tools(registry: dict) -> list:
    # Sort by name, sort keys inside each schema.
    return [json.loads(json.dumps(registry[name], sort_keys=True)) for name in sorted(registry)]

def docs_block(paths: list[str]) -> str:
    parts = [f"=== {p} ===\n{open(p).read()}" for p in sorted(paths)]
    return "\n\n".join(parts)

def build_messages(history: list, user_text: str, now_iso: str, doc_paths: list[str]):
    return (
        [{"role": "system", "content": SYSTEM},
         {"role": "user", "content": docs_block(doc_paths)}]
        + history                                   # append-only
        + [{"role": "user", "content": f"(now: {now_iso})\n{user_text}"}]
    )

resp = client.chat.completions.create(
    model="broadnet/deepseek-v4.1-flash",
    messages=build_messages(history, question, now_iso, files),
    tools=stable_tools(TOOLS),
)
history += [{"role": "user", "content": question}, resp.choices[0].message.model_dump(exclude_none=True)]

Two details. The current time goes in the last message, not the system prompt. And the history stores the assistant message exactly as returned, so the next call resends identical bytes.

How do you prove the prefix is cached?

Run a two-call test in CI or a notebook:

  1. Build a request with your real system prompt, tools and documents.
  2. Send it. Note usage.prompt_tokens.
  3. Change only the final user message and send again.
  4. Read usage.prompt_tokens_details.cached_tokens on the second call. It should cover nearly all of the shared prefix.
  5. If it is zero or small, diff the serialised request bodies byte by byte. The first difference is your cache breaker.

Caching is best-effort on most providers, and DeepSeek’s docs say so directly, so a single miss is not proof of a bug. A consistent miss is. Track the ratio of cached to prompt tokens per request in your usage logs, and use the prompt caching calculator to see what a higher hit rate saves.

How does this change for agents and long context?

Agents benefit most because they resend the same large prefix dozens of times per task. Two habits matter:

  • One conversation per task. Starting a fresh conversation for each sub-step throws the prefix away.
  • Read files once, near the top. If the agent re-reads a file, append the new copy rather than replacing the old one, unless the file changed.

With a 1M-token context window, layer 3 can hold a whole repository. That is affordable on a per-token key only when it caches, which is why the layer order matters more as prompts grow. Cached prefixes also tend to reduce time to first token; see TTFT vs throughput.

Where BroadNet Inference fits

On BroadNet Inference the usage block reports prompt_tokens_details.cached_tokens when the model reports it, and cached input on DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B is billed at the lower rate listed on the pricing page. BroadNet does not offer explicit cache breakpoints or a cache key parameter in its docs, so the layout above is the whole technique. On an unlimited plan the cache does not change the bill, because sessions have no token meter, but the same structure still helps latency.

Questions

Why is my prompt cache hit rate low?

Something near the start of the prompt changes on every call. The usual causes are a timestamp or request ID in the system prompt, tools serialised in a different order, retrieved documents placed before the instructions, or history that gets summarised and rewritten.

Should the user question go before or after long documents?

After. Put the documents first and the question last. The documents then form a reusable prefix, and many models also answer better when the question follows the material.

Do tool definitions break prompt caching?

Only if they change. Tools are part of the prefix, so keep the same set in the same order with the same JSON. To limit tools for one call, use tool_choice rather than editing the list.

Does few-shot example order matter for caching?

Yes. Shuffled examples produce a different prefix each time. Fix the order, or place examples that change per request after the stable part.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity