---
title: "Cache LLM API responses to cut cost (Python, SQLite)"
description: "A tested exact-match cache for LLM API responses in Python: what goes in the key, what never to cache, TTLs, privacy and how to measure the hit rate before building."
url: https://broadrouter.com/blog/cache-llm-responses-cut-cost/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Cache LLM API responses to cut cost: an exact-match cache in Python

By the BroadNet Inference team · 8 min read · Published 2026-11-15

Hash the full request (model, messages, temperature, max\_tokens, response\_format, tools, reasoning settings and your prompt version) into a key, and store the answer under it with an expiry. Serve a stored answer only for deterministic, side-effect-free requests. Measure how often identical requests repeat in your logs first: that share is the most an exact cache can save.

## Key takeaways

-   An exact cache saves the whole call, output and reasoning tokens included, but only for byte-identical requests. Paraphrased questions always miss.
-   Put every parameter that changes the answer in the key, plus a prompt version. Leaving one out serves wrong answers; adding noise such as a timestamp makes every request miss.
-   Do not cache sampled replies (temperature above 0), streamed replies, tool calls, or answers cut off at max\_tokens.
-   Measure first: hash each request in your usage logs and count repeats. Published savings figures say nothing about your traffic.
-   A cache stores prompts and answers. If your provider keeps no inference content, the cache is now where that content lives, so give it a TTL and access rules.

**On this page**

1.  Is response caching the same as prompt caching?
2.  Will a cache actually save you money?
3.  What goes in the cache key?
4.  The cache: SQLite, TTL and a skip policy
5.  What should never be served from a cache?
6.  When should cached answers expire?
7.  What about semantic caching?
8.  What does a cache mean for privacy?
9.  How does caching fit with other cost controls?
10.  Where BroadNet Inference fits

Every identical LLM request you send twice is paid for twice. A response cache stores the answer the first time and returns it the next time, with no tokens billed and no wait. The idea is simple. The hard parts are deciding what makes two requests “identical”, which requests must never be served from storage, and whether your traffic repeats enough to matter.

This post builds a small exact-match cache in Python on SQLite, tested against a mock OpenAI-compatible server, and shows how to measure the hit rate from your logs before you write any of it.

## Is response caching the same as prompt caching?

No. They save different things, and you can use both.

|  | Prompt caching (provider) | Response caching (you) |
| --- | --- | --- |
| What is reused | The repeated start of the prompt | The whole answer |
| Who runs it | The API provider | Your code |
| Model still runs? | Yes, writes a fresh answer | No |
| What it saves | Part of the input cost | Input, output and reasoning tokens, plus latency |
| When it hits | Shared prefix, different questions | Exactly the same request |

[Cached input tokens](https://broadrouter.com/blog/cached-input-tokens-explained/) explains the provider side, and [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/) shows how to lay out a prompt so the prefix is reused. The [prompt caching calculator](https://broadrouter.com/tools/prompt-caching-calculator/) prices it. A response cache sits in front of all that: on a hit, no request leaves your server.

## Will a cache actually save you money?

Measure before you build. The cost-saving percentages in many caching tutorials come from someone else’s traffic. Your number depends on one thing: how often the same request arrives again within your expiry window.

If you already log each call, as in [LLM API usage logging](https://broadrouter.com/blog/llm-api-usage-logging/), add one field: `request_hash`, the cache key described below. It stores no prompt text, so it fits a log that keeps content out. Then count repeats:

```python
import json
from collections import Counter

def potential_hit_rate(log_path: str) -> float:
    """Share of calls whose request_hash was already seen: an upper bound for an exact cache."""
    hashes = [json.loads(line)["request_hash"] for line in open(log_path)]
    seen, repeats = set(), 0
    for h in hashes:
        repeats += h in seen
        seen.add(h)
    print("top repeated requests:", Counter(hashes).most_common(3))
    return repeats / len(hashes) if hashes else 0.0

print(f"potential hit rate: {potential_hit_rate('calls.jsonl'):.1%}")
```

On a synthetic log of 20 calls, one request five times, another three times and twelve unique, it printed a potential hit rate of 30.0%. That is an upper bound: it ignores expiry, and it assumes every repeat is safe to serve from storage.

Then turn the rate into money:

```text
monthly saving = hit rate × requests per month
                 × (input tokens × input price + output tokens × output price) ÷ 1,000,000
```

Output tokens include reasoning. BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering and that reasoning counts toward `max_tokens`, so a hit saves tokens you never see in the reply. Take `completion_tokens` from the logged `usage`, not the length of the answer.

A worked example in tokens: 100,000 requests a month at 1,200 input and 150 output tokens each, with a 30% hit rate, avoid 36 million input tokens and 4.5 million output tokens. On [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) those are billed at $0.175 and $0.52 per 1M respectively; plug your own figures into the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/). If most of that input was already a provider cache hit, the input saving is at the lower cached rate ($0.0055 per 1M on the same model), so the output side is often what matters.

## What goes in the cache key?

Everything that can change the answer, and nothing else.

-   **Include:** `model`, the full `messages` list (system prompt included), `temperature`, `top_p`, `max_tokens`, `seed`, `response_format`, `tools`, `tool_choice` and `reasoning_effort`.
-   **Add a prompt version** you control, such as `faq-v3`. Bump it when you change prompt templates, retrieval data or post-processing that the request body does not show.
-   **Leave out** anything that varies per call without changing the answer: request IDs, timestamps, trace headers, a `user` field you only send for abuse tracking.

Serialise the key material the same way every time. Python’s `json.dumps` with `sort_keys=True` and `separators=(",", ":")` gives one canonical string regardless of dictionary order; `ensure_ascii=False` keeps Arabic and other non-Latin text as UTF-8 instead of escapes. Hash it with SHA-256.

A missing field is the dangerous mistake. If `max_tokens` is not in the key, a reply cut short under a small limit will be served to a request that allowed a long one.

## The cache: SQLite, TTL and a skip policy

```python
import hashlib, json, sqlite3, time

# Every field that changes the answer belongs in the key.
KEY_FIELDS = ("model", "messages", "temperature", "top_p", "max_tokens", "seed",
              "response_format", "tools", "tool_choice", "reasoning_effort")

def cache_key(params: dict, prompt_version: str) -> str:
    material = {k: params[k] for k in KEY_FIELDS if k in params}
    material["prompt_version"] = prompt_version
    canon = json.dumps(material, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(canon.encode("utf-8")).hexdigest()

class ResponseCache:
    def __init__(self, path="llm_cache.sqlite", ttl_s=7 * 86400):
        self.db, self.ttl = sqlite3.connect(path), ttl_s
        self.db.execute("CREATE TABLE IF NOT EXISTS cache (k TEXT PRIMARY KEY, created REAL,"
                        " body TEXT, prompt_tokens INT, completion_tokens INT)")
        self.hits = self.misses = self.skips = self.saved_in = self.saved_out = 0

    def create(self, client, prompt_version: str, **params):
        if params.get("stream") or params.get("temperature", 1) > 0:
            self.skips += 1                       # not cacheable: call the model directly
            return client.chat.completions.create(**params)
        k = cache_key(params, prompt_version)
        row = self.db.execute("SELECT body, created, prompt_tokens, completion_tokens"
                              " FROM cache WHERE k = ?", (k,)).fetchone()
        if row and time.time() - row[1] < self.ttl:
            self.hits += 1; self.saved_in += row[2]; self.saved_out += row[3]
            return json.loads(row[0])             # a plain dict, not an SDK object
        self.misses += 1
        r = client.chat.completions.create(**params)
        choice = r.choices[0]
        if choice.finish_reason == "stop" and not choice.message.tool_calls:
            self.db.execute("INSERT OR REPLACE INTO cache VALUES (?, ?, ?, ?, ?)",
                            (k, time.time(), r.model_dump_json(),
                             r.usage.prompt_tokens, r.usage.completion_tokens))
            self.db.commit()
        return r.model_dump()
```

And a run against it:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
cache = ResponseCache("demo.sqlite", ttl_s=86400)
SYSTEM = {"role": "system", "content": "Answer questions about our refund policy in one sentence."}
questions = ["How long do refunds take?", "Can I refund a gift card?",
             "How long do refunds take?", "How long do refunds take?", "Can I refund a gift card?"]
for q in questions:
    r = cache.create(client, "faq-v3", model="broadnet/deepseek-v4.1-flash", temperature=0,
                     max_tokens=1200, messages=[SYSTEM, {"role": "user", "content": q}])
    print(r["choices"][0]["message"]["content"])
# a personalised, sampled request is never cached
cache.create(client, "faq-v3", model="broadnet/deepseek-v4.1-flash", temperature=0.7,
             messages=[SYSTEM, {"role": "user", "content": "Write me a friendly reminder"}])
# a new prompt version is a new key
cache.create(client, "faq-v4", model="broadnet/deepseek-v4.1-flash", temperature=0,
             max_tokens=1200, messages=[SYSTEM, {"role": "user", "content": "How long do refunds take?"}])
print(f"hits={cache.hits} misses={cache.misses} skipped={cache.skips} "
      f"saved_input={cache.saved_in} saved_output={cache.saved_out}")
```

We ran this with the OpenAI Python SDK 3.51.0 against a local mock server that counted upstream calls and reported 1,200 prompt and 150 completion tokens per reply (no live API call). Seven requests made four upstream calls. The script printed `hits=3 misses=3 skipped=1 saved_input=3600 saved_output=450`. Run again on the same database file, all six cacheable requests were hits (`hits=6 misses=0`): SQLite keeps the cache across restarts. Keys built from the same fields in a different order matched.

Notes on the code:

-   **`max_tokens=1200`** leaves room for reasoning before the answer. With a small limit a reasoning model can return an empty reply ([why](https://broadrouter.com/guides/reasoning-tokens-empty-response/)). Those replies are not stored, because `finish_reason` is `length`, not `stop`.
-   **Hits return a dict.** Misses also return `model_dump()`, so callers see one shape. If your code expects SDK objects, wrap both.
-   **A missing `temperature` counts as sampled.** The server then applies its own default, which may be above 0, so the wrapper skips caching unless you set 0 explicitly.
-   **SQLite suits one process or a few.** For several servers, put the same key and TTL logic in Redis or your database.

## What should never be served from a cache?

-   **Sampled replies (temperature above 0).** You chose variety. Caching removes it, and a user who presses “regenerate” gets the same text.
-   **Personalised answers** unless the user ID is in the key. Two users sending “summarise my account” have different accounts. Better: cache only requests whose messages contain no per-user data.
-   **Tool calls.** A [tool call](https://broadrouter.com/glossary/tool-calling/) is an instruction to act. Replaying “refund order 1182” from storage can repeat a side effect, and the right next step depends on current state. The wrapper refuses to store any reply with `tool_calls`.
-   **Streamed replies.** The wrapper bypasses the cache for `stream=True`. Caching a stream means buffering it, checking it finished, and replaying it, which is rarely worth it for chat UIs.
-   **Truncated or failed replies.** Store only `finish_reason == "stop"`. A cut-off answer or an error cached for a week is a bug that looks like a model problem.
-   **Answers that depend on time.** “What is today’s exchange rate?” needs a short TTL or no cache.

What does cache well: classification of a fixed set of inputs, extraction from documents you reprocess, FAQ answers from a fixed prompt, test suites and evals you rerun, and development loops where you send the same request again and again.

## When should cached answers expire?

Use three kinds of invalidation:

1.  **TTL.** A time limit per entry: days for stable FAQ answers, minutes for anything that reads live data.
2.  **Version in the key.** Change the prompt version when the system prompt, retrieval index or output schema changes. Old entries stop matching and age out.
3.  **Model in the key.** It is already there, so moving from one model to another starts a fresh cache, which is what you want while you [evaluate a switch](https://broadrouter.com/blog/evaluate-llm-before-switching/).

Delete expired rows on a schedule (`DELETE FROM cache WHERE created < ?`). Otherwise the file only grows.

There is a side benefit. Serving engines do not promise identical output for identical input: vLLM’s documentation says it “does not guarantee the reproducibility of the results by default”. A cache gives every user the same answer to the same question for the life of the entry.

## What about semantic caching?

A semantic cache embeds each question and returns a stored answer when a new question is close enough, so “how long is a refund?” can hit “how long do refunds take?”. GPTCache is the best-known open-source example. Its README says it uses a vector store for similarity search and a similarity evaluator to decide a hit, warns that “you may encounter false positives during cache hits and false negatives during cache misses”, and notes that disabling embeddings turns it into a keyword-matching cache.

Two practical points. First, it needs an [embeddings](https://broadrouter.com/glossary/embeddings/) model and a vector store. BroadNet Inference does not offer embeddings, so you would use another provider or a local model for that layer; the [embedding cost calculator](https://broadrouter.com/tools/embedding-cost-calculator/) prices the options. Second, a false hit returns a confident answer to a different question. Use a strict similarity threshold, test it on real near-miss pairs, and keep semantic caching for low-stakes answers. Put the exact cache in front either way: it is free to check and never wrong about what was asked.

## What does a cache mean for privacy?

A cache is a store of prompts and answers. That changes where your data lives.

BroadNet’s [inference privacy statement](https://www.broadnet.ai/inference-privacy), checked 2026-11-15, says its gateway keeps no prompts, messages, responses or transcripts, while content-free traffic and usage records (token counts, cost, timings) are kept. Inference content is not retained. Necessary account, billing and security metadata is retained as described in that statement. Once you add a response cache, your system holds the content the provider does not. Treat it like any other store of user data:

-   Cache only requests without personal data where you can, which also raises the hit rate.
-   Set a TTL short enough for your retention policy, and delete on schedule.
-   Encrypt the file or table at rest and limit who can read it.
-   Include the cache in data-deletion requests if it can hold user text.

[Sending sensitive data to an LLM API](https://broadrouter.com/blog/llm-api-sensitive-data/) covers the rest of that checklist.

## How does caching fit with other cost controls?

Caching removes repeated calls. It does nothing for unique ones, so it sits next to the other controls rather than replacing them: [shorter prompts](https://broadrouter.com/blog/reduce-prompt-tokens/), provider prompt caching, a `max_tokens` cap per request, and budgets and loop limits from [LLM API spend control](https://broadrouter.com/blog/llm-api-spend-control/). For large one-off jobs, deduplicate inputs before you send them; that is the same idea as a cache, done once. [Processing a large job without a Batch API](https://broadrouter.com/blog/llm-batch-processing-without-batch-api/) shows the job side.

On a flat-rate plan the maths changes. BroadNet’s DeepSeek sessions have no token meter, so a cache hit saves no money there. It still saves time: a session runs one request at a time, so every repeat you skip frees the session for new work.

## Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference exposes DeepSeek V4.1 Flash, GLM 5.3, Qwen3.8 27B and Gemma 4 VL 26B behind an OpenAI-compatible `/v1/chat/completions` endpoint, so the wrapper above works with the [OpenAI Python SDK](https://broadrouter.com/guides/openai-python-sdk/) by changing the base URL. It has no response-caching feature of its own; the cache in this post is yours. Provider-side cache hits are reported in `usage.prompt_tokens_details.cached_tokens` and billed at the cached input price, per its [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-cache-llm-responses-cut-cost&utm_content=body). Prices per model are on the [pricing page](https://inference.broadnet.ai/pricing?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-cache-llm-responses-cut-cost&utm_content=body).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

[Vector database](https://broadrouter.com/glossary/vector-database/)

A vector database is a data store that holds embedding vectors, often with their source text and metadata, and answers nearest-neighbour queries: given a query vector, it returns the stored vectors most similar to it, usually by cosine distance, Euclidean distance or inner product.

[Base URL](https://broadrouter.com/glossary/base-url/)

A base URL is the fixed root address of an API, such as https://api.example.com/v1, that a client puts in front of every endpoint path, such as /chat/completions. In OpenAI-compatible SDKs, changing the base URL, together with the API key and model ID, points the same code at a different provider.

## Questions

### What is the difference between prompt caching and response caching?

Prompt caching is done by the provider: a repeated prompt prefix is billed at a lower cached-input price, and the model still writes a new answer. Response caching is done by you: you store the whole answer and return it for an identical request without calling the model at all.

### Should I cache LLM responses at temperature 0?

Yes, if the request has no side effects and the same answer is acceptable for everyone who sends it. Temperature 0 does not guarantee identical output from a serving engine anyway, so a cache also makes the answer consistent. Skip caching when temperature is above 0 or omitted.

### How much does LLM response caching save?

The share of requests that are exact repeats within your expiry window, times the cost of those requests. Count repeated request hashes in your own logs to get that share. It can be near zero for open-ended chat and high for fixed FAQ or classification inputs.

### Is semantic caching better than exact-match caching?

It hits more often, because similar questions match, but it can return an answer to a different question. It also needs an embeddings model and a vector store. Start with an exact cache, and add a semantic layer only where a wrong hit is cheap.

### Can I cache streamed responses?

Not with the simple wrapper here. You would have to collect the full stream, check it finished normally, store it, and replay it as a stream later. Most teams cache only non-streamed calls such as classification, extraction and FAQ answers.

## Keep reading

-   [Cached input tokens: how to cut an LLM API bill](https://broadrouter.com/blog/cached-input-tokens-explained/)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/)
-   [LLM API spend control: budgets, key limits and loop protection](https://broadrouter.com/blog/llm-api-spend-control/)
-   [Reduce prompt tokens: measured cuts for system prompts, tools, data](https://broadrouter.com/blog/reduce-prompt-tokens/)
-   [LLM API usage logging: cost per user and feature, PII redacted](https://broadrouter.com/blog/llm-api-usage-logging/)
-   [LLM batch processing without a Batch API: a resumable runner](https://broadrouter.com/blog/llm-batch-processing-without-batch-api/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-cache-llm-responses-cut-cost&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference: reasoning counts toward max\_tokens, usage fields for reasoning and cached tokens, pay-as-you-go limits (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-cache-llm-responses-cut-cost&utm_content=sources)
-   [BroadNet inference privacy statement: no prompts, messages, responses or transcripts kept; content-free traffic and usage records retained (checked 2026-11-15)](https://www.broadnet.ai/inference-privacy)
-   [GPTCache on GitHub: semantic cache with embeddings and a similarity evaluator, false positives and false negatives, keyword-matching mode without embeddings (checked 2026-11-15)](https://github.com/zilliztech/GPTCache)
-   [vLLM documentation, Reproducibility: "vLLM does not guarantee the reproducibility of the results by default" (checked 2026-11-15)](https://docs.vllm.ai/en/latest/usage/reproducibility.html)
-   [Python documentation, json module: sort\_keys, separators and ensure\_ascii in json.dumps (checked 2026-11-15)](https://docs.python.org/3/library/json.html)
