---
title: "Context management for long-running LLM agents | BroadRouter"
description: "How to keep long agent sessions accurate and affordable with a 1M-token window. Append-only history, cache-friendly prompts, when to compact, and what to drop first."
url: https://broadrouter.com/blog/long-running-agent-context-management/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Context management for long-running agents with a 1M window

By the BroadNet Inference team · 6 min read · Published 2026-11-14 · Updated 2026-11-16

Treat the context as an append-only log: a fixed system prompt and tool list first, then history that only grows, so each turn reuses the cached prefix. With a 1,048,576-token window, as on DeepSeek V4.1 Flash, compact rarely and in large steps, trim bulky tool output first, and keep durable facts in a short notes file the agent rereads.

## Key takeaways

-   A big window postpones context problems; it does not remove them. Plan for what happens near the limit.
-   Append-only history keeps the prompt prefix stable, so cached input stays cheap.
-   Every rewrite of earlier history breaks the cache from that point on.
-   Trim bulky tool output (logs, file dumps) before trimming reasoning and decisions.
-   Compact rarely and in large steps, and keep a short notes file of durable facts.

**On this page**

1.  What does a 1M-token window buy a long-running agent?
2.  Rule 1: why should history be append-only?
3.  Rule 2: how do you measure the cache rate?
4.  Rule 3: what should you cut first when context grows?
5.  Rule 4: when should an agent compact its history?
6.  Rule 5: what belongs in a notes file?
7.  How do reasoning tokens affect the budget?
8.  Does this matter on a flat plan?
9.  Context management checklist

A coding agent that works for an afternoon builds up a lot of history: files read, commands run, test output, plans, corrections. Every turn sends all of it back to the model. How you manage that growing context decides three things at once: what each turn costs, how fast it answers, and whether the agent still remembers what it was doing.

A 1M-token window changes the trade-offs. It does not make them go away.

## What does a 1M-token window buy a long-running agent?

On BroadNet Inference, [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) and [GLM 5.3](https://broadrouter.com/models/glm-5.3/) accept up to 1,048,576 tokens per request, prompt and answer together. [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) accepts 262,144. With smaller windows, an agent has to summarise after an hour or two of work. With a million tokens, it can usually keep a full session’s raw history and only compact near the end of a very long run.

That is a real advantage: summaries are lossy, and every summary is a chance to forget the one detail that mattered. But three costs still grow with the prompt:

-   **Money**, on a per-token key, because every turn resends the whole prompt.
-   **Latency**, because the server must process every input token before the first output token.
-   **Attention.** Quality can drop as inputs grow, an effect called [context rot](https://broadrouter.com/glossary/context-rot/). The paper [“Lost in the Middle”](https://arxiv.org/abs/2307.03172) found that models tend to use information at the start and end of a long input more reliably than information in the middle. Newer models do better, but the habit of keeping important facts near the edges still pays.

## Rule 1: why should history be append-only?

Prompt caching matches from the start of the prompt. If the first 300,000 tokens of this turn are identical to the last turn, they can be served from cache; the first changed token ends the match. BroadNet’s API reports this in `usage.prompt_tokens_details.cached_tokens` when the model reports it, and bills cached input at a lower rate: $0.0055 per 1M tokens on DeepSeek V4.1 Flash against $0.175 for new input.

So structure every request in the same order and only ever add to the end:

```text
[ system prompt ]        never changes during a session
[ tool definitions ]     fixed order, never changes
[ project notes ]        changes rarely, and only by appending
[ turn 1 ... turn N ]    grows at the end
[ new message ]
```

The usual prefix breakers are a timestamp in the system prompt, tool definitions in unstable order, and re-rendering old tool results. For agents, add one more: “sliding window” trimming that drops the oldest turn every time, which shifts everything after it. Each of those turns a cached prompt into a new one; [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/) lists the rest, with a test that proves the prefix is cached. On a long session the difference is large: most of the prompt can be cached input instead of new input. [How cached input is billed](https://broadrouter.com/blog/cached-input-tokens-explained/).

## Rule 2: how do you measure the cache rate?

Log it per request. With the OpenAI SDK, three lines after the call do it:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

resp = client.chat.completions.create(model="broadnet/deepseek-v4.1-flash", messages=history, max_tokens=8000)
u = resp.usage
cached = (u.prompt_tokens_details.cached_tokens if u.prompt_tokens_details else 0) or 0
print(f"turn prompt={u.prompt_tokens:,} cached={cached:,} ({cached / max(1, u.prompt_tokens):.0%})")
```

A healthy append-only agent shows a high cached share on every turn after the first. A sudden drop to near zero means something early in the prompt changed. Find it and pin it. The [prompt caching calculator](https://broadrouter.com/tools/prompt-caching-calculator/) turns a cache rate into money per session.

## Rule 3: what should you cut first when context grows?

When context does need to shrink, what you drop matters. In most agent transcripts the bulk is tool output: a 4,000-line test log, a whole file read to find one function, a directory listing. The valuable part is small: the plan, the decisions and why, the errors that were fixed.

Order of what to cut, first to last:

1.  **Stale tool output.** Replace the body of old tool results with a one-line stub: “ran pytest: 3 failures in test\_api.py (fixed in turn 14)”.
2.  **Superseded file contents.** If the agent read `api.py`, edited it, then read it again, the first read is dead weight.
3.  **Exploration that went nowhere.** Searches and reads that did not lead to a change.
4.  **Reasoning and plans.** Keep these longest. They explain why the code looks the way it does.

Truncating tool output at the moment it is produced is even better. Cap each tool result (for example, the last 200 lines of a log plus a line count) and let the agent ask for more if it needs it. That keeps history append-only and small.

## Rule 4: when should an agent compact its history?

Compaction (summarising older history into a shorter block) is the heavy tool. It rewrites the prefix, so the turn after a compaction pays full input price for everything after the system prompt. Compacting a little every few turns means paying that price over and over.

Better:

-   **Pick a high threshold.** With a 1M window, compacting at, say, 60–70% full leaves room for the summary turn and a long tail of new work.
-   **Keep the recent tail verbatim.** Summarise the old part and keep the last stretch of turns intact. opencode’s [compaction docs](https://opencode.ai/docs/compaction/) describe this: it summarises the conversation and keeps the most recent part, about 15,000 tokens by default.
-   **Summarise with the full prompt.** Sending the whole conversation to the summary request lets it reuse the cache from the previous turn.
-   **Compact once, then append again.** After a compaction the new prefix is stable, and caching resumes from there.

## Rule 5: what belongs in a notes file?

Some facts must survive any number of compactions: the goal, the constraints, the decisions taken, the commands that build and test the project. Keep them in a short notes file (a `NOTES.md` or a task file) that the agent appends to and rereads. It sits near the top of the prompt, it changes rarely, and it is the first thing a fresh session reads if the run restarts.

This is also the cheapest form of memory: a few thousand tokens that save the agent from rediscovering the same facts after every summary.

## How do reasoning tokens affect the budget?

DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and the reasoning counts toward `max_tokens`. A long session does not change that, but large prompts tempt people to lower `max_tokens` to “save room”. Do not. A tight output budget produces empty answers, which the agent then retries. [Why answers come back empty](https://broadrouter.com/guides/reasoning-tokens-empty-response/).

Remember too that the window counts prompt and answer together. Near 1,048,576 tokens of prompt, there is no room left to answer. Compact before you get there; the [context window calculator](https://broadrouter.com/tools/context-window-calculator/) shows how much room a given prompt leaves.

## Does this matter on a flat plan?

On a BroadNet plan session, DeepSeek V4.1 Flash has no token meter: a Full Moon session is $60 a month however long the prompts are. The money argument for caching disappears. The other two do not. A cached prefix is typically faster to process in many serving systems, and a well-managed context gives better answers. The habits are the same; only the reason changes. [How plan sessions work](https://broadrouter.com/guides/unlimited-llm-api-sessions/).

## Context management checklist

-   System prompt and tool list fixed for the whole session.
-   History only appended; old turns never reformatted.
-   Cache rate logged per request.
-   Tool output capped when produced; stale output stubbed later.
-   Compaction at a high threshold, keeping the recent tail verbatim.
-   A notes file with goal, constraints and decisions.
-   Output budget large enough for reasoning.

For more on when a million-token window is worth using at all, see the [1M context window guide](https://broadrouter.com/blog/million-token-context-window-guide/). For stopping sessions that grow without making progress, see [agent loop cost control](https://broadrouter.com/blog/cut-coding-agent-llm-costs/).

Free tools for this

-   [Agent run cost estimator](https://broadrouter.com/tools/agent-run-cost-estimator/) Context growth per step, total tokens and cost of one agent run, with a DeepSeek session comparison.
-   [LLM batch job planner](https://broadrouter.com/tools/llm-batch-job-planner/) Run time, binding limit, cost per model and the flat sessions needed to finish a big job by a deadline.
-   [Flat-rate vs per-token LLM cost calculator](https://broadrouter.com/tools/flat-rate-vs-per-token-calculator/) Break-even between an unlimited plan and paying per token, per agent.
-   [LLM rate limit and concurrency calculator](https://broadrouter.com/tools/llm-rate-limit-calculator/) Which limit binds first: RPM, TPM or concurrency, with Little's law.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Context compaction](https://broadrouter.com/glossary/context-compaction/)

Context compaction is the practice of summarizing a conversation that is nearing the context window limit and continuing the session from that summary, so an agent keeps key decisions and state while dropping raw history such as old tool output.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[AI agent](https://broadrouter.com/glossary/agent/)

An AI agent is a system in which a language model directs its own steps toward a goal: it decides which tools to call, reads the results and repeats until it produces a final answer or hits a limit. A fixed script that calls a model at set points is usually called a workflow instead.

[AI coding agent](https://broadrouter.com/glossary/coding-agent/)

An AI coding agent is a software tool in which a language model works on a codebase in a loop: it reads files, edits them, runs commands such as tests, reads the output and decides the next step until the task is done or a limit is reached.

## Questions

### If the model has a 1M window, why manage context at all?

Because cost, latency and accuracy all change as the prompt grows. Long prompts take longer to process, cost more per turn on a per-token key, and research shows models use information in the middle of long inputs less reliably.

### How do I know whether my agent is hitting the cache?

When the model reports it, the usage block includes prompt\_tokens\_details.cached\_tokens. Divide it by prompt\_tokens for each request and log the rate.

### Does compaction matter on a flat plan?

Not for the bill, since a plan session has no token meter. It still matters for speed and for answer quality on very long sessions.

## Keep reading

-   [When a 1M-token context window is worth it (and when not)](https://broadrouter.com/blog/million-token-context-window-guide/)
-   [Cached input tokens: how to cut an LLM API bill](https://broadrouter.com/blog/cached-input-tokens-explained/)
-   [10 ways to cut coding agent LLM costs, with estimated savings](https://broadrouter.com/blog/cut-coding-agent-llm-costs/)
-   [How to run coding agents unattended overnight, safely](https://broadrouter.com/blog/overnight-coding-agent-runs/)
-   [Claude Code with open models: proxies, native endpoints and limits](https://broadrouter.com/blog/claude-code-with-openai-compatible-models/)
-   [OpenClaw and Hermes Agent on a cheap model: setup and token math](https://broadrouter.com/blog/openclaw-hermes-agent-cheap-model/)
-   [Guide: Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [Guide: How the unlimited LLM API plans work for agents](https://broadrouter.com/guides/unlimited-llm-api-sessions/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-long-running-agent-context-management&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference, reasoning and cached tokens](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-long-running-agent-context-management&utm_content=sources)
-   [Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv)](https://arxiv.org/abs/2307.03172)
-   [opencode docs, compaction](https://opencode.ai/docs/compaction/)
