BroadRouter
Get started

Context engineering

Updated

Context engineering is the work of choosing and maintaining the set of tokens a language model sees on each request, including system prompt, tool definitions, history, retrieved documents and notes, so the model gets the smallest set of high-signal information it needs to do the task well.

Also called: context management, agent context engineering

A prompt is what you write once. Context is everything the model sees on each turn, and in an agent it changes every turn.

How it works

Anthropic’s engineering post describes context engineering as “the set of strategies for curating and maintaining the optimal set of tokens” during inference, including information beyond the prompt. It sets the goal as finding the “smallest possible set of high-signal tokens” that makes the desired outcome likely.

The reason is limited attention. The post says models have an “attention budget,” and that recall falls as context grows, which it calls context rot. Position matters too: facts in the middle of a long prompt are easier to miss (lost in the middle).

The post names several techniques for long-running agents:

Technique What it does
Compaction Summarize history near the limit and continue from the summary
Tool result clearing Drop old raw tool outputs from the history
Structured note-taking The agent writes notes to a file outside the context and rereads them later
Subagents Do focused work in a clean context and return a short summary
Just-in-time retrieval Keep references (file paths, queries) and load the data only when needed

A worked example of one turn’s budget in a coding agent:

System prompt + rules          4,000 tokens
Tool definitions               6,000 tokens
Notes file                     1,500 tokens
History (compacted)           20,000 tokens
Latest tool output             8,000 tokens   <- largest single item

Why it matters

  • Quality. Less noise in the context usually means better answers and fewer missed instructions.
  • Cost. On per-token billing, every token in the context is billed on every turn. Stable content placed first is cheaper as cached input; see prompt structure for caching.
  • Length of run. Good context management lets an agent work for hours without hitting the context window.

On BroadNet Inference

Context engineering happens in your code: BroadNet’s Chat Completions API receives whatever message list you send. DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576 tokens, Qwen3.8 27B 262,144 and Gemma 4 VL 26B 32K, so a tight context matters most on Gemma.

When the model reports them, usage.prompt_tokens_details.cached_tokens shows the tokens read from cache, which are billed at the cached input price on pay-as-you-go. Context management for long-running agents gives five working rules, and the agent token budget post shows where agent tokens go.

Questions

How is context engineering different from prompt engineering?

Prompt engineering is about writing good instructions. Context engineering covers everything the model sees on a turn, including tool definitions, history and retrieved data, and how that set changes over a long session.

Why not just use a bigger context window?

A bigger window delays the problem but does not remove it. Anthropic notes that recall drops as context grows, and on per-token billing every extra token is paid again on each turn.

What is the first thing to try?

Measure what fills your context. Tool output and old history are usually the largest parts, and trimming or summarizing them is cheap to try.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity