---
title: "Long context vs RAG - cost, latency and recall compared"
description: "When a 1M-token context window can replace a vector database and when RAG still wins, compared on cost per query, latency, recall and upkeep."
url: https://broadrouter.com/blog/long-context-vs-rag/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Long context vs RAG: when 1M tokens replaces a vector DB

By the BroadNet Inference team · 5 min read · Published 2026-11-14 · Updated 2026-11-16

Long context replaces RAG when the corpus fits in the window, changes rarely and questions need many parts of it at once; caching keeps repeat queries cheap. RAG still wins when the corpus outgrows the window, updates constantly, needs per-user permissions or must answer in a second or two. Many systems retrieve with RAG and reason with long context.

## Key takeaways

-   Corpus size is the first filter: if it does not fit in the window with room for the answer, you need retrieval of some kind.
-   Per query, long context sends the whole corpus; RAG sends a few thousand tokens. Prefix caching narrows the cost gap only when the corpus is stable.
-   Long prompts raise time to first token. RAG keeps prompts short and fast.
-   Recall in long context is not perfect: models find facts at the start and end of a prompt more reliably than in the middle.
-   RAG fails differently: if retrieval misses the passage, the model never sees it. Measure both on your own questions.

**On this page**

1.  When does long context replace RAG?
2.  How do cost per query compare?
3.  How does latency compare?
4.  Which has better recall?
5.  What about upkeep?
6.  The hybrid most teams end up with
7.  Where BroadNet Inference fits

The [1M-token context window guide](https://broadrouter.com/blog/million-token-context-window-guide/) covers what fits in a million tokens and what one request costs. This post answers a narrower question that comes up in every architecture review: **can we delete the [vector database](https://broadrouter.com/glossary/vector-database/) and the [RAG](https://broadrouter.com/glossary/rag/) pipeline around it?** Sometimes yes. Here is how to decide, on four axes: cost, latency, recall and upkeep.

## When does long context replace RAG?

All four of these should be true:

1.  **The corpus fits.** Prompt plus answer plus reasoning stays inside the [context window](https://broadrouter.com/glossary/context-window/) with a margin. On [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) and [GLM 5.3](https://broadrouter.com/models/glm-5.3/) that window is 1,048,576 tokens ([the two compared](https://broadrouter.com/compare/deepseek-v4.1-flash-vs-glm-5.3/)). The [context window calculator](https://broadrouter.com/tools/context-window-calculator/) checks whether your documents fit.
2.  **It changes rarely.** A manual updated monthly is fine. A ticket queue updated every minute is not.
3.  **Everyone may see all of it.** One shared corpus, no per-user permissions to filter.
4.  **Questions span the corpus.** “Which contracts conflict with the new policy?” needs everything in view; “what is the refund window?” does not.

If one is false, keep retrieval, or use a hybrid.

## How do cost per query compare?

Take a 400,000-token document set and a question with a 500-token answer. RAG retrieves, say, 8,000 tokens of chunks. On a [pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/) key with DeepSeek V4.1 Flash:

| Approach | Input per query | Input cost per query |
| --- | --- | --- |
| Long context, first query | 400,000 new | 0.4 × $0.175 |
| Long context, repeat query, prefix cached | about 400,000 cached | about 0.4 × $0.0055 |
| RAG | 8,000 new | 0.008 × $0.175 |

Output is the same in each case: 500 tokens × $0.52 per 1M. (Each multiplier is millions of tokens times the price per 1M.)

Three things follow. Without caching, long context costs 50 times RAG per query on input, because it sends 50 times the tokens. With the corpus cached, the gap shrinks a lot, since cached input on this model costs a few percent of new input. But caching only holds while the prefix stays byte-identical, so the corpus must come first and stay stable; see [prompt structure for caching](https://broadrouter.com/blog/prompt-structure-for-caching/). RAG has its own costs that this table leaves out: embedding the corpus, running the vector store, and the engineering time to tune chunking. The [embedding cost calculator](https://broadrouter.com/tools/embedding-cost-calculator/) prices the embedding part. The [RAG chunking calculator](https://broadrouter.com/tools/rag-chunking-calculator/) prices the query side: chunks, tokens per query and the monthly LLM bill.

To see what your own cache rate is worth, try the [prompt caching calculator](https://broadrouter.com/tools/prompt-caching-calculator/). On a flat plan the input column disappears: Full Moon is $60 a month per session with no token meter, so a 400,000-token prompt costs the same as a short one.

## How does latency compare?

A model reads the whole prompt before it writes the first token. Time to first token grows with prompt length, and a prompt hundreds of thousands of tokens long takes noticeably longer to start than an 8,000-token one. Cached prefixes help in many serving systems. Vector search itself usually adds little; the retrieval call is small next to generation.

So for an interactive assistant that must reply in a second or two, RAG is the safer default. For an analyst who asks one deep question and waits for the answer, long context is fine. [Measure TTFT on your own prompts](https://broadrouter.com/blog/llm-latency-ttft-throughput/) at the size you plan to send.

## Which has better recall?

Neither is perfect, and they fail in different ways.

**Long context** can lose facts in the middle ([lost in the middle](https://broadrouter.com/glossary/lost-in-the-middle/)). The _Lost in the Middle_ study found models used information best when it sat at the start or end of the input, and worse in between. The RULER benchmark found that models scoring near-perfect on simple needle-in-a-haystack tests degraded on harder tasks as inputs grew, and that of the models claiming 32K tokens or more, “only half of them” kept satisfactory performance at 32K. Newer models do better, but the advertised window is not the same as the window where quality holds.

**RAG** fails before the model starts: if the retriever misses the passage, the model never sees it. Questions that need many passages, or that use different words from the source, are the classic misses.

| Failure mode | Long context | RAG |
| --- | --- | --- |
| Fact buried mid-prompt | Risk grows with length | Low, prompt is short |
| Answer needs 20 scattered passages | Strong | Weak unless top-k is large |
| Query wording differs from the source | Strong | Depends on embeddings and reranking |
| Corpus bigger than the window | Not possible | Strong |

Two mitigations for long context: put the question at the end, after the documents, and ask the model to quote the lines it relies on.

## What about upkeep?

RAG is a system: chunking, embeddings, an index, re-indexing on change, permission filters, evaluation of retrieval quality. Long context is a prompt: concatenate files with clear labels and send. For a small team with a corpus that fits, deleting the pipeline removes real work. For a large, changing, permissioned corpus, the pipeline is doing a job long context cannot.

## The hybrid most teams end up with

1.  Retrieve generously, not five chunks but whole documents, up to a few hundred thousand [tokens](https://broadrouter.com/glossary/token/).
2.  Order them stably (by document ID, not by score) so the prefix caches across related questions.
3.  Put the question last and send it to a long-context model.
4.  Ask for citations to document labels, and spot-check them.

This keeps RAG’s reach over a huge corpus and long context’s ability to reason across many sources. It also tolerates a weaker retriever, because a near-miss document is still in the prompt.

## Where BroadNet Inference fits

BroadNet Inference offers two 1M-token models, DeepSeek V4.1 Flash and GLM 5.3, and it does not trim your context, so an oversized request fails rather than losing documents silently. Cached input is billed at the lower cached rate on a per-token key, and the unlimited plans remove the token meter entirely for DeepSeek V4.1 Flash. Its API reference lists chat completions and transcription, not an embeddings endpoint or a vector store, so for the RAG half you bring your own. Leave room for [reasoning tokens](https://broadrouter.com/glossary/reasoning-tokens/) in the window; the [reasoning guide](https://broadrouter.com/guides/reasoning-tokens-empty-response/) explains why. Prices are on the [pricing page](https://broadrouter.com/pricing/).

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[Vector database](https://broadrouter.com/glossary/vector-database/)

A vector database is a data store that holds embedding vectors, often with their source text and metadata, and answers nearest-neighbour queries: given a query vector, it returns the stored vectors most similar to it, usually by cosine distance, Euclidean distance or inner product.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[Lost in the middle](https://broadrouter.com/glossary/lost-in-the-middle/)

Lost in the middle is the observed tendency of language models to use relevant information best when it appears near the beginning or end of a long input and worst when it sits in the middle. It was named in a 2023 study by Liu et al.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

## Questions

### Is RAG dead now that models have 1M-token context windows?

No. Long context removes the need for retrieval when the whole corpus fits and is stable. Most company knowledge bases are larger than a million tokens, change daily and need permissions, which is still RAG territory.

### Is long context more accurate than RAG?

It depends on the question. Long context does better when the answer needs many passages combined. RAG can do better on single-fact lookups in huge corpora, because a short prompt keeps the model focused. Test both on real questions.

### How many words fit in a 1 million token context window?

About 810,000 words of plain English prose in DeepSeek V4.1 Flash's 1,048,576-token window, at the 1,290 tokens per 1,000 words measured across five tokenizers in BroadRouter's tokenizer dataset. Technical English, code, tables and Arabic need more tokens per word, so less fits. Leave room for the answer and reasoning.

### Can I combine RAG and long context?

Yes, and it is often the best option. Retrieve generously, for example 100,000 to 200,000 tokens of candidate documents, then let a long-context model reason over all of them instead of the top five chunks.

## Keep reading

-   [When a 1M-token context window is worth it (and when not)](https://broadrouter.com/blog/million-token-context-window-guide/)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/)
-   [Cached input tokens: how to cut an LLM API bill](https://broadrouter.com/blog/cached-input-tokens-explained/)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-long-context-vs-rag&utm_content=page-cta) ·

**Sources**

-   [Liu et al., Lost in the Middle - How Language Models Use Long Contexts (arXiv 2307.03172)](https://arxiv.org/abs/2307.03172)
-   [Hsieh et al., RULER - What's the Real Context Size of Your Long-Context Language Models? (arXiv 2404.06654)](https://arxiv.org/abs/2404.06654)
-   [BroadNet Inference machine-readable prices and context lengths (pricing.json)](https://inference.broadnet.ai/pricing.json?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-long-context-vs-rag&utm_content=sources)
-   [BroadNet Inference API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-long-context-vs-rag&utm_content=sources)
