---
title: "RAG chunk size and overlap - retrieval and token cost"
description: "How chunk size and overlap change RAG recall, index size, embedding cost and the tokens every query sends to the LLM, with published results and worked numbers."
url: https://broadrouter.com/blog/rag-chunk-size-overlap/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# RAG chunk size and overlap: what they do to retrieval and cost

By the BroadNet Inference team · 7 min read · Published 2026-11-15 · Updated 2026-11-16

Start with chunks of about 256 to 512 tokens, 0 to 15% overlap, and top-k tuned on your own queries. Bigger chunks raise the tokens every query sends to the LLM. Overlap multiplies the tokens you embed and store by size ÷ (size − overlap), so 50% overlap doubles the index. Published tests found small or no-overlap chunks often recall as well or better.

## Key takeaways

-   Input per query is roughly top-k × chunk size + prompt. Moving from 5 × 400 to 8 × 1,024 tokens sends about 3.4 times the input on every query.
-   Overlap inflates the index by size ÷ (size − overlap): 15% overlap adds up to about 18% (17% on the 10,000-token documents measured here), and 50% doubles it.
-   In Chroma's 2024 evaluation, 200- to 400-token chunks with no overlap matched or beat 800-token chunks with 400 overlap, which is OpenAI file search's default.
-   NVIDIA's 2025 tests found no single best size: factoid questions favoured 256 to 512 tokens, analytical ones 1,024 tokens or a page.
-   Count chunk size in tokens of your embedding model. Some splitters count characters by default.

**On this page**

1.  What do chunk size and overlap change?
2.  What do published tests say about chunk size?
3.  How much does overlap inflate the index?
4.  What does chunk size do to the LLM bill?
5.  How do you chunk by tokens without cutting words?
6.  How do you pick the numbers for your data?
7.  Where BroadNet Inference fits

Chunk size and overlap are the two numbers every RAG pipeline sets early and rarely revisits. They decide how many vectors you store, how many tokens you embed, how many tokens every question sends to the LLM, and whether the right passage is found at all. This post shows what published tests found, then works through the token math so you can see the cost of each setting before you embed anything.

If you are still deciding whether to use RAG at all, start with [long context vs RAG](https://broadrouter.com/blog/long-context-vs-rag/). For Arabic documents, read [Arabic RAG](https://broadrouter.com/blog/arabic-rag-search/) and [Arabic text normalization](https://broadrouter.com/blog/arabic-text-normalization-search/) alongside this post.

## What do chunk size and overlap change?

-   **Chunk size** is how many tokens go into each piece you embed and retrieve.
-   **Overlap** is how many tokens consecutive chunks share, so a sentence cut at a boundary appears whole in at least one chunk.
-   **Top-k** is how many chunks you retrieve per question.

Together they set four things:

| Quantity | Formula (long documents) |
| --- | --- |
| Chunks stored | corpus tokens ÷ (size − overlap) |
| Tokens embedded | corpus tokens × size ÷ (size − overlap) |
| Context per query | top-k × size |
| LLM input per query | top-k × size + system prompt + question |

The first two are paid once per indexing run, and again every time you re-chunk. The last is paid on every question, forever.

## What do published tests say about chunk size?

Two public evaluations are worth knowing, because they measured different things.

**Chroma (Smith and Troynikov, July 2024)** measured retrieval at the token level: how much of the relevant text was retrieved (recall), and how much of what was retrieved was relevant. It used five retrieved chunks and OpenAI’s text-embedding-3-large. Selected rows:

| Splitter | Size (tokens) | Overlap | Recall | IoU |
| --- | --- | --- | --- | --- |
| Recursive | 800 | 400 | 85.4% | 1.5% |
| Token | 800 | 400 | 87.9% | 1.4% |
| Recursive | 400 | 200 | 88.1% | 3.3% |
| Recursive | 400 | 0 | 89.5% | 3.6% |
| Recursive | 200 | 0 | 88.1% | 6.9% |

IoU (intersection over union) measures how much of the retrieved text was relevant, so it doubles as a measure of wasted tokens. The 800/400 rows match OpenAI’s default for file search; assuming a token splitter, Chroma found that setting gave slightly below-average recall and the lowest scores on the other metrics. Smaller chunks with no overlap recalled as much or more and carried far less irrelevant text.

**NVIDIA (June 2025)** measured end-to-end answer accuracy on five datasets, with token chunks of 128 to 2,048 at 15% overlap, plus page-level and section-level chunking. Page-level chunking had the highest average accuracy (0.648) and the lowest variance. The best token size depended on the questions: factoid sets favoured 256 to 512 tokens, analytical ones 1,024 tokens or whole pages. Both extremes, 128 and 2,048, usually did worse.

The two agree on one thing: **there is no universal best size**, and very large chunks with heavy overlap are rarely the right default.

## How much does overlap inflate the index?

Each chunk advances by size − overlap tokens, so the index holds size ÷ (size − overlap) copies of the corpus. For a 2,000,000-token corpus of 10,000-token documents:

| Size | Overlap | Chunks | Tokens embedded | Inflation |
| --- | --- | --- | --- | --- |
| 200 | 0 | 10,000 | 2,000,000 | 1.00× |
| 400 | 0 | 5,000 | 2,000,000 | 1.00× |
| 512 | 77 (15%) | 4,600 | 2,338,800 | 1.17× |
| 1,024 | 154 (15%) | 2,400 | 2,338,800 | 1.17× |
| 800 | 400 (50%) | 4,800 | 3,840,000 | 1.92× |
| 400 | 200 (50%) | 9,800 | 3,920,000 | 1.96× |

Short documents inflate a little less, because the last chunk of each one is partial. The formula gives the limit: 15% overlap costs about 18% more, and 50% costs about 100% more, in embedding tokens, vector storage and re-indexing time.

Overlap also costs at query time. With 50% overlap, two neighbouring chunks retrieved for the same question repeat half their text. You pay for those duplicate tokens, and they push other relevant passages out of top-k.

BroadNet doesn’t sell embeddings, so this side of the bill goes to your embedding provider. Price it with the [embedding cost calculator](https://broadrouter.com/tools/embedding-cost-calculator/).

## What does chunk size do to the LLM bill?

Every question sends top-k chunks plus the prompt. Take 100,000 questions a month, 600 tokens of system prompt and question, and the same 300-token answer each time:

| Size × top-k | Context per query | Input per query | Input per month |
| --- | --- | --- | --- |
| 200 × 10 | 2,000 | 2,600 | 260 million |
| 400 × 5 | 2,000 | 2,600 | 260 million |
| 512 × 5 | 2,560 | 3,160 | 316 million |
| 800 × 5 | 4,000 | 4,600 | 460 million |
| 1,024 × 5 | 5,120 | 5,720 | 572 million |
| 1,024 × 8 | 8,192 | 8,792 | 879 million |

Output is 30 million tokens a month in every row, so only input changes. At a price per 1M tokens, monthly input cost is the millions in the last column times that price. On [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) at $0.10 per 1M input tokens, the 1,024 × 8 setup costs 879 × $0.10 a month in input against 260 × $0.10 for 400 × 5. That is 3.4 times as much, for a setting the published tests don’t show recalling better. On [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) ($0.175 input) or [GLM 5.3](https://broadrouter.com/models/glm-5.3/) ($0.78 input), the ratio is the same and the totals are larger.

Two levers move this more than the model price does:

1.  **Lower k when you raise the size.** 200 × 10 and 400 × 5 send the same context. Keep context per query roughly fixed while you test sizes, so you compare retrieval quality, not context budget.
2.  **Cache the fixed part.** Put the system prompt first and keep it byte-identical, so it can bill at a [cached input](https://broadrouter.com/blog/cached-input-tokens-explained/) rate on models that have one. Retrieved chunks change per question and rarely cache.

Context limits matter too. Gemma 4 VL 26B accepts 32,768 tokens per request, plenty for any row above. If you push k × size toward tens of thousands of tokens, you have moved from RAG towards [long context](https://broadrouter.com/blog/long-context-vs-rag/), and that comparison has its own math. The [RAG chunking calculator](https://broadrouter.com/tools/rag-chunking-calculator/) runs these formulas for your own corpus, chunk size, overlap and top-k.

## How do you chunk by tokens without cutting words?

Count in tokens, but cut at structure. LangChain’s `RecursiveCharacterTextSplitter` measures size with its `length_function`, which is character length by default. A 512 “size” is then 512 characters, roughly 100 to 150 tokens of English, and a different number again for Arabic; see [tokens per word in Arabic](https://broadrouter.com/blog/tokens-per-word-arabic-english/). Embedding limits and LLM bills are in tokens.

This chunker packs whole paragraphs up to a token budget and carries trailing paragraphs forward as overlap. Cutting at paragraph boundaries avoids splitting words or multi-byte Arabic characters, which raw token windows can do.

```python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")    # an estimate: count with your embedder's tokenizer
count = lambda s: len(enc.encode(s))

def chunk(text, size=512, overlap=64):
    """Pack whole paragraphs up to `size` tokens; repeat trailing paragraphs up to `overlap` tokens."""
    chunks, cur, cur_len = [], [], 0
    for para in (p.strip() for p in text.split("\n\n")):
        if not para:
            continue
        n = count(para)
        if cur and cur_len + n > size:
            chunks.append("\n\n".join(cur))
            tail, t = [], 0
            for q in reversed(cur):                  # carry the end of this chunk forward
                if t + count(q) > overlap:
                    break
                tail.insert(0, q); t += count(q)
            cur, cur_len = tail, t
        cur.append(para); cur_len += n              # a paragraph over `size` stays whole
    if cur:
        chunks.append("\n\n".join(cur))
    return chunks

def plan(corpus_tokens, size, overlap, k, queries_per_month, prompt_tokens=600):
    index_tokens = corpus_tokens * size / (size - overlap)   # long-document limit
    per_query = k * size + prompt_tokens
    return {"chunks": round(index_tokens / size), "tokens_to_embed": round(index_tokens),
            "input_per_query": per_query, "input_per_month": per_query * queries_per_month}

print(plan(2_000_000, 512, 64, 5, 100_000))
# {'chunks': 4464, 'tokens_to_embed': 2285714, 'input_per_query': 3160, 'input_per_month': 316000000}
```

Overlap here is “up to”: a paragraph longer than the overlap budget is not repeated. A single paragraph longer than `size` stays whole; split such paragraphs at sentences first if your documents have them. Prefix each chunk with its document title and section heading, so a retrieved chunk still says what it is about. That adds a few tokens per chunk and often helps more than overlap.

## How do you pick the numbers for your data?

1.  **Build an evaluation set.** 50 to 200 real questions, each with the passage that answers it. The [LLM evaluation guide](https://broadrouter.com/blog/evaluate-llm-before-switching/) covers scoring.
2.  **Fix the context budget.** Hold k × size near one value, such as 2,000 tokens, while you compare sizes.
3.  **Try three sizes and two overlaps.** For example 256, 512 and 1,024 tokens, each at 0 and 15% overlap. Measure recall of the right passage at k.
4.  **Try structure.** If your documents have pages or numbered sections, test page-level or section-level chunks against the token sizes. NVIDIA found page-level the most consistent.
5.  **Pick the cheapest setting within a point or two of the best recall,** then price it at your query volume with the [monthly cost method](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/).

Do this on a sample before you embed the full corpus. Changing chunking later means re-embedding everything.

## Where BroadNet Inference fits

BroadNet Inference serves the answer step of RAG, not retrieval. It sells no embeddings or vector storage, so pair it with an embeddings provider and a vector database of your choice. For the answer, Gemma 4 VL 26B is the low-cost option and runs on the $5 starting credit. Qwen3.8 27B (262,144-token context) and GLM 5.3 (1,048,576) suit larger top-k or long documents, after a top-up. Prompts and retrieved passages are not retained after the request. If your corpus is small and stable enough to fit in the context window, [long context](https://broadrouter.com/blog/long-context-vs-rag/) may beat chunking altogether.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Chunking (RAG)](https://broadrouter.com/glossary/chunking/)

Chunking is the step in a retrieval pipeline that splits documents into smaller pieces, called chunks, before they are embedded and indexed. Chunk size sets how much text each piece holds, and chunk overlap repeats some text between neighbouring chunks so ideas cut at a boundary are not lost.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

## Questions

### What is a good chunk size for RAG?

Start around 256 to 512 tokens and test. Chroma's evaluation found 200 to 400 tokens with no overlap performed well. NVIDIA found 256 to 512 tokens suited factoid questions, while analytical questions did better with 1,024-token or page-level chunks.

### How much chunk overlap should I use?

Less than most defaults. NVIDIA found 15% best of the 10%, 15% and 20% it tried on one dataset. Chroma found no-overlap settings often scored as well or better. Every point of overlap adds tokens to embed, store and send.

### Does a bigger chunk size cost more?

Not to embed, without overlap: the corpus is the same number of tokens. It costs more per query, because each retrieved chunk is longer, so top-k × chunk size grows. Lower k when you raise the size.

### Is chunk size measured in tokens or characters?

It depends on the splitter. LangChain's RecursiveCharacterTextSplitter measures with its length\_function, which is character length by default. Embedding limits and LLM bills are in tokens, so pass a token counter.

### Do I have to re-embed when I change chunk size?

Yes. New boundaries make new chunks, and each needs a new vector. Re-embedding costs the whole index again, so settle chunking on a sample before you embed the full corpus.

## Keep reading

-   [Long context vs RAG: when 1M tokens replaces a vector DB](https://broadrouter.com/blog/long-context-vs-rag/)
-   [Arabic RAG: normalisation, chunking and retrieval that works](https://broadrouter.com/blog/arabic-rag-search/)
-   [Arabic text normalization for search and RAG: what to change, what not](https://broadrouter.com/blog/arabic-text-normalization-search/)
-   [How to estimate LLM API cost before you launch a feature](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/)
-   [Cached input tokens: how to cut an LLM API bill](https://broadrouter.com/blog/cached-input-tokens-explained/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-rag-chunk-size-overlap&utm_content=page-cta) ·

**Sources**

-   [Smith and Troynikov, Evaluating Chunking Strategies for Retrieval, Chroma technical report, July 2024 (checked 2026-11-15)](https://research.trychroma.com/evaluating-chunking)
-   [NVIDIA Technical Blog, Finding the Best Chunking Strategy for Accurate AI Responses, June 2025 (checked 2026-11-15)](https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/)
-   [OpenAI retrieval guide: default 800-token chunks with 400-token overlap, limits on chunking\_strategy (checked 2026-11-15)](https://developers.openai.com/api/docs/guides/retrieval)
-   [LangChain docs, RecursiveCharacterTextSplitter: chunk size measured by length\_function, characters by default (checked 2026-11-15)](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter)
-   [BroadNet Inference pricing and API reference: models, context lengths, cached input (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-rag-chunk-size-overlap&utm_content=sources)
