---
title: "What is chunking in RAG? Chunk size and overlap explained"
description: "Chunking splits documents into pieces a RAG system can embed, index and retrieve. How chunk size and overlap work, common settings, and the cost trade-offs."
url: https://broadrouter.com/glossary/chunking/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Chunking (RAG)

Updated 2026-11-15

Chunking is the step in a retrieval pipeline that splits documents into smaller pieces, called chunks, before they are embedded and indexed. Chunk size sets how much text each piece holds, and chunk overlap repeats some text between neighbouring chunks so ideas cut at a boundary are not lost.

Also called: chunk size, chunk overlap, text chunking, document chunking, chunking strategy

A retriever cannot return half a document. Chunking decides what the pieces are.

## How it works

LangChain’s docs put the purpose simply: text splitters “break large docs into smaller chunks that will be retrievable individually and fit within model context window limit.” Each chunk is then turned into [embeddings](https://broadrouter.com/glossary/embeddings/) and stored in a [vector database](https://broadrouter.com/glossary/vector-database/), or indexed for keyword search.

Two numbers control most chunkers:

-   **Chunk size**: the maximum length of a chunk, in characters or tokens.
-   **Chunk overlap**: how much text from the end of one chunk is repeated at the start of the next.

Splitters also differ in where they cut. LangChain recommends starting with its recursive splitter for most cases. It tries to keep larger units such as paragraphs whole, and falls back to sentences, then words, when a unit is too big. LlamaIndex’s `SentenceSplitter` works on sentence boundaries; its docs show it with explicit settings:

```python
from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
nodes = splitter.get_nodes_from_documents(documents)
```

A worked number: a 100,000-token corpus split into 500-token chunks with 50 tokens of overlap gives about 222 chunks (each new chunk adds 450 new tokens) and about 111,000 tokens to embed, 11% more than with no overlap.

## Why it matters

-   **Recall.** Chunks that are too small lose context; chunks that are too big dilute the match with unrelated text.
-   **LLM cost.** Retrieved chunks are sent as input tokens. Five 1,000-token chunks add 5,000 input tokens to every question.
-   **Index cost.** Overlap and small chunks both increase the number of vectors to embed and store.
-   **Language.** Arabic words can carry clitics and diacritics that change token counts and matching. [Arabic RAG search](https://broadrouter.com/blog/arabic-rag-search/) covers chunking and normalization for Arabic.

Chunking is one lever. Adding [hybrid search](https://broadrouter.com/glossary/hybrid-search/) or a [reranker](https://broadrouter.com/glossary/reranking/) often helps more than tuning size alone.

## On BroadNet Inference

BroadNet does not offer an embeddings endpoint, so you chunk and embed with your own tools or another provider, and send the retrieved chunks to a BroadNet chat model. Long-context models change the trade-off: [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) and [GLM 5.3](https://broadrouter.com/models/glm-5.3/) accept 1,048,576 tokens, so a small corpus may fit without retrieval at all. [Long context vs RAG](https://broadrouter.com/blog/long-context-vs-rag/) compares the two.

The [RAG chunking calculator](https://broadrouter.com/tools/rag-chunking-calculator/) estimates chunk count, index size and per-query tokens for your settings, and [RAG chunk size and overlap](https://broadrouter.com/blog/rag-chunk-size-overlap/) walks through published tests.

## Questions

### What chunk size should I start with?

There is no single right value. Pick a starting size, build a small test set of real questions with known answers, and compare retrieval hit rates at two or three sizes. The RAG chunk size post collects published results.

### Does overlap make the index bigger?

Yes. Overlapping text is stored and embedded more than once, so index size and embedding cost grow with the overlap share.

### Should I chunk by characters or by tokens?

Token-based sizes match what the embedding model and LLM actually count. Character sizes are simpler, but the same number of characters can be a very different number of tokens in different languages, including Arabic.

## See also

-   [RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/) (term)
-   [Embeddings](https://broadrouter.com/glossary/embeddings/) (term)
-   [Vector database](https://broadrouter.com/glossary/vector-database/) (term)
-   [Hybrid search](https://broadrouter.com/glossary/hybrid-search/) (term)
-   [Reranking](https://broadrouter.com/glossary/reranking/) (term)
-   [Context window](https://broadrouter.com/glossary/context-window/) (term)
-   [Use LlamaIndex with BroadNet Inference through OpenAILike](https://broadrouter.com/guides/llamaindex-openailike/) (guide)
-   [Use LangChain with BroadNet Inference](https://broadrouter.com/guides/langchain-openai-base-url/) (guide)
-   [RAG chunk size and overlap: what they do to retrieval and cost](https://broadrouter.com/blog/rag-chunk-size-overlap/) (article)
-   [Arabic RAG: normalisation, chunking and retrieval that works](https://broadrouter.com/blog/arabic-rag-search/) (article)
-   [Long context vs RAG: when 1M tokens replaces a vector DB](https://broadrouter.com/blog/long-context-vs-rag/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-chunking&utm_content=page-cta) ·

**Sources**

-   [LangChain docs, text splitters (checked 2026-11-15)](https://docs.langchain.com/oss/python/integrations/splitters)
-   [LlamaIndex docs, node parser usage pattern (SentenceSplitter) (checked 2026-11-15)](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/)
