BroadRouter
Get started

Chunking (RAG)

Updated

Chunking is the step in a retrieval pipeline that splits documents into smaller pieces, called chunks, before they are embedded and indexed. Chunk size sets how much text each piece holds, and chunk overlap repeats some text between neighbouring chunks so ideas cut at a boundary are not lost.

Also called: chunk size, chunk overlap, text chunking, document chunking, chunking strategy

A retriever cannot return half a document. Chunking decides what the pieces are.

How it works

LangChain’s docs put the purpose simply: text splitters “break large docs into smaller chunks that will be retrievable individually and fit within model context window limit.” Each chunk is then turned into embeddings and stored in a vector database, or indexed for keyword search.

Two numbers control most chunkers:

  • Chunk size: the maximum length of a chunk, in characters or tokens.
  • Chunk overlap: how much text from the end of one chunk is repeated at the start of the next.

Splitters also differ in where they cut. LangChain recommends starting with its recursive splitter for most cases. It tries to keep larger units such as paragraphs whole, and falls back to sentences, then words, when a unit is too big. LlamaIndex’s SentenceSplitter works on sentence boundaries; its docs show it with explicit settings:

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
nodes = splitter.get_nodes_from_documents(documents)

A worked number: a 100,000-token corpus split into 500-token chunks with 50 tokens of overlap gives about 222 chunks (each new chunk adds 450 new tokens) and about 111,000 tokens to embed, 11% more than with no overlap.

Why it matters

  • Recall. Chunks that are too small lose context; chunks that are too big dilute the match with unrelated text.
  • LLM cost. Retrieved chunks are sent as input tokens. Five 1,000-token chunks add 5,000 input tokens to every question.
  • Index cost. Overlap and small chunks both increase the number of vectors to embed and store.
  • Language. Arabic words can carry clitics and diacritics that change token counts and matching. Arabic RAG search covers chunking and normalization for Arabic.

Chunking is one lever. Adding hybrid search or a reranker often helps more than tuning size alone.

On BroadNet Inference

BroadNet does not offer an embeddings endpoint, so you chunk and embed with your own tools or another provider, and send the retrieved chunks to a BroadNet chat model. Long-context models change the trade-off: DeepSeek V4.1 Flash and GLM 5.3 accept 1,048,576 tokens, so a small corpus may fit without retrieval at all. Long context vs RAG compares the two.

The RAG chunking calculator estimates chunk count, index size and per-query tokens for your settings, and RAG chunk size and overlap walks through published tests.

Questions

What chunk size should I start with?

There is no single right value. Pick a starting size, build a small test set of real questions with known answers, and compare retrieval hit rates at two or three sizes. The RAG chunk size post collects published results.

Does overlap make the index bigger?

Yes. Overlapping text is stored and embedded more than once, so index size and embedding cost grow with the overlap share.

Should I chunk by characters or by tokens?

Token-based sizes match what the embedding model and LLM actually count. Character sizes are simpler, but the same number of characters can be a very different number of tokens in different languages, including Arabic.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity