BroadRouter
Get started

RAG (retrieval-augmented generation)

Updated

RAG (retrieval-augmented generation) is a pattern in which an application first retrieves passages relevant to a question from an external store, such as a search or vector index, and then adds them to the prompt so a language model can answer from that text rather than from memory alone.

Also called: retrieval-augmented generation, retrieval augmented generation, RAG pipeline, retrieval-augmented LLM

RAG gives a model an open book: before it answers, your code looks up the relevant pages and puts them in the prompt.

How it works

The term comes from Lewis et al. (2020). They paired a generator model (“parametric memory”) with “a dense vector index of Wikipedia” (“non-parametric memory”) reached through a neural retriever, and reported better results on open-domain question answering than models that relied on their weights alone.

A modern RAG pipeline with an LLM API usually looks like this:

  1. Index. Split documents into chunks, turn each into embeddings and store them in a vector database, often next to a keyword index.
  2. Retrieve. Embed the user’s question and fetch the top matching chunks, say 5 to 20.
  3. Generate. Put those chunks in the prompt with an instruction to answer only from them and cite them. Send it to the chat model.

The model never searches by itself. Retrieval quality sets the ceiling: if the right passage is not retrieved, the model cannot use it.

Why it matters

  • Freshness. The model’s weights stop at its training cutoff. RAG lets it answer from today’s price list or policy.
  • Cost. Sending 10 chunks costs far fewer input tokens than sending the whole corpus on every question.
  • Traceability. Answers can cite the passages they came from, which supports grounding and makes hallucinations easier to spot.
  • Order matters. Models use the start and end of a long prompt better than the middle. Put the best passages first. See lost in the middle.

On BroadNet Inference

BroadNet covers the generation step. Its docs do not list an embeddings endpoint, so the retriever needs a separate embedding model or a keyword search. DeepSeek V4.1 Flash and GLM 5.3 accept up to 1,048,576 tokens of context, which makes generous retrieval (or no retrieval for small corpora) practical. The long context vs RAG post compares both on cost and recall, and Arabic RAG search covers normalisation and hybrid retrieval for Arabic.

Common mistakes

  • Judging the model when the retriever is at fault. Log which chunks were retrieved for each answer.
  • Chunks that cut tables or sentences in half.
  • No instruction to say “not found”. Without it, the model fills gaps from memory.

Questions

Is RAG the same as fine-tuning?

No. RAG adds documents to the prompt at request time and changes nothing in the model. Fine-tuning changes the model's weights. RAG is easier to update; you re-index a document instead of retraining.

Do I need a vector database for RAG?

Not always. Keyword search, a SQL query or a small in-memory index can all serve as the retriever. A vector database helps when you have many documents and need semantic matches.

Does a 1M-token context window replace RAG?

Sometimes. If the whole corpus fits and is queried often with caching, you can send it all. For large or fast-changing corpora, retrieval is still cheaper and faster per question.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity