RAG gives a model an open book: before it answers, your code looks up the relevant pages and puts them in the prompt.
How it works
The term comes from Lewis et al. (2020). They paired a generator model (“parametric memory”) with “a dense vector index of Wikipedia” (“non-parametric memory”) reached through a neural retriever, and reported better results on open-domain question answering than models that relied on their weights alone.
A modern RAG pipeline with an LLM API usually looks like this:
- Index. Split documents into chunks, turn each into embeddings and store them in a vector database, often next to a keyword index.
- Retrieve. Embed the user’s question and fetch the top matching chunks, say 5 to 20.
- Generate. Put those chunks in the prompt with an instruction to answer only from them and cite them. Send it to the chat model.
The model never searches by itself. Retrieval quality sets the ceiling: if the right passage is not retrieved, the model cannot use it.
Why it matters
- Freshness. The model’s weights stop at its training cutoff. RAG lets it answer from today’s price list or policy.
- Cost. Sending 10 chunks costs far fewer input tokens than sending the whole corpus on every question.
- Traceability. Answers can cite the passages they came from, which supports grounding and makes hallucinations easier to spot.
- Order matters. Models use the start and end of a long prompt better than the middle. Put the best passages first. See lost in the middle.
On BroadNet Inference
BroadNet covers the generation step. Its docs do not list an embeddings endpoint, so the retriever needs a separate embedding model or a keyword search. DeepSeek V4.1 Flash and GLM 5.3 accept up to 1,048,576 tokens of context, which makes generous retrieval (or no retrieval for small corpora) practical. The long context vs RAG post compares both on cost and recall, and Arabic RAG search covers normalisation and hybrid retrieval for Arabic.
Common mistakes
- Judging the model when the retriever is at fault. Log which chunks were retrieved for each answer.
- Chunks that cut tables or sentences in half.
- No instruction to say “not found”. Without it, the model fills gaps from memory.