A vector database answers one question very fast: which stored items mean something close to this query?
How it works
Each record is an embedding, a fixed-length list of numbers, plus metadata such as the source text, a document ID and a date. At query time you embed the question with the same model and ask for the top k nearest vectors.
“Nearest” needs a distance measure. pgvector, the Postgres extension, offers three operators: <-> for Euclidean (L2) distance, <#> for negative inner product and <=> for cosine distance. Use the one your embedding model was trained for; OpenAI recommends cosine similarity for its vectors.
Comparing a query against every stored vector is exact but slow at scale. Approximate nearest-neighbour (ANN) indexes avoid that:
| Index | How it works | Trade-off (per pgvector docs) |
|---|---|---|
| HNSW | Layered proximity graph | Better speed-recall, slower build, more memory |
| IVFFlat | Vectors grouped into lists | Faster build, less memory, lower query speed at the same recall |
The HNSW paper by Malkov and Yashunin describes a stack of graphs where search starts at sparse upper layers, which “allows a logarithmic complexity scaling.” pgvector’s README sums up the deal: an approximate index “trades some recall for speed.”
Why it matters
- It is the retriever in most RAG systems. If the store misses the right chunk, the LLM never sees it.
- Recall is a setting, not a given. ANN parameters change which results come back. Measure recall on real queries.
- Filtering. Real queries add conditions such as language, customer or date. Check that filters apply before ranking, not after a top-k cut that drops valid hits.
On BroadNet Inference
BroadNet does not host vector storage and its docs do not list an embeddings endpoint. A typical setup is your own vector store, an embedding model from another source, and a BroadNet chat model that writes the answer from retrieved chunks. For small corpora, a large context window can replace the store; see long context vs RAG. For Arabic text, Arabic RAG search explains why a keyword index next to the vector index helps.
Common mistakes
- Querying with a different embedding model than the one used to index.
- Tuning the LLM prompt when the real problem is low retrieval recall.