First-pass search is fast but rough. Reranking spends more compute on a short list to get the order right.
How it works
A two-stage pipeline:
- Retrieve. A fast method, such as embeddings in a vector database, BM25 or hybrid search, returns the top 50 to 100 candidates.
- Rerank. A reranker reads the query and each candidate together and gives each pair a relevance score. Keep the top 3 to 10.
Most rerankers are cross-encoders. The Sentence Transformers docs say a cross-encoder “Generally provides superior performance compared to a Sentence Transformer (a.k.a. bi-encoder) model,” but is “Often slower” because “it requires computation for each pair rather than each text.” The docs describe cross-encoders as “often used to re-rank the top-k results” from a bi-encoder.
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
ranks = model.rank(query, passages) # passages from your first-stage search
top = [passages[r["corpus_id"]] for r in ranks[:5]]
The approach goes back at least to Nogueira and Cho (2019), who fine-tuned BERT as a passage re-ranker and reported a 27% relative improvement in MRR@10 over the previous best result on MS MARCO passage retrieval.
Hosted rerank APIs exist too. Cohere’s rerank endpoint “takes in a query and a list of texts and produces an ordered array with each text assigned a relevance score,” with scores normalized to the range 0 to 1 and a top_n field to limit results.
Why it matters
- Answer quality. The LLM can only use what you send it. Better ordering puts the right chunk in the prompt.
- Token cost. With a good reranker you can send 3 chunks instead of 10, which cuts input tokens on every question.
- Latency. The reranker adds a step. Its cost grows with the number of candidates, so keep that list short.
On BroadNet Inference
BroadNet does not offer a rerank endpoint or an embeddings endpoint. Run a cross-encoder yourself, for example with Sentence Transformers, or use a hosted rerank API, then send the top chunks to a BroadNet chat model over the OpenAI-compatible Chat Completions API.
For Arabic documents, check that your reranker was trained on Arabic or multilingual data; Arabic RAG search covers the rest of the pipeline. If your corpus is small, a long-context model such as DeepSeek V4.1 Flash may let you skip retrieval; see long context vs RAG.