BroadRouter
Get started

Reranking

Updated

Reranking is a second retrieval stage in which a more accurate but slower model, usually a cross-encoder, scores each query and candidate document pair from a first-pass search and reorders them, so only the most relevant few are passed to the language model.

Also called: reranker, rerank, re-ranking, cross-encoder reranker

First-pass search is fast but rough. Reranking spends more compute on a short list to get the order right.

How it works

A two-stage pipeline:

  1. Retrieve. A fast method, such as embeddings in a vector database, BM25 or hybrid search, returns the top 50 to 100 candidates.
  2. Rerank. A reranker reads the query and each candidate together and gives each pair a relevance score. Keep the top 3 to 10.

Most rerankers are cross-encoders. The Sentence Transformers docs say a cross-encoder “Generally provides superior performance compared to a Sentence Transformer (a.k.a. bi-encoder) model,” but is “Often slower” because “it requires computation for each pair rather than each text.” The docs describe cross-encoders as “often used to re-rank the top-k results” from a bi-encoder.

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
ranks = model.rank(query, passages)   # passages from your first-stage search
top = [passages[r["corpus_id"]] for r in ranks[:5]]

The approach goes back at least to Nogueira and Cho (2019), who fine-tuned BERT as a passage re-ranker and reported a 27% relative improvement in MRR@10 over the previous best result on MS MARCO passage retrieval.

Hosted rerank APIs exist too. Cohere’s rerank endpoint “takes in a query and a list of texts and produces an ordered array with each text assigned a relevance score,” with scores normalized to the range 0 to 1 and a top_n field to limit results.

Why it matters

  • Answer quality. The LLM can only use what you send it. Better ordering puts the right chunk in the prompt.
  • Token cost. With a good reranker you can send 3 chunks instead of 10, which cuts input tokens on every question.
  • Latency. The reranker adds a step. Its cost grows with the number of candidates, so keep that list short.

On BroadNet Inference

BroadNet does not offer a rerank endpoint or an embeddings endpoint. Run a cross-encoder yourself, for example with Sentence Transformers, or use a hosted rerank API, then send the top chunks to a BroadNet chat model over the OpenAI-compatible Chat Completions API.

For Arabic documents, check that your reranker was trained on Arabic or multilingual data; Arabic RAG search covers the rest of the pipeline. If your corpus is small, a long-context model such as DeepSeek V4.1 Flash may let you skip retrieval; see long context vs RAG.

Questions

Why not use the cross-encoder for the whole search?

It scores each query and document pair separately, so it is too slow to run over a whole collection. The first stage narrows millions of documents to tens or hundreds, and the reranker orders only those.

Can an LLM be the reranker?

Yes. You can ask a chat model to score or order candidates. It is usually slower and costs more per query than a small cross-encoder, so test both on your data.

How many candidates should I rerank?

Enough that the right answer is usually among them. Measure how often the correct chunk appears in the first-stage top 20, 50 and 100, and pick the smallest number with good recall.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity