BroadRouter
Get started

BM25

Updated

BM25 (Okapi BM25) is a ranking function for keyword search that scores a document by how often the query terms appear in it, how rare those terms are across the collection, and how long the document is. It is the default similarity in Lucene and Elasticsearch.

Also called: Okapi BM25, lexical search

BM25 is the keyword half of most search systems: fast, cheap and good at exact terms.

How it works

BM25 comes from the probabilistic relevance framework, summarized by Robertson and Zaragoza in “The Probabilistic Relevance Framework: BM25 and Beyond” (2009). Lucene’s class notes that it was introduced by Robertson, Walker, Jones and others at TREC-3.

For each query term, BM25 combines three signals:

  • Term frequency: more occurrences in the document raise the score, but with diminishing returns.
  • Inverse document frequency (IDF): rare terms count more. Lucene implements it as log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5)).
  • Length normalization: a match in a short document counts more than the same match in a long one.

Two parameters tune it. Elasticsearch’s docs:

Parameter Default What it controls
k1 1.2 “non-linear term frequency normalization (saturation)”
b 0.75 “to what degree document length normalizes tf values”

BM25 is the default similarity in Elasticsearch.

Why it matters

  • Exact matches. Product codes, names and error messages are where embeddings are weakest and BM25 is strong.
  • Cost. It needs no model calls or GPU, only an inverted index.
  • Hybrid search. Most production RAG systems pair it with vector search; see hybrid search.

Arabic text and BM25

BM25 matches tokens exactly, so it depends on how text is analyzed. In Arabic, the same word can be written with different alef forms, with or without diacritics, or with teh marbuta (ة) written as heh (ه), and articles and conjunctions attach to the word. Without Arabic text normalization and stemming, those variants do not match.

Elasticsearch’s built-in arabic analyzer chains these filters: lowercase, decimal_digit, arabic_stop, arabic_normalization, a keyword marker, and arabic_stemmer. Normalize queries and documents the same way. Arabic text normalization for search tests which rules help and which hurt, and the Arabic text normalizer tool lets you try them on your own text.

On BroadNet Inference

BM25 runs in your search engine, not in the LLM. BroadNet does not offer embeddings or rerank endpoints; it serves the chat models that read what you retrieve. Send your top BM25 or hybrid results to a model such as Qwen3.8 27B over the Chat Completions API. Arabic RAG search shows a full pipeline.

Questions

Is BM25 still useful when I have embeddings?

Yes. BM25 is strong on exact terms such as names, codes and error strings that embeddings can blur. Many RAG systems run both and merge the results with hybrid search.

What do k1 and b do?

k1 controls how quickly repeated terms stop adding score (term frequency saturation). b controls how much document length is normalized. Lucene and Elasticsearch default to k1 1.2 and b 0.75.

Does BM25 work for Arabic?

Yes, if the text is analyzed for Arabic first. Without normalization and stemming, spelling variants and attached prefixes stop words from matching.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity