BM25 is the keyword half of most search systems: fast, cheap and good at exact terms.
How it works
BM25 comes from the probabilistic relevance framework, summarized by Robertson and Zaragoza in “The Probabilistic Relevance Framework: BM25 and Beyond” (2009). Lucene’s class notes that it was introduced by Robertson, Walker, Jones and others at TREC-3.
For each query term, BM25 combines three signals:
- Term frequency: more occurrences in the document raise the score, but with diminishing returns.
- Inverse document frequency (IDF): rare terms count more. Lucene implements it as
log(1 + (docCount - docFreq + 0.5)/(docFreq + 0.5)). - Length normalization: a match in a short document counts more than the same match in a long one.
Two parameters tune it. Elasticsearch’s docs:
| Parameter | Default | What it controls |
|---|---|---|
k1 |
1.2 | “non-linear term frequency normalization (saturation)” |
b |
0.75 | “to what degree document length normalizes tf values” |
BM25 is the default similarity in Elasticsearch.
Why it matters
- Exact matches. Product codes, names and error messages are where embeddings are weakest and BM25 is strong.
- Cost. It needs no model calls or GPU, only an inverted index.
- Hybrid search. Most production RAG systems pair it with vector search; see hybrid search.
Arabic text and BM25
BM25 matches tokens exactly, so it depends on how text is analyzed. In Arabic, the same word can be written with different alef forms, with or without diacritics, or with teh marbuta (ة) written as heh (ه), and articles and conjunctions attach to the word. Without Arabic text normalization and stemming, those variants do not match.
Elasticsearch’s built-in arabic analyzer chains these filters: lowercase, decimal_digit, arabic_stop, arabic_normalization, a keyword marker, and arabic_stemmer. Normalize queries and documents the same way. Arabic text normalization for search tests which rules help and which hurt, and the Arabic text normalizer tool lets you try them on your own text.
On BroadNet Inference
BM25 runs in your search engine, not in the LLM. BroadNet does not offer embeddings or rerank endpoints; it serves the chat models that read what you retrieve. Send your top BM25 or hybrid results to a model such as Qwen3.8 27B over the Chat Completions API. Arabic RAG search shows a full pipeline.