BroadRouter

Models BroadNet evaluated and ruled out, and why

By the BroadNet Inference team · 4 min read · Published · Updated

BroadNet's model catalogue marks about 30 models and methods as ruled out after testing in its lab. The reasons mostly fall into five groups: a model lost to a stronger one for the same job, a credential-safety gate, a licence that blocks API serving, a quantisation format that lost a head-to-head, and Arabic regressions. All results are BroadNet's own measurements from September 2026.

BroadNet’s model catalogue lists every model it says it has run, benchmarked, quantised or fine-tuned on its own hardware, and marks the ones it rejected with a reason. About 30 models and methods carry the RULED OUT label. The figures below were measured in BroadNet’s lab in September 2026; they are BroadNet’s results, not ours.

The catalogue uses six labels: PROD (carried live traffic), SERVED (in the catalogue’s words, “ran on our fleet”), BENCH (benchmarked), QUANT (BroadNet produced the quantisation), FT (BroadNet fine-tuned it) and RULED OUT.

Which chat models did BroadNet rule out?

Model What BroadNet did Why it was ruled out
Qwen3.5 35B A3B (AWQ 4-bit) Served as a speed tier, about 164 tokens/s Failed BroadNet’s credential-safety battery for security roles
GLM 4.7 Flash (AWQ) Served experimentally Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production
Nemotron 3 Nano 30B A3B (NVFP4) Served with a custom reasoning parser Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production
Qwen3.6 35B A3B Served on RTX 5090 and GB10 Lost the chatbot slot to Gemma 4 26B
Gemma 4 E4B (AWQ INT4) Served on GB10 as a co-tenant, about 94 tokens/s Lost the chatbot slot to Gemma 4 26B
Gemma 4 26B A4B NVFP4 (three publishers) Quantisation evaluated Retired in favour of AWQ after a head-to-head on sm_120 and sm_121 GPUs
EAGLE3 speculative decoding Tested for saturated serving Cost 69% of throughput at 32 concurrent requests; multi-token-prediction drafters kept

The pattern matters for anyone choosing an open model. Three of the rejections are about safety, not quality: models that were fast enough but repeated credentials. That is the failure mode BroadNet’s Safety Map study found most often in small-active mixture-of-experts models. The model that won the chatbot slot, Gemma 4 26B A4B, is the one BroadNet sells as Gemma 4 VL 26B.

Which speech models were ruled out?

Model Why it was ruled out
Gemma 4 E2B, Gemma 4 12B (as transcribers) The E4B audio variant was kept instead
Omnilingual ASR 300M, 1B, 3B and CTC variants (Meta) The 7B variant earned the slot
Voxtral Mini 3B / 4B Realtime (Mistral) Ruled out for Arabic
ArTSTv3 (MBZUAI lineage) Ruled out for the Arabic slot; Cohere Transcribe Arabic kept
FireRedVAD, DeepFilterNet, BNR (front-ends) Silero voice activity detection adopted after a head-to-head

BroadNet reports that the model it kept, Cohere Transcribe Arabic 07-2026, scores 21.1 word error rate on MASC dialect speech, against 22.3 for ElevenLabs Scribe v2 and 31.5 for Whisper large-v3, on its own test. It serves that model as BroadNet Transcribe Arabic. The catalogue also says Whisper large-v3 was benchmarked as a fallback.

Which document, embedding and image models lost?

OCR and document parsing

  • LlamaParse, the incumbent API, scored 16.6 on BroadNet’s OmniDocBench subset. BroadNet ruled it out because “it loses tables, and self-hosting is 98% cheaper per 1,000 pages”.
  • olmOCR-2, HunyuanOCR, GOT-OCR 2.0 and Nanonets OCR2 were evaluated and ruled out; Nanonets on licence terms. dots.mocr was kept as the production pick because it both parses and describes figures, even though PaddleOCR-VL 1.6 scored higher (94.74 against 86.81).

Embeddings

  • gemini-embedding-2 was ruled out for Arabic after it regressed against gemini-embedding-001 (0.767 against 0.824 on BroadNet’s retrieval benchmark).
  • nomic-embed-text v1.5, the former incumbent, scored 0.468 overall and 0.130 on Arabic, and was retired.
  • Octen 8B, Harrier 0.6B and Nemotron Embed 1B were served and ruled out on real Arabic traffic, “where our podium held despite their public-board wins”.

BroadNet adds one finding that outranked model choice: a workload-specific query instruction moved retrieval quality (nDCG) by 0.048, more than moving from a 4B to an 8B embedding model.

Image generation and safety

  • Cosmos3 Super 64B and HunyuanImage 3.0 80B were ruled out on hardware fit and licence terms.
  • openai/privacy-filter 1.5B caught 100% of credentials but 0% of salary figures in BroadNet’s leak corpus, so it was ruled out as a firewall.

What can you learn from the rejections?

Four lessons hold beyond BroadNet’s own stack:

  1. Test the job, not the leaderboard. Several embedding models with public-board wins lost on real Arabic traffic.
  2. Safety is a separate gate. A fast, capable model can still fail a credential test. Run one before a model sees secrets; BroadNet’s evaluation method is a starting point.
  3. Licences decide what can be served. Non-commercial models such as Voxtral 4B TTS and Higgs Audio v3 appear in the catalogue only as quality ceilings.
  4. Serving choices are measured too. Quantisation formats and drafting methods were ruled out on head-to-head numbers, not by default. See BroadNet’s GPU hardware and serving approach.

Every number here comes from one company’s lab and traffic. Use it as a shortlist of what to test, then evaluate on your own data before you switch. A ruled-out or benchmarked row is also not something you can call; reading a model catalogue explains how to tell live, on-request and research entries apart.

Questions

Why does BroadNet publish the models it rejects?

Its catalogue says it publishes "what we rule out as carefully as what we ship", so buyers can see the reasoning behind each recommendation. A rejection with a reason is evidence you can test against your own workload.

Does "ruled out" mean the model is bad?

No. Most rejections are for one slot in BroadNet's own pipeline, on the setup and traffic BroadNet reports. Qwen3.6 35B A3B and Gemma 4 E4B lost a chatbot slot to Gemma 4 26B; they may still suit your job.

What is the credential-safety battery?

BroadNet's test of whether a model repeats credentials it sees, based on its published sensitive-data evaluation method. Models that fail are held below production for security roles.

Why do licences rule models out?

Some strong models, such as Voxtral 4B TTS and Higgs Audio v3, are non-commercial, so BroadNet benchmarks them only as a quality ceiling and cannot serve them through a paid API.

Are the ruled-out models available on BroadNet Inference?

No. The API sells six models, and three more are available on request. Models marked ruled out are not offered.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity