---
title: "Models BroadNet ruled out: LLM, speech, OCR, embeddings"
description: "BroadNet publishes the models it tested and rejected, with reasons: chat models that failed its credential-safety gate, speech, OCR and embedding models that lost."
url: https://broadrouter.com/blog/models-broadnet-ruled-out/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Models BroadNet evaluated and ruled out, and why

By the BroadNet Inference team · 4 min read · Published 2026-11-15 · Updated 2026-11-16

BroadNet's model catalogue marks about 30 models and methods as ruled out after testing in its lab. The reasons mostly fall into five groups: a model lost to a stronger one for the same job, a credential-safety gate, a licence that blocks API serving, a quantisation format that lost a head-to-head, and Arabic regressions. All results are BroadNet's own measurements from September 2026.

## Key takeaways

-   BroadNet's catalogue labels each model PROD, SERVED, BENCH, QUANT, FT or RULED OUT, and gives the reason for every rejection.
-   BroadNet's credential-safety battery kept three chat models out of security roles: Qwen3.5 35B A3B, plus GLM 4.7 Flash and Nemotron 3 Nano 30B A3B, which stay labelled SERVED rather than RULED OUT.
-   EAGLE3 speculative decoding was ruled out after it cost 69% of throughput at 32 concurrent requests.
-   LlamaParse was ruled out as the document-parsing incumbent: it lost tables, and BroadNet says self-hosting was 98% cheaper per 1,000 pages.
-   Public leaderboard wins did not survive BroadNet's real Arabic traffic for several embedding models.

**On this page**

1.  Which chat models did BroadNet rule out?
2.  Which speech models were ruled out?
3.  Which document, embedding and image models lost?
4.  What can you learn from the rejections?

BroadNet’s [model catalogue](https://www.broadnet.ai/inference/models) lists every model it says it has run, benchmarked, quantised or fine-tuned on its own hardware, and marks the ones it rejected with a reason. About 30 models and methods carry the RULED OUT label. The figures below were measured in BroadNet’s lab in September 2026; they are BroadNet’s results, not ours.

The catalogue uses six labels: PROD (carried live traffic), SERVED (in the catalogue’s words, “ran on our fleet”), BENCH (benchmarked), QUANT (BroadNet produced the quantisation), FT (BroadNet fine-tuned it) and RULED OUT.

## Which chat models did BroadNet rule out?

| Model | What BroadNet did | Why it was ruled out |
| --- | --- | --- |
| Qwen3.5 35B A3B (AWQ 4-bit) | Served as a speed tier, about 164 tokens/s | Failed BroadNet’s credential-safety battery for security roles |
| GLM 4.7 Flash (AWQ) | Served experimentally | Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production |
| Nemotron 3 Nano 30B A3B (NVFP4) | Served with a custom reasoning parser | Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production |
| Qwen3.6 35B A3B | Served on RTX 5090 and GB10 | Lost the chatbot slot to Gemma 4 26B |
| Gemma 4 E4B (AWQ INT4) | Served on GB10 as a co-tenant, about 94 tokens/s | Lost the chatbot slot to Gemma 4 26B |
| Gemma 4 26B A4B NVFP4 (three publishers) | Quantisation evaluated | Retired in favour of AWQ after a head-to-head on sm\_120 and sm\_121 GPUs |
| EAGLE3 speculative decoding | Tested for saturated serving | Cost 69% of throughput at 32 concurrent requests; multi-token-prediction drafters kept |

The pattern matters for anyone choosing an open model. Three of the rejections are about safety, not quality: models that were fast enough but repeated credentials. That is the failure mode BroadNet’s [Safety Map study](https://broadrouter.com/blog/llm-safety-map-39-models/) found most often in small-active mixture-of-experts models. The model that won the chatbot slot, Gemma 4 26B A4B, is the one BroadNet sells as [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/).

## Which speech models were ruled out?

| Model | Why it was ruled out |
| --- | --- |
| Gemma 4 E2B, Gemma 4 12B (as transcribers) | The E4B audio variant was kept instead |
| Omnilingual ASR 300M, 1B, 3B and CTC variants (Meta) | The 7B variant earned the slot |
| Voxtral Mini 3B / 4B Realtime (Mistral) | Ruled out for Arabic |
| ArTSTv3 (MBZUAI lineage) | Ruled out for the Arabic slot; Cohere Transcribe Arabic kept |
| FireRedVAD, DeepFilterNet, BNR (front-ends) | Silero voice activity detection adopted after a head-to-head |

BroadNet reports that the model it kept, Cohere Transcribe Arabic 07-2026, scores 21.1 word error rate on MASC dialect speech, against 22.3 for ElevenLabs Scribe v2 and 31.5 for Whisper large-v3, on its own test. It serves that model as [BroadNet Transcribe Arabic](https://broadrouter.com/models/cohere-transcribe-arabic/). The catalogue also says Whisper large-v3 was benchmarked as a fallback.

## Which document, embedding and image models lost?

**OCR and document parsing**

-   **LlamaParse**, the incumbent API, scored 16.6 on BroadNet’s OmniDocBench subset. BroadNet ruled it out because “it loses tables, and self-hosting is 98% cheaper per 1,000 pages”.
-   **olmOCR-2, HunyuanOCR, GOT-OCR 2.0 and Nanonets OCR2** were evaluated and ruled out; Nanonets on licence terms. dots.mocr was kept as the production pick because it both parses and describes figures, even though PaddleOCR-VL 1.6 scored higher (94.74 against 86.81).

**Embeddings**

-   **gemini-embedding-2** was ruled out for Arabic after it regressed against gemini-embedding-001 (0.767 against 0.824 on BroadNet’s retrieval benchmark).
-   **nomic-embed-text v1.5**, the former incumbent, scored 0.468 overall and 0.130 on Arabic, and was retired.
-   **Octen 8B, Harrier 0.6B and Nemotron Embed 1B** were served and ruled out on real Arabic traffic, “where our podium held despite their public-board wins”.

BroadNet adds one finding that outranked model choice: a workload-specific query instruction moved retrieval quality (nDCG) by 0.048, more than moving from a 4B to an 8B embedding model.

**Image generation and safety**

-   **Cosmos3 Super 64B and HunyuanImage 3.0 80B** were ruled out on hardware fit and licence terms.
-   **openai/privacy-filter 1.5B** caught 100% of credentials but 0% of salary figures in BroadNet’s leak corpus, so it was ruled out as a firewall.

## What can you learn from the rejections?

Four lessons hold beyond BroadNet’s own stack:

1.  **Test the job, not the leaderboard.** Several embedding models with public-board wins lost on real Arabic traffic.
2.  **Safety is a separate gate.** A fast, capable model can still fail a credential test. Run one before a model sees secrets; BroadNet’s [evaluation method](https://broadrouter.com/blog/llm-sensitive-data-safety-evaluation-method/) is a starting point.
3.  **Licences decide what can be served.** Non-commercial models such as Voxtral 4B TTS and Higgs Audio v3 appear in the catalogue only as quality ceilings.
4.  **Serving choices are measured too.** Quantisation formats and drafting methods were ruled out on head-to-head numbers, not by default. See [BroadNet’s GPU hardware and serving approach](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/).

Every number here comes from one company’s lab and traffic. Use it as a shortlist of what to test, then [evaluate on your own data](https://broadrouter.com/blog/evaluate-llm-before-switching/) before you switch. A ruled-out or benchmarked row is also not something you can call; [reading a model catalogue](https://broadrouter.com/blog/understand-model-status/) explains how to tell live, on-request and research entries apart.

Free tools for this

-   [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/) Price a workload on every model at once, with cache hits, reasoning and a flat-plan check.
-   [LLM token counter](https://broadrouter.com/tools/token-counter/) Exact token counts with each model's real tokenizer, with a token view and cost.
-   [Context window calculator](https://broadrouter.com/tools/context-window-calculator/) Will it fit? Window used per model, room left for output, cost to send.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/)

AWQ (activation-aware weight quantization) is a post-training method that compresses an LLM's weights to low precision, usually 4-bit integers. It uses activation statistics from a small calibration set to find the most important weight channels and scales them so they lose less accuracy. It needs no backpropagation.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[INT4](https://broadrouter.com/glossary/int4/)

INT4 is a 4-bit integer number format that can represent 16 levels. In LLM inference it usually means 4-bit weight-only quantization (W4A16): weights are stored as INT4 with scale factors and expanded to 16-bit during computation, using about a quarter of the memory of 16-bit weights.

[Mixture of experts (MoE)](https://broadrouter.com/glossary/mixture-of-experts/)

Mixture of experts (MoE) is a neural network design in which a layer holds many expert sub-networks and a learned router sends each token to only a few of them. An MoE language model has a large total parameter count but uses a small active subset for each token.

[Open-weight model](https://broadrouter.com/glossary/open-weight-model/)

An open-weight model is a machine learning model whose trained weights are published for anyone to download and run, usually under a licence with conditions. It differs from fully open-source AI, which also releases the training code and detailed information about the training data, so others can study and reproduce the model.

[Speculative decoding](https://broadrouter.com/glossary/speculative-decoding/)

Speculative decoding is an inference technique in which a small, fast draft model proposes several next tokens and the large target model verifies them in a single forward pass, keeping the accepted ones. It speeds up generation without changing the distribution of the output.

## Questions

### Why does BroadNet publish the models it rejects?

Its catalogue says it publishes "what we rule out as carefully as what we ship", so buyers can see the reasoning behind each recommendation. A rejection with a reason is evidence you can test against your own workload.

### Does "ruled out" mean the model is bad?

No. Most rejections are for one slot in BroadNet's own pipeline, on the setup and traffic BroadNet reports. Qwen3.6 35B A3B and Gemma 4 E4B lost a chatbot slot to Gemma 4 26B; they may still suit your job.

### What is the credential-safety battery?

BroadNet's test of whether a model repeats credentials it sees, based on its published sensitive-data evaluation method. Models that fail are held below production for security roles.

### Why do licences rule models out?

Some strong models, such as Voxtral 4B TTS and Higgs Audio v3, are non-commercial, so BroadNet benchmarks them only as a quality ceiling and cannot serve them through a paid API.

### Are the ruled-out models available on BroadNet Inference?

No. The API sells six models, and three more are available on request. Models marked ruled out are not offered.

## Keep reading

-   [BroadNet's GPU hardware and serving approach, as BroadNet describes it](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/)
-   [The Safety Map: how 39 LLMs handle sensitive data](https://broadrouter.com/blog/llm-safety-map-39-models/)
-   [Arabic speech-to-text API: how to choose one in 2026](https://broadrouter.com/blog/arabic-speech-to-text-api-buyers-guide/)
-   [Live, on request, research: read a model catalogue clearly](https://broadrouter.com/blog/understand-model-status/)
-   [Artificial Analysis alternatives: where to compare LLMs in 2026](https://broadrouter.com/blog/artificial-analysis-alternatives/)
-   [LLM API SLAs compared: what Azure, Bedrock, Google and OpenAI promise](https://broadrouter.com/blog/llm-api-sla-uptime-compared/)
-   [Guide: Try BroadNet Inference in your browser, no account](https://broadrouter.com/guides/try-broadnet-inference-playground/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-models-broadnet-ruled-out&utm_content=page-cta) ·

**Sources**

-   [BroadNet model catalogue: every model it runs, benchmarks, quantizes or fine-tunes, including ruled-out models (measured September 2026; fetched 2026-11-15)](https://www.broadnet.ai/inference/models)
-   [BroadNet Research: findings on speculative decoding and serving (fetched 2026-11-15)](https://www.broadnet.ai/research)
-   [BroadNet Gemma 4 VL 26B model page (fetched 2026-11-15)](https://www.broadnet.ai/inference/models/gemma-4-vl-26b)
-   [BroadNet Transcribe Arabic model page (fetched 2026-11-15)](https://www.broadnet.ai/inference/models/cohere-transcribe-arabic)
