BroadRouter
Get started

Quantization

Updated

Quantization is the process of storing a neural network's numbers, mainly its weights, in a lower-precision format such as 8-bit or 4-bit instead of 16-bit, so the model uses less memory and often runs faster. Good methods keep the accuracy loss small.

Also called: model quantization, LLM quantization, quantized model, weight quantization

Most open models ship with 16-bit weights. Quantization shrinks those numbers so a model fits on fewer GPUs.

How it works

Hugging Face describes quantization as “representing weights and activations with lower-precision data types like 8-bit integers (int8).” Each block of weights gets a scale factor. The weights are rounded to a small set of levels and multiplied back by the scale at run time.

Memory for the weights scales with bits per parameter. Take Google’s Gemma 4 26B A4B, which has 25.2B parameters:

Precision Bytes per weight Weights only
16-bit (BF16) 2 25.2B × 2 = 50.4 GB
8-bit (FP8 or INT8) 1 25.2 GB
4-bit (INT4) 0.5 12.6 GB

The KV cache and activations need room on top of that.

Methods differ in what they quantize and how they pick the levels:

  • LLM.int8() cuts inference memory in half and keeps a few outlier features in 16-bit; the paper says more than 99.9% of values still use 8-bit math.
  • GPTQ is a one-shot weight method that reaches 3 or 4 bits per weight with “negligible accuracy degradation”, and quantizes a 175B model in about four GPU hours.
  • Weight-only vs weight-and-activation. Weight-only formats save memory. Quantizing activations too can also speed up the matrix math on hardware that supports it.

Why it matters

For someone calling an API, quantization is invisible in the request but shows up in three places. Price: smaller models need fewer GPUs, which lowers serving cost. Speed: generation is often limited by memory bandwidth, so fewer bytes per weight can mean more tokens per second. Quality: fewer bits means more rounding error, and a model can keep its average score while slipping on the specific tasks you rely on.

If you compare two providers serving the “same” open model and get different results, precision is one likely cause.

On BroadNet Inference

BroadNet’s API reference does not state serving precision. Its model catalogue lists the production Gemma 4 26B A4B build, offered as BroadNet Gemma 4 VL 26B, as AWQ 4-bit weights with an FP8 KV cache. It does not state the API precision of the other models. If precision matters for your use, run the same evaluation set against each provider; the checklist for evaluating a model before switching covers how.

Questions

Does quantization make a model worse?

Sometimes slightly. LLM.int8() reports no degradation for 8-bit inference up to 175B parameters, and GPTQ reports negligible loss at 3 or 4 bits per weight. Lower bits and harder tasks raise the risk, so test on your own prompts.

Can I tell if an API serves a quantized model?

Usually only if the provider says so. The API returns the same response shape either way, so compare outputs on your own tests and ask the provider what precision it uses.

How much memory does quantization save?

Weight memory scales with bits per weight. A 25.2B-parameter model needs about 50.4 GB at 16-bit, 25.2 GB at 8-bit and 12.6 GB at 4-bit, before the KV cache and activations.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity