Most open models ship with 16-bit weights. Quantization shrinks those numbers so a model fits on fewer GPUs.
How it works
Hugging Face describes quantization as “representing weights and activations with lower-precision data types like 8-bit integers (int8).” Each block of weights gets a scale factor. The weights are rounded to a small set of levels and multiplied back by the scale at run time.
Memory for the weights scales with bits per parameter. Take Google’s Gemma 4 26B A4B, which has 25.2B parameters:
| Precision | Bytes per weight | Weights only |
|---|---|---|
| 16-bit (BF16) | 2 | 25.2B × 2 = 50.4 GB |
| 8-bit (FP8 or INT8) | 1 | 25.2 GB |
| 4-bit (INT4) | 0.5 | 12.6 GB |
The KV cache and activations need room on top of that.
Methods differ in what they quantize and how they pick the levels:
- LLM.int8() cuts inference memory in half and keeps a few outlier features in 16-bit; the paper says more than 99.9% of values still use 8-bit math.
- GPTQ is a one-shot weight method that reaches 3 or 4 bits per weight with “negligible accuracy degradation”, and quantizes a 175B model in about four GPU hours.
- Weight-only vs weight-and-activation. Weight-only formats save memory. Quantizing activations too can also speed up the matrix math on hardware that supports it.
Why it matters
For someone calling an API, quantization is invisible in the request but shows up in three places. Price: smaller models need fewer GPUs, which lowers serving cost. Speed: generation is often limited by memory bandwidth, so fewer bytes per weight can mean more tokens per second. Quality: fewer bits means more rounding error, and a model can keep its average score while slipping on the specific tasks you rely on.
If you compare two providers serving the “same” open model and get different results, precision is one likely cause.
On BroadNet Inference
BroadNet’s API reference does not state serving precision. Its model catalogue lists the production Gemma 4 26B A4B build, offered as BroadNet Gemma 4 VL 26B, as AWQ 4-bit weights with an FP8 KV cache. It does not state the API precision of the other models. If precision matters for your use, run the same evaluation set against each provider; the checklist for evaluating a model before switching covers how.