INT4 is where quantization gets aggressive: each weight gets only 16 possible values.
How it works
TensorRT-LLM’s precision guide describes the common form, W4A16: weights are quantized to 4 bits and dequantized “on-the-fly” inside the matrix multiply, while activations stay in FP16 or BF16. TensorRT-LLM supports per-group scaling factors: each small group of weights gets its own scale so the 16 levels cover the right range. vLLM’s INT4 example saves a W4A16-G128 checkpoint, one group per 128 weights.
Plain rounding to 16 levels loses too much, so the popular methods are smarter about it:
- GPTQ uses approximate second-order information to choose roundings that limit output error. The paper reports 3 or 4 bits per weight with negligible loss, enough to run a 175B model on a single GPU.
- AWQ observes that “not all weights in an LLM are equally important.” Protecting about 1% of salient channels, found from activation statistics, greatly reduces error. Its TinyChat runtime reports more than 3 times speedup for 4-bit models on devices.
Why it matters
The memory saving is large. At 0.5 bytes per weight, a model’s weights take a quarter of their 16-bit size. For a 25.2B-parameter model: 25.2B × 2 bytes = 50.4 GB in BF16, against 25.2B × 0.5 bytes = 12.6 GB in INT4, plus a little for the scale factors. Generation speed is often limited by how fast weights stream from memory, so 4-bit weights can also cut latency. vLLM’s docs say INT4 W4A16 is most useful for keeping latency low “in workloads with low queries per second”. At high load, the cost of dequantizing weights matters more and the gain shrinks.
With only 16 levels, the rounding error is larger than with FP8, which is why GPTQ and AWQ exist at all. An average benchmark score can hide losses on the tasks you care about, such as long reasoning, exact arithmetic, code or a less common language. If an INT4 build does worse than the 16-bit version on your own test set, believe your test.
Mixture-of-experts models are frequent INT4 candidates, because all experts must fit in memory even though few run per token.
On BroadNet Inference
BroadNet’s model catalogue says its production Gemma 4 26B A4B build (API ID broadnet/gemma-4-26b-a4b) uses AWQ 4-bit weights, with multi-token prediction and an FP8 KV cache. That is BroadNet’s own description; the API docs do not state the precision of the other models. If you are choosing between self-hosting a 4-bit model and calling an API, the self-hosting cost comparison and the LLM API cost calculator help you compare the totals.