---
title: "What is INT4 quantization? 4-bit LLM weights explained"
description: "INT4 stores each LLM weight as a 4-bit integer, a quarter of its 16-bit size. How 4-bit weight-only quantization, GPTQ and AWQ work, and the quality risks."
url: https://broadrouter.com/glossary/int4/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# INT4

Updated 2026-11-16

INT4 is a 4-bit integer number format that can represent 16 levels. In LLM inference it usually means 4-bit weight-only quantization (W4A16): weights are stored as INT4 with scale factors and expanded to 16-bit during computation, using about a quarter of the memory of 16-bit weights.

Also called: 4-bit quantization, W4A16, 4-bit integer, INT4 weight-only, 4-bit LLM

INT4 is where quantization gets aggressive: each weight gets only 16 possible values.

## How it works

TensorRT-LLM’s precision guide describes the common form, **W4A16**: weights are quantized to 4 bits and dequantized “on-the-fly” inside the matrix multiply, while activations stay in FP16 or BF16. TensorRT-LLM supports per-group scaling factors: each small group of weights gets its own scale so the 16 levels cover the right range. vLLM’s INT4 example saves a `W4A16-G128` checkpoint, one group per 128 weights.

Plain rounding to 16 levels loses too much, so the popular methods are smarter about it:

-   **GPTQ** uses approximate second-order information to choose roundings that limit output error. The paper reports 3 or 4 bits per weight with negligible loss, enough to run a 175B model on a single GPU.
-   **AWQ** observes that “not all weights in an LLM are equally important.” Protecting about 1% of salient channels, found from activation statistics, greatly reduces error. Its TinyChat runtime reports more than 3 times speedup for 4-bit models on devices.

## Why it matters

The memory saving is large. At 0.5 bytes per weight, a model’s weights take a quarter of their 16-bit size. For a 25.2B-parameter model: 25.2B × 2 bytes = 50.4 GB in BF16, against 25.2B × 0.5 bytes = 12.6 GB in INT4, plus a little for the scale factors. Generation speed is often limited by how fast weights stream from memory, so 4-bit weights can also cut [latency](https://broadrouter.com/glossary/latency/). vLLM’s docs say INT4 W4A16 is most useful for keeping latency low “in workloads with low queries per second”. At high load, the cost of dequantizing weights matters more and the gain shrinks.

With only 16 levels, the rounding error is larger than with [FP8](https://broadrouter.com/glossary/fp8/), which is why GPTQ and AWQ exist at all. An average benchmark score can hide losses on the tasks you care about, such as long reasoning, exact arithmetic, code or a less common language. If an INT4 build does worse than the 16-bit version on your own test set, believe your test.

[Mixture-of-experts](https://broadrouter.com/glossary/mixture-of-experts/) models are frequent INT4 candidates, because all experts must fit in memory even though few run per token.

## On BroadNet Inference

BroadNet’s [model catalogue](https://www.broadnet.ai/inference/models) says its production Gemma 4 26B A4B build (API ID `broadnet/gemma-4-26b-a4b`) uses AWQ 4-bit weights, with multi-token prediction and an FP8 KV cache. That is BroadNet’s own description; the API docs do not state the precision of the other models. If you are choosing between self-hosting a 4-bit model and calling an API, the [self-hosting cost comparison](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) and the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/) help you compare the totals.

## Questions

### How much memory does INT4 save?

Weights take half a byte each instead of two, so about a quarter of 16-bit size, plus a small overhead for scale factors. A 25.2B-parameter model drops from about 50.4 GB to about 12.6 GB of weights.

### Is INT4 the same as NF4 or FP4?

No. All three use 4 bits, but NF4 and FP4 place their 16 levels unevenly, while INT4 spaces them evenly. Hugging Face's bitsandbytes integration offers FP4 and NF4; GPTQ and AWQ checkpoints usually use INT4.

### Does INT4 make a model faster?

Often at low load, because generation is limited by reading weights from memory and 4-bit weights are smaller. vLLM's docs say INT4 W4A16 helps most in workloads with low queries per second; under heavy load the gain shrinks.

## See also

-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [FP8](https://broadrouter.com/glossary/fp8/) (term)
-   [Mixture of experts (MoE)](https://broadrouter.com/glossary/mixture-of-experts/) (term)
-   [Latency](https://broadrouter.com/glossary/latency/) (term)
-   [AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-int4&utm_content=page-cta) ·

**Sources**

-   [BroadNet model catalogue (Gemma 4 26B A4B, AWQ 4-bit)](https://www.broadnet.ai/inference/models)
-   [NVIDIA TensorRT-LLM, numerical precision (INT4 and INT8 weight-only)](https://nvidia.github.io/TensorRT-LLM/reference/precision.html)
-   [Lin et al., AWQ, Activation-aware Weight Quantization, arXiv 2306.00978](https://arxiv.org/abs/2306.00978)
-   [Frantar et al., GPTQ, arXiv 2210.17323](https://arxiv.org/abs/2210.17323)
-   [vLLM docs, INT4 W4A16 quantization](https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/int4/)
-   [Hugging Face Transformers docs, bitsandbytes (NF4 and FP4 4-bit types) (checked 2026-11-16)](https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes)
