---
title: "What is quantization in LLMs? Fewer bits per weight | BroadRouter"
description: "Quantization stores LLM weights, and sometimes activations, in fewer bits to save GPU memory and speed up inference. Formats, memory math and quality trade-offs."
url: https://broadrouter.com/glossary/quantization/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Quantization

Updated 2026-11-14

Quantization is the process of storing a neural network's numbers, mainly its weights, in a lower-precision format such as 8-bit or 4-bit instead of 16-bit, so the model uses less memory and often runs faster. Good methods keep the accuracy loss small.

Also called: model quantization, LLM quantization, quantized model, weight quantization

Most open models ship with 16-bit weights. Quantization shrinks those numbers so a model fits on fewer GPUs.

## How it works

Hugging Face describes quantization as “representing weights and activations with lower-precision data types like 8-bit integers (int8).” Each block of weights gets a scale factor. The weights are rounded to a small set of levels and multiplied back by the scale at run time.

Memory for the weights scales with bits per parameter. Take Google’s Gemma 4 26B A4B, which has 25.2B parameters:

| Precision | Bytes per weight | Weights only |
| --- | --- | --- |
| 16-bit (BF16) | 2 | 25.2B × 2 = 50.4 GB |
| 8-bit ([FP8](https://broadrouter.com/glossary/fp8/) or INT8) | 1 | 25.2 GB |
| 4-bit ([INT4](https://broadrouter.com/glossary/int4/)) | 0.5 | 12.6 GB |

The [KV cache](https://broadrouter.com/glossary/kv-cache/) and activations need room on top of that.

Methods differ in what they quantize and how they pick the levels:

-   **LLM.int8()** cuts inference memory in half and keeps a few outlier features in 16-bit; the paper says more than 99.9% of values still use 8-bit math.
-   **GPTQ** is a one-shot weight method that reaches 3 or 4 bits per weight with “negligible accuracy degradation”, and quantizes a 175B model in about four GPU hours.
-   **Weight-only vs weight-and-activation.** Weight-only formats save memory. Quantizing activations too can also speed up the matrix math on hardware that supports it.

## Why it matters

For someone calling an API, quantization is invisible in the request but shows up in three places. Price: smaller models need fewer GPUs, which lowers serving cost. Speed: generation is often limited by memory bandwidth, so fewer bytes per weight can mean more [tokens per second](https://broadrouter.com/glossary/tokens-per-second/). Quality: fewer bits means more rounding error, and a model can keep its average score while slipping on the specific tasks you rely on.

If you compare two providers serving the “same” open model and get different results, precision is one likely cause.

## On BroadNet Inference

BroadNet’s API reference does not state serving precision. Its model catalogue lists the production Gemma 4 26B A4B build, offered as [BroadNet Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/), as AWQ 4-bit weights with an FP8 KV cache. It does not state the API precision of the other models. If precision matters for your use, run the same evaluation set against each provider; the [checklist for evaluating a model before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/) covers how.

## Questions

### Does quantization make a model worse?

Sometimes slightly. LLM.int8() reports no degradation for 8-bit inference up to 175B parameters, and GPTQ reports negligible loss at 3 or 4 bits per weight. Lower bits and harder tasks raise the risk, so test on your own prompts.

### Can I tell if an API serves a quantized model?

Usually only if the provider says so. The API returns the same response shape either way, so compare outputs on your own tests and ask the provider what precision it uses.

### How much memory does quantization save?

Weight memory scales with bits per weight. A 25.2B-parameter model needs about 50.4 GB at 16-bit, 25.2 GB at 8-bit and 12.6 GB at 4-bit, before the KV cache and activations.

## See also

-   [FP8](https://broadrouter.com/glossary/fp8/) (term)
-   [INT4](https://broadrouter.com/glossary/int4/) (term)
-   [KV cache](https://broadrouter.com/glossary/kv-cache/) (term)
-   [Mixture of experts (MoE)](https://broadrouter.com/glossary/mixture-of-experts/) (term)
-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/) (term)
-   [GGUF](https://broadrouter.com/glossary/gguf/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/) (article)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-quantization&utm_content=page-cta) ·

**Sources**

-   [Hugging Face Transformers, quantization reference](https://huggingface.co/docs/transformers/en/main_classes/quantization)
-   [Dettmers et al., LLM.int8() 8-bit Matrix Multiplication for Transformers at Scale, arXiv 2208.07339](https://arxiv.org/abs/2208.07339)
-   [Frantar et al., GPTQ, Accurate Post-Training Quantization for Generative Pre-trained Transformers, arXiv 2210.17323](https://arxiv.org/abs/2210.17323)
-   [Google, Gemma 4 26B A4B model card (parameter count)](https://huggingface.co/google/gemma-4-26B-A4B-it)
-   [BroadNet model catalogue (lab measurements, September 2026)](https://www.broadnet.ai/inference/models)
