FP8 is the 8-bit number format most new GPUs speak natively. It is often the first step down from 16-bit when serving a large model.
How it works
A floating-point number has a sign, an exponent and a mantissa. The FP8 paper by Micikevicius et al. defines two layouts, each with one sign bit:
| Encoding | Exponent bits | Mantissa bits | Trade-off |
|---|---|---|---|
| E4M3 | 4 | 3 | More precision; vLLM lists a maximum of ±448 |
| E5M2 | 5 | 2 | Less precision; vLLM lists a maximum of ±57,344 |
E5M2 follows IEEE 754 rules for special values. E4M3 gives up infinities and keeps a single NaN pattern to stretch its range. Both have only 256 possible values, so a scale factor maps each tensor into the range the format covers.
As a quantization target, FP8 is usually applied as W8A8: both weights and activations in 8-bit, with a scale factor per tensor or per block. The paper reports FP8 matching 16-bit training quality on models up to 175B parameters. It also notes FP8 post-training quantization worked on language models “that resisted fixed point int8 quantization”, because the exponent copes better with outliers.
FP8 also shows up inside other formats. DeepSeek’s card for DeepSeek V4.1 Flash stores its main KV cache in 4-bit floats with “one E4M3 scale per 16 channels”.
Why it matters
vLLM’s docs state that FP8 gives “a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy.” For a provider, that means a model fits on fewer GPUs, or the same GPUs hold more requests at once, which raises throughput and lowers cost per token.
For an API user, FP8 is the mild end of quantization: vLLM reports minimal accuracy impact, though you should still test the tasks you rely on. It is a less aggressive step than INT4, which saves more memory but carries more risk on hard tasks.
On BroadNet Inference
BroadNet’s model catalogue says its production Gemma 4 26B A4B build stores the KV cache in FP8, with INT4 AWQ weights. That is BroadNet’s own description; the API docs do not state the precision of the other models. If an exact number format matters for an audit or a regulated workflow, ask BroadNet directly. For what does affect your bill, see the per-token prices on the pricing page.