BroadRouter
Get started

FP8

Updated

FP8 is an 8-bit floating-point number format used in deep learning, defined in two encodings: E4M3, with 4 exponent and 3 mantissa bits, and E5M2, with 5 exponent and 2 mantissa bits. It halves memory compared with 16-bit formats and runs natively on recent GPUs.

Also called: 8-bit floating point, E4M3, E5M2, FP8 quantization, W8A8 FP8

FP8 is the 8-bit number format most new GPUs speak natively. It is often the first step down from 16-bit when serving a large model.

How it works

A floating-point number has a sign, an exponent and a mantissa. The FP8 paper by Micikevicius et al. defines two layouts, each with one sign bit:

Encoding Exponent bits Mantissa bits Trade-off
E4M3 4 3 More precision; vLLM lists a maximum of ±448
E5M2 5 2 Less precision; vLLM lists a maximum of ±57,344

E5M2 follows IEEE 754 rules for special values. E4M3 gives up infinities and keeps a single NaN pattern to stretch its range. Both have only 256 possible values, so a scale factor maps each tensor into the range the format covers.

As a quantization target, FP8 is usually applied as W8A8: both weights and activations in 8-bit, with a scale factor per tensor or per block. The paper reports FP8 matching 16-bit training quality on models up to 175B parameters. It also notes FP8 post-training quantization worked on language models “that resisted fixed point int8 quantization”, because the exponent copes better with outliers.

FP8 also shows up inside other formats. DeepSeek’s card for DeepSeek V4.1 Flash stores its main KV cache in 4-bit floats with “one E4M3 scale per 16 channels”.

Why it matters

vLLM’s docs state that FP8 gives “a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy.” For a provider, that means a model fits on fewer GPUs, or the same GPUs hold more requests at once, which raises throughput and lowers cost per token.

For an API user, FP8 is the mild end of quantization: vLLM reports minimal accuracy impact, though you should still test the tasks you rely on. It is a less aggressive step than INT4, which saves more memory but carries more risk on hard tasks.

On BroadNet Inference

BroadNet’s model catalogue says its production Gemma 4 26B A4B build stores the KV cache in FP8, with INT4 AWQ weights. That is BroadNet’s own description; the API docs do not state the precision of the other models. If an exact number format matters for an audit or a regulated workflow, ask BroadNet directly. For what does affect your bill, see the per-token prices on the pricing page.

Questions

Is FP8 the same as INT8?

No. Both use 8 bits, but FP8 spends some bits on an exponent, so it covers a wider range of magnitudes with less precision between them. INT8 spaces its 256 levels evenly and depends more on good scale factors.

Which GPUs support FP8?

vLLM's docs list FP8 compute on NVIDIA GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell) and AMD MI300x. Older Turing and Ampere cards can load FP8 weights but compute in 16-bit.

How is FP8 different from INT4?

FP8 uses 8 bits with an exponent and is usually applied to weights and activations. INT4 uses 16 evenly spaced integer levels and is usually applied to weights only. INT4 saves more memory; FP8 carries less accuracy risk.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity