---
title: "What is FP8? 8-bit floating point for LLM inference | BroadRouter"
description: "FP8 is an 8-bit floating-point format (E4M3, E5M2) that runs LLMs in half the memory of 16-bit. How it differs from INT8, and which GPUs support it."
url: https://broadrouter.com/glossary/fp8/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# FP8

Updated 2026-11-14

FP8 is an 8-bit floating-point number format used in deep learning, defined in two encodings: E4M3, with 4 exponent and 3 mantissa bits, and E5M2, with 5 exponent and 2 mantissa bits. It halves memory compared with 16-bit formats and runs natively on recent GPUs.

Also called: 8-bit floating point, E4M3, E5M2, FP8 quantization, W8A8 FP8

FP8 is the 8-bit number format most new GPUs speak natively. It is often the first step down from 16-bit when serving a large model.

## How it works

A floating-point number has a sign, an exponent and a mantissa. The FP8 paper by Micikevicius et al. defines two layouts, each with one sign bit:

| Encoding | Exponent bits | Mantissa bits | Trade-off |
| --- | --- | --- | --- |
| E4M3 | 4 | 3 | More precision; vLLM lists a maximum of ±448 |
| E5M2 | 5 | 2 | Less precision; vLLM lists a maximum of ±57,344 |

E5M2 follows IEEE 754 rules for special values. E4M3 gives up infinities and keeps a single NaN pattern to stretch its range. Both have only 256 possible values, so a scale factor maps each tensor into the range the format covers.

As a [quantization](https://broadrouter.com/glossary/quantization/) target, FP8 is usually applied as **W8A8**: both weights and activations in 8-bit, with a scale factor per tensor or per block. The paper reports FP8 matching 16-bit training quality on models up to 175B parameters. It also notes FP8 post-training quantization worked on language models “that resisted fixed point int8 quantization”, because the exponent copes better with outliers.

FP8 also shows up inside other formats. DeepSeek’s card for DeepSeek V4.1 Flash stores its main [KV cache](https://broadrouter.com/glossary/kv-cache/) in 4-bit floats with “one E4M3 scale per 16 channels”.

## Why it matters

vLLM’s docs state that FP8 gives “a 2x reduction in model memory requirements and up to a 1.6x improvement in throughput with minimal impact on accuracy.” For a provider, that means a model fits on fewer GPUs, or the same GPUs hold more requests at once, which raises [throughput](https://broadrouter.com/glossary/throughput/) and lowers cost per token.

For an API user, FP8 is the mild end of quantization: vLLM reports minimal accuracy impact, though you should still test the tasks you rely on. It is a less aggressive step than [INT4](https://broadrouter.com/glossary/int4/), which saves more memory but carries more risk on hard tasks.

## On BroadNet Inference

BroadNet’s [model catalogue](https://www.broadnet.ai/inference/models) says its production Gemma 4 26B A4B build stores the KV cache in FP8, with [INT4](https://broadrouter.com/glossary/int4/) AWQ weights. That is BroadNet’s own description; the API docs do not state the precision of the other models. If an exact number format matters for an audit or a regulated workflow, ask BroadNet directly. For what does affect your bill, see the per-token prices on the [pricing page](https://broadrouter.com/pricing/).

## Questions

### Is FP8 the same as INT8?

No. Both use 8 bits, but FP8 spends some bits on an exponent, so it covers a wider range of magnitudes with less precision between them. INT8 spaces its 256 levels evenly and depends more on good scale factors.

### Which GPUs support FP8?

vLLM's docs list FP8 compute on NVIDIA GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell) and AMD MI300x. Older Turing and Ampere cards can load FP8 weights but compute in 16-bit.

### How is FP8 different from INT4?

FP8 uses 8 bits with an exponent and is usually applied to weights and activations. INT4 uses 16 evenly spaced integer levels and is usually applied to weights only. INT4 saves more memory; FP8 carries less accuracy risk.

## See also

-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [INT4](https://broadrouter.com/glossary/int4/) (term)
-   [KV cache](https://broadrouter.com/glossary/kv-cache/) (term)
-   [Throughput](https://broadrouter.com/glossary/throughput/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-fp8&utm_content=page-cta) ·

**Sources**

-   [BroadNet model catalogue (Gemma 4 26B A4B, FP8 KV cache)](https://www.broadnet.ai/inference/models)
-   [Micikevicius et al., FP8 Formats for Deep Learning, arXiv 2209.05433](https://arxiv.org/abs/2209.05433)
-   [vLLM docs, FP8 W8A8 quantization](https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8/)
-   [NVIDIA TensorRT-LLM, numerical precision](https://nvidia.github.io/TensorRT-LLM/reference/precision.html)
-   [DeepSeek, DeepSeek-V4.1-Flash model card (E4M3 scales in its FP4 KV cache)](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
