FP8 Mixed-Precision Training

FP8 Mixed-Precision Training

· 210 words · 2 minutes reading time

FP8 Mixed-Precision Training (DeepSeek-V3 recipe)

TL;DR

  • DeepSeek-V3 is the first extremely large model (671B/37B-active) to validate end-to-end FP8 pre-training at scale.
  • The team uses E4M3 on all tensors (not the NVIDIA hybrid E4M3/E5M2), enabled by fine-grained 1×128 activation tiles and 128×128 weight blocks.
  • Optimizer state uses BF16 instead of FP32, saving memory; accumulation uses promoted FP32 (NVIDIA CUTLASS pattern).

Recipe (verbatim from DeepSeek-V3, §3.3)

"we adopt the E4M3 format on all tensors for higher precision. We attribute the feasibility of this approach to our fine-grained quantization strategy, i.e., tile and block-wise scaling." "we calculate the maximum absolute value online for each 1x128 activation tile or 128x128 weight block."

Why it works

Tile/block-wise scaling shares the exponent across small element groups, capturing dynamic range that tensor-wide scaling can't. Outliers in any single element don't blow up the global scale.

FAQ

Q: Is FP4 next? Yes — Blackwell supports it; DeepSeek's hardware companion paper (arXiv:2505.09343) discusses low-precision directions. Q: Does FP8 hurt loss? DeepSeek-V3 reports <0.25% loss perturbation vs BF16 — negligible at scale.

References

  • arXiv:2412.19437 (V3); arXiv:2505.09343 (V3/R1 hardware insights); Colfax: "DeepSeek-R1 and FP8 Mixed-Precision Training"

Series

LLM Pre-Training 2026

LLM Pre-Training in 2026: The Frontier in Numbers

The state of frontier LLM pre-training in 2026 — token counts, parameter counts, cluster sizes, costs, and what it all means for CTOs and ML leads.

2026 大模型预训练：前沿数据全景

Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

Where 2026 frontier LLMs get their pre-training data — Common Crawl, FineWeb, DCLM, StackV2, multilingual, PDFs — and how Qwen 3, DeepSeek-V3, Llama 3.1 sized their corpora.

Quality Filtering for LLM Pre-Training: FineWeb-Edu, DCLM, Nemotron-CC, Ultra-FineWeb

How frontier labs filter trillions of web tokens — heuristic, perplexity-based, and classifier-based filtering, with concrete recipes from FineWeb-Edu and DCLM.

Mixture-of-Experts Design: DeepSeek-V3, Qwen 3, Llama 4 Compared

Fine-grained experts, shared experts, auxiliary-loss-free routing — the modern MoE recipe in 2026, with side-by-side comparison of DeepSeek-V3, Qwen 3, Llama 4.

Multi-head Latent Attention (MLA): The KV-Cache Compression Behind DeepSeek-V3

How DeepSeek's Multi-head Latent Attention compresses the KV cache via low-rank projections + decoupled RoPE, achieving large memory reductions versus MHA at equal or better quality.

Frontier AI Training Hardware in 2026: H100, H200, GB200 NVL72, TPU Ironwood, Trainium2/3

Side-by-side specifications of every chip and pod system used at the 2026 LLM frontier, with primary-source numbers from NVIDIA, Google Cloud, and AWS.

Cluster Scale 2026: Colossus, Stargate, Project Rainier

From 16K to 1M-GPU systems — how xAI built Colossus in 122 days, how AWS deployed 500K Trainium2 chips for Anthropic, and what OpenAI's $500B Stargate commitment actually covers.

FP8 Mixed-Precision Training

Optimizers in 2026: AdamW, Muon, Shampoo/SOAP

Training Cost Economics 2026

Energy, Power & Capex