Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Mixed Precision Training
A training technique that uses lower-precision formats (FP16/BF16) for computations while maintaining FP32 master weights for numerical stability.
Mixed Precision Training is a technique that accelerates training and reduces memory usage by performing computations in lower-precision formats (typically FP16 or BF16) while maintaining a master copy of the weights in FP32 for numerical stability. The key insight is that most neural network computations do not require the full dynamic range of FP32, but gradient values can become small enough to underflow in FP16, leading to training instability.
The standard mixed-precision approach uses three components: a master copy of weights in FP32, forward and backward computations in FP16/BF16, and a loss scaling factor that multiplies the loss before backpropagation to prevent gradient underflow. The scaled gradients are computed in FP16, then unscaled and used to update the FP32 master weights. BF16 (bfloat16) has become increasingly popular because it has the same dynamic range as FP32 (8 exponent bits) with reduced precision (7 mantissa bits), eliminating the need for loss scaling in many cases.
Mixed precision can provide 2–3× speedup on modern GPUs with tensor cores (which are optimized for FP16/BF16 matrix multiplication) and reduce memory usage by approximately 50% for weights and activations. This enables training larger models or using larger batch sizes within the same memory budget. The technique is now standard practice in training large language models and is supported natively by frameworks like PyTorch (AMP) and TensorFlow.
- ■Computes in FP16/BF16; maintains FP32 master weights for stability
- ■Loss scaling prevents gradient underflow in FP16
- ■BF16 eliminates loss scaling need (same exponent range as FP32)
- ■Provides 2–3× speedup on tensor-core GPUs; ~50% memory reduction
- ■Standard practice for LLM training; supported by PyTorch AMP and TensorFlow
Training a 7B-parameter model with BF16 mixed precision on an A100 GPU, achieving 2× faster training while maintaining numerical stability without loss scaling.