Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Gradient Accumulation
A technique that simulates larger batch sizes by accumulating gradients across multiple forward passes before applying a weight update.
Gradient Accumulation is a technique that allows training with an effectively larger batch size than what fits in GPU memory. Instead of performing a weight update after each forward-backward pass, the method accumulates gradients over multiple smaller mini-batches (steps) and only applies the optimizer update after a specified number of accumulation steps. The effective batch size equals the mini-batch size multiplied by the number of accumulation steps.
This is particularly valuable when training large models that consume significant GPU memory. For example, if a GPU can only fit a mini-batch of 4 examples but the desired effective batch size is 32, gradient accumulation over 8 steps achieves the same result: gradients from 8 mini-batches of 4 are summed (or averaged) before the optimizer updates the weights. Mathematically, this produces the same gradient estimate as a single batch of 32, assuming proper gradient scaling.
While gradient accumulation solves the memory constraint, it does not improve training speed — the same number of forward and backward passes must be computed. The trade-off is between memory savings and wall-clock training time. Care must also be taken with gradient averaging (ensuring the loss is divided by the accumulation steps to maintain correct gradient magnitude) and with techniques that depend on batch-level statistics, such as batch normalization, which may behave differently under accumulation.
- ■Accumulates gradients over N mini-batches before applying an update
- ■Effective batch size = mini-batch size × accumulation steps
- ■Solves GPU memory constraints without reducing effective batch size
- ■Does not improve training speed; same number of forward/backward passes
- ■Requires careful gradient scaling and interacts with batch normalization
Training a 13B-parameter model on a single 24GB GPU by using a mini-batch of 1 with 32 gradient accumulation steps, achieving an effective batch size of 32.