Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Backpropagation
The algorithm used to compute gradients of the loss with respect to all model weights by applying the chain rule backward through the network.
Backpropagation (backprop) is the algorithm used to compute the gradients of a loss function with respect to all weights in a neural network, enabling weight updates via gradient descent. It works by applying the chain rule of calculus backward through the network: starting from the loss at the output, it computes gradients layer-by-layer from the last layer to the first, reusing intermediate computations from the forward pass to efficiently calculate all gradients in a single backward pass.
The forward pass computes the model's output and stores intermediate activations. The backward pass then propagates the gradient of the loss backward through each layer: for each layer, it computes the gradient of the loss with respect to the layer's inputs (which becomes the input gradient for the previous layer) and the gradient with respect to the layer's weights (used for the weight update). This recursive application of the chain rule makes computing all gradients require only about twice the computation of a single forward pass, regardless of the number of parameters.
Backpropagation is implemented automatically by deep learning frameworks (PyTorch autograd, TensorFlow GradientTape, JAX grad) through computational graphs. Each operation records its gradient computation, and the framework traverses the graph backward to accumulate gradients. Modern implementations handle complex architectures (skip connections, attention, custom layers) transparently. The computed gradients are then used by optimizers (SGD, Adam, AdamW) to update weights, with the learning rate controlling the step size.
- ■Computes gradients of loss w.r.t. all weights via the chain rule, backward
- ■Reuses forward-pass intermediate activations for efficiency
- ■Single backward pass ≈ 2× forward pass cost, regardless of parameter count
- ■Automated by frameworks (PyTorch autograd, TF GradientTape, JAX grad)
- ■Gradients fed to optimizers (SGD, Adam, AdamW) for weight updates
During language model training, backpropagation computes gradients for all transformer layer weights in a single backward pass from the cross-entropy loss.