Weight Decay

Weight Decay is an L2 regularization technique that penalizes large weight values by adding a term proportional to the squared magnitude of the weights to the loss function. The regularized loss becomes L_total = L_task + λ × Σ(w_i²), where λ is the weight decay coefficient. This encourages the optimizer to keep weights small, producing simpler models that are less likely to overfit to training data noise.

By penalizing large weights, weight decay prevents any single weight from dominating the model's predictions, encouraging the model to distribute its reliance across many features rather than depending heavily on a few. This produces smoother decision boundaries and more robust generalization. In practice, weight decay is often implemented not by modifying the loss function but by directly shrinking weights at each update step: w_new = w_old × (1 − lr × λ) − lr × gradient. This is the implementation used by the AdamW optimizer, which decouples weight decay from the gradient-based update.

The distinction between L2 regularization and weight decay matters for adaptive optimizers like Adam. In Adam, coupling weight decay to the gradient (as in L2 regularization) causes it to be scaled by the adaptive per-parameter learning rates, leading to uneven regularization. AdamW (Adam with decoupled Weight decay) addresses this by applying weight decay directly to the weights, independent of the gradient and adaptive learning rates. This decoupled form is the standard for training modern language models.