Regularization

Regularization refers to a set of techniques designed to prevent overfitting by constraining or penalizing model complexity, encouraging the model to learn simpler, more generalizable patterns. Without regularization, models with sufficient capacity can fit the training data perfectly — including its noise — but fail to generalize. Regularization introduces a bias that trades a small amount of training performance for significantly improved performance on unseen data.

The most common forms of regularization add penalty terms to the loss function. L1 regularization (Lasso) adds the absolute value of weights, encouraging sparsity (many weights become exactly zero). L2 regularization (Ridge, weight decay) adds the squared magnitude of weights, encouraging small but non-zero weights. Other regularization techniques include dropout (randomly deactivating neurons during training), early stopping (halting training before overfitting), data augmentation (artificially expanding the training set), and label smoothing (softening target distributions).

The strength of regularization is controlled by a hyperparameter (often denoted λ or α) that balances the task loss against the regularization penalty. Too little regularization allows overfitting; too much causes underfitting (the model is too constrained to learn the data's patterns). The optimal regularization strength is typically found through hyperparameter tuning using validation data. For large language models, regularization strategies often combine weight decay, dropout, and data-level techniques like instruction diversity.