Dropout

Dropout is a regularization technique that randomly deactivates (sets to zero) a fraction of neurons during each training step, forcing the network to learn redundant, robust representations rather than relying on specific neurons. The dropout rate p (typically 0.1–0.5) determines the fraction of activations zeroed out. During inference, all neurons are active and their outputs are scaled by (1−p) to compensate for the training-time dropout.

By randomly dropping neurons, dropout prevents co-adaptation — a phenomenon where neurons become overly dependent on specific other neurons and fail to function independently. This effectively trains an ensemble of sub-networks (all possible dropout masks) within a single model, and the full model at inference can be viewed as an approximation of averaging over this ensemble. Dropout is conceptually similar to bagging in traditional machine learning.

Dropout can be applied to different parts of the network: hidden layer activations (standard dropout), attention weights (attention dropout), or embeddings. While effective, dropout interacts with other techniques — it is generally not combined with batch normalization in the same layer, and its effect must be accounted for when using techniques like weight decay. For large language models, relatively low dropout rates (0.0–0.1) are typically used, as these models benefit more from the data and scale than from aggressive regularization.