Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Overfitting
A phenomenon where a model learns to memorize training data patterns — including noise — rather than generalizable patterns, degrading performance on unseen data.
Overfitting is a phenomenon in which a model learns to fit the training data too closely, capturing noise and idiosyncratic patterns that do not generalize to unseen data. An overfit model achieves high accuracy on the training set but poor accuracy on validation and test sets, indicating that it has memorized specific examples rather than learned the underlying data distribution. Overfitting is one of the most common failure modes in machine learning.
Overfitting typically occurs when a model has too much capacity (too many parameters) relative to the amount of training data, when training continues for too many epochs, or when the training data is not representative of the deployment distribution. The model effectively 'memorizes' the training examples, including their noise and outliers, rather than learning generalizable patterns. Symptoms include a training loss that continues to decrease while validation loss starts to increase — the point where validation loss begins to rise is the onset of overfitting.
Common countermeasures include regularization techniques (L1/L2 weight decay, dropout), early stopping (halting training when validation performance degrades), data augmentation (increasing effective training set size), reducing model capacity, and cross-validation (to detect overfitting across different data splits). Monitoring the gap between training and validation metrics throughout training is essential for detecting and addressing overfitting before it significantly degrades model quality.
- ■Model memorizes training data (including noise) instead of learning generalizable patterns
- ■Symptom: training loss decreases while validation loss increases
- ■Caused by excess model capacity, too many epochs, or unrepresentative data
- ■Countermeasures: regularization, early stopping, data augmentation, capacity reduction
- ■Monitor train-validation metric gap to detect onset
A 100M-parameter model trained on only 1,000 examples achieves 99% training accuracy but 65% test accuracy — classic overfitting that requires more data or regularization.