Learning Rate

The Learning Rate is a hyperparameter that controls the magnitude of weight updates during training. At each step, the model's weights are adjusted in the direction of the negative gradient, scaled by the learning rate: w_new = w_old − learning_rate × gradient. The learning rate determines how large each step is — too high and the model may overshoot optimal solutions or diverge; too low and training becomes excessively slow or may get stuck in suboptimal local minima.

Selecting an appropriate learning rate is one of the most critical decisions in training neural networks. A common practice is to use learning rate schedules that start with a higher rate for rapid initial progress, then decay over time for fine-grained convergence. Popular schedules include cosine annealing, step decay, and warmup (where the learning rate linearly increases from a small value at the start of training before following a decay schedule). Warmup is particularly important for Transformer training to stabilize early iterations.

The learning rate is closely coupled with the batch size — larger batch sizes generally require larger learning rates (the linear scaling rule). Modern optimizers like Adam and AdamW adapt the effective learning rate per parameter based on gradient history, making them more robust to the choice of global learning rate than plain SGD. However, the base learning rate still needs careful tuning, and techniques like learning rate warmup, clipping, and cycling are commonly employed.