Batch Size

Batch Size is the number of training examples processed together in a single forward and backward pass of a neural network. During each training step, the model computes predictions for all examples in the batch, calculates the average loss, and updates weights based on the averaged gradient. The batch size is one of the most important training hyperparameters, directly affecting memory usage, gradient estimation quality, and overall training throughput.

Larger batch sizes produce gradient estimates that are closer to the true gradient (lower variance), leading to more stable and consistent weight updates. They also improve hardware utilization by keeping GPUs fully occupied with parallel computation. However, very large batch sizes can lead to poorer generalization (the 'generalization gap'), require more memory, and may need learning rate scaling. Small batch sizes introduce more gradient noise, which can act as a form of regularization and improve generalization, but may underutilize hardware and slow training.

When GPU memory is insufficient for a desired batch size, gradient accumulation can be used to simulate a larger effective batch size by accumulating gradients across multiple smaller mini-batches before applying a weight update. The choice of batch size is closely tied to the learning rate — the linear scaling rule suggests scaling the learning rate proportionally with batch size, though this relationship breaks down at very large batch sizes.