Cross-Entropy Loss

Cross-Entropy Loss is a loss function that measures the difference between a model's predicted probability distribution and the true distribution (typically a one-hot encoded label). For classification tasks, it quantifies how far the model's predicted probabilities are from the correct class. In language modeling, cross-entropy loss is computed at each token position: the model predicts a probability distribution over the vocabulary, and the loss measures how much probability the model assigned to the actual next token.

Mathematically, cross-entropy loss for a single example is L = −Σ y_i log(p_i), where y_i is the true probability (1 for the correct class, 0 otherwise) and p_i is the model's predicted probability for class i. For language models, this simplifies to L = −log(p_correct_token), the negative log-likelihood of the correct next token. Minimizing cross-entropy loss is equivalent to maximum likelihood estimation — the model learns to assign high probability to the correct outputs.

Cross-entropy loss is the standard loss function for training language models and classification networks. It is typically paired with the softmax function, which converts raw logits into a probability distribution. In knowledge distillation, a modified cross-entropy loss (using the teacher's softened distribution with temperature scaling) is used to transfer the teacher's 'dark knowledge.' The gradient of cross-entropy with respect to the logits has a simple form (predicted − true), making it computationally efficient for backpropagation.