LoRA (Low-Rank Adaptation)

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that adapts a pre-trained model to a new task by injecting trainable low-rank matrices into each layer while keeping the original model weights frozen. The key insight is that the weight updates during fine-tuning often reside in a low-dimensional subspace, so the update matrix ΔW can be approximated as the product of two smaller matrices: ΔW = B × A, where A is a d×r matrix and B is an r×d matrix, with the rank r being much smaller than the original dimension d.

By constraining the update to a low-rank subspace, LoRA reduces the number of trainable parameters by orders of magnitude — often to less than 1% of the original model's parameters. This dramatically lowers memory requirements for optimizer states and gradients, enabling fine-tuning of large models on consumer-grade GPUs. The original frozen weights can be merged with the LoRA adapters for efficient inference, or kept separate to enable hot-swapping between multiple tasks.

LoRA has become the de facto standard for parameter-efficient fine-tuning of large language models. It supports multi-tenant serving, where a single base model serves multiple tasks by loading different lightweight LoRA adapters. The rank r is the primary hyperparameter, with typical values ranging from 4 to 64; higher ranks provide more expressivity at the cost of more parameters.