PEFT (Parameter-Efficient Fine-Tuning)

Parameter-Efficient Fine-Tuning (PEFT) refers to a family of techniques that adapt a pre-trained model to a new task by updating only a small subset of parameters — or a small set of newly added parameters — while keeping the majority of the pre-trained weights frozen. The goal is to achieve performance comparable to full fine-tuning while training only a tiny fraction (often <1%) of the total parameters, dramatically reducing memory, compute, and storage costs.

PEFT methods work on the principle that task-specific adaptation resides in a low-dimensional subspace of the full parameter space. Common approaches include LoRA (which injects low-rank update matrices), adapter modules (which add small bottleneck layers between existing layers), prefix tuning (which prepends learnable tokens to the input), and prompt tuning (which optimizes continuous prompt embeddings). Each method trades off expressivity, parameter count, and inference overhead differently.

PEFT is particularly valuable for multi-tenant scenarios, where a single base model can serve many tasks by loading different lightweight adapters. It also enables fine-tuning of very large models on limited hardware, preserves the base model's general capabilities (reducing catastrophic forgetting), and allows efficient storage and versioning of task-specific adaptations. The trained adapters are typically tiny (megabytes rather than gigabytes), making them easy to distribute and deploy.