Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning (SFT) is the process of fine-tuning a pre-trained model on a dataset of labeled input-output pairs to teach it a specific task or behavior. Unlike unsupervised pre-training, which learns general patterns from unlabelled text, SFT uses explicit supervision: for each input (e.g., a user question), the model is trained to produce the target output (e.g., the correct answer). This is typically done with standard cross-entropy loss on the target tokens.

SFT is the foundational step in aligning language models for instruction-following. Before SFT, a base language model simply continues text given a prompt; after SFT, it learns to respond as an assistant. The quality and diversity of the SFT dataset heavily influence the model's behavior — high-quality demonstrations covering a wide range of tasks produce a model that generalizes well to new instructions. SFT is also the first stage in RLHF, providing the initial policy that is subsequently refined through preference optimization.

A common practice in SFT for language models is to mask the loss on the prompt tokens and compute loss only on the response tokens, ensuring the model learns to generate good responses rather than memorize prompts. SFT can be performed with full parameter updates or with parameter-efficient methods like LoRA, depending on the available compute and the desired trade-off between adaptability and preserving the base model's knowledge.