Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Quantization
A technique that reduces the precision of model weights and activations (e.g., from FP32 to INT8) to decrease memory usage and accelerate inference.
Quantization is a model optimization technique that reduces the numerical precision of a model's weights and activations, typically from 32-bit floating-point (FP32) to lower-precision formats such as 16-bit float (FP16/BF16), 8-bit integer (INT8), or even 4-bit representations. By representing each value with fewer bits, quantization reduces the model's memory footprint, lowers memory bandwidth requirements, and enables faster inference on hardware that supports low-precision arithmetic.
Quantization can be applied post-training (Post-Training Quantization, PTQ) or during training (Quantization-Aware Training, QAT). PTQ is simpler — it calibrates the model on a small sample of data to determine the optimal scaling factors for each layer and then quantizes the weights. QAT simulates quantization during training, allowing the model to adapt to the reduced precision and typically achieving higher accuracy. The trade-off is between the simplicity of PTQ and the accuracy retention of QAT.
Modern large language model inference heavily relies on quantization techniques such as GPTQ, AWQ, and bitsandbytes, which can quantize models to 4-bit or 8-bit with minimal accuracy loss. This enables running models that would otherwise exceed available memory — for example, a 7B-parameter model in FP32 requires ~28GB of memory, but in 4-bit quantization it fits in ~4GB, making it runnable on consumer GPUs.
- ■Reduces precision from FP32 to FP16, INT8, or even 4-bit
- ■Decreases memory footprint, bandwidth, and inference latency
- ■Post-Training Quantization (PTQ) is simple; QAT achieves higher accuracy
- ■GPTQ, AWQ, and bitsandbytes enable 4-bit LLM inference with minimal loss
- ■A 7B model drops from ~28GB (FP32) to ~4GB (4-bit)
Quantizing a 13B-parameter model to 8-bit using GPTQ so it can be served on a single consumer GPU with 12GB of VRAM instead of requiring a data-center GPU.