Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Attention Mechanism
A neural network component that computes weighted relationships between all positions in a sequence, enabling models to focus on relevant context.
The Attention Mechanism is a neural network component that computes weighted relationships between all positions in a sequence, enabling a model to dynamically focus on the most relevant parts of its input when producing each output. Rather than processing a sequence position-by-position with fixed-size context (as in RNNs), attention allows every position to directly attend to every other position, capturing long-range dependencies without information degradation.
The most common form is scaled dot-product attention. For each query vector, the mechanism computes dot products with all key vectors, scales by 1/√d_k, applies a softmax to obtain attention weights, and then computes a weighted sum of value vectors. Multi-head attention extends this by running multiple attention operations in parallel with different learned projections, allowing the model to attend to different types of relationships simultaneously (e.g., one head might focus on syntactic dependencies while another captures semantic similarity).
Attention is the foundational component of the Transformer architecture, which replaced recurrence and convolution with self-attention as the primary mechanism for sequence processing. Its quadratic computational complexity with respect to sequence length (O(n²)) is a known limitation, motivating efficient attention variants such as sparse attention, linear attention, and flash attention that reduce this cost for long sequences.
- ■Computes weighted relationships between all positions in a sequence
- ■Scaled dot-product: softmax(QK^T / √d_k) × V
- ■Multi-head attention captures different relationship types in parallel
- ■Foundational component of the Transformer architecture
- ■O(n²) complexity motivates efficient variants (sparse, linear, flash attention)
In machine translation, attention allows the decoder to focus on the most relevant source-language words when generating each target-language word, regardless of their position in the input.