Transformer Architecture

The Transformer is a neural network architecture introduced by Vaswani et al. (2017) that processes sequences using self-attention mechanisms rather than recurrence (RNNs) or convolution (CNNs). By replacing sequential processing with parallel attention operations, Transformers can process all positions in a sequence simultaneously, enabling significantly faster training and better utilization of modern parallel hardware (GPUs, TPUs). This architectural shift enabled the scaling that produced modern large language models.

A standard Transformer consists of an encoder and a decoder, each built from stacked layers. Each layer contains a multi-head self-attention sub-layer and a position-wise feed-forward network. Layer normalization and residual connections are applied around each sub-layer. Since attention is permutation-invariant, positional encodings (sinusoidal or learned) are added to the input embeddings to inject information about token order. Decoder layers additionally include cross-attention to attend to the encoder's output.

Modern language models use variations of the Transformer: encoder-only models (BERT) for understanding tasks, decoder-only models (GPT, Llama) for generation, and encoder-decoder models (T5, BART) for sequence-to-sequence tasks. The decoder-only architecture has become dominant for large language models due to its simplicity and effectiveness in autoregressive generation. Key innovations beyond the original Transformer include rotary positional embeddings (RoPE), grouped-query attention, and mixture-of-experts layers.