Latent Space

Latent Space (also called latent space or embedding space) is a learned, typically lower-dimensional continuous representation that captures the essential structure of complex data. Rather than operating on raw, high-dimensional data (e.g., pixels, audio samples, text tokens), models project data into a latent space where semantically meaningful variations are represented as directions and magnitudes. Points close together in latent space correspond to similar data points, and traversing the space produces smooth interpolations.

Latent spaces are central to many model architectures. Autoencoders learn to compress data into a latent code and reconstruct it, with the latent space capturing the data's intrinsic structure. Variational autoencoders (VAEs) impose a probabilistic structure on the latent space, enabling generation of new samples by sampling from the prior. Diffusion and flow-matching models operate in latent space to generate images, audio, and other data. World models represent environment states in latent space for efficient prediction and planning.

The dimensionality and structure of the latent space are critical design choices. Too few dimensions lose information (underfitting); too many fail to compress meaningfully and may overfit. Techniques like disentangled representations aim to make individual latent dimensions correspond to interpretable, independent factors of variation (e.g., pose, color, scale). Latent space arithmetic — combining and manipulating latent vectors — enables applications like image editing, style transfer, and controlled generation.