Embedding

An Embedding is a learned dense vector representation that maps discrete entities — such as tokens, words, sentences, images, or graph nodes — into a continuous vector space where geometric proximity reflects semantic similarity. In language models, an embedding layer maps each token ID to a learned vector of fixed dimension (e.g., 768, 4096), transforming discrete symbolic input into the continuous numerical representation that neural networks process.

The key property of embedding spaces is that they capture semantic relationships: entities with similar meanings are positioned close together in the space, and directions in the space can encode relationships (e.g., the classic word2vec result that vec('king') − vec('man') + vec('woman') ≈ vec('queen')). This structure emerges naturally from training objectives — language model embeddings learn from context prediction, while contrastive embeddings (like sentence transformers) learn from positive/negative pairs.

Embeddings are used throughout modern AI systems: token embeddings at the input of language models, positional embeddings to encode token order, sentence/document embeddings for semantic search and RAG, image embeddings for visual similarity, and multimodal embeddings that align text and image spaces (as in CLIP). The quality of embeddings directly impacts downstream task performance, and embedding models are often evaluated on retrieval benchmarks, analogy tasks, and clustering quality.