Intermediate · NLP

Encoder

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A model component that compresses input data into a dense, information-rich representation.

Technical Definition

A model component that compresses input data into a dense, information-rich representation.

How it works

In Transformers, the encoder stack reads the input sequence and produces contextual embeddings used by the decoder or downstream tasks. BERT is encoder-only. In autoencoders, the encoder maps inputs to a low-dimensional latent code.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.
• Decoder — A model component that generates output sequences, typically one token at a time.