Intermediate · NLP

Sequence-to-Sequence (Seq2Seq)

Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An architecture that maps an input sequence to an output sequence of potentially different length.

Technical Definition

An architecture that maps an input sequence to an output sequence of potentially different length.

How it works

Seq2Seq models have an encoder processing input into a context representation and a decoder generating output token by token. Originally built with RNNs, modern versions use Transformer encoder-decoder architectures. Applications include machine translation, summarization, and code generation.

Visual Explanation (flowchart)

Input Sequence → Encoder → Context Vector → Decoder → Output Sequence (token by token)

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Recurrent Neural Network (RNN) — A neural network with loops that maintain hidden state, designed to process sequential data like text and time series.