Beginner · Systems

Streaming

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Returning model output incrementally — token by token — as it is generated.

Technical Definition

Returning model output incrementally — token by token — as it is generated.

How it works

Streaming dramatically improves perceived latency for LLM apps; users see the first words within hundreds of milliseconds instead of waiting for the full response. Implemented with server-sent events or WebSockets on top of token-by-token decoding.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Completion — The text a language model generates in response to a prompt.
• Latency — The time between sending a request and receiving the first (or final) response.