Intermediate · NLP

Whisper

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. OpenAI's open-source speech recognition model trained on 680k hours of multilingual audio, handling transcription, translation, and language ID.

Technical Definition

OpenAI's open-source speech recognition model trained on 680k hours of multilingual audio, handling transcription, translation, and language ID.

How it works

Whisper (Radford et al., 2022) is an encoder-decoder Transformer that ingests 30-second log-Mel spectrograms and emits text tokens with special tokens for task and language. Trained on weakly supervised web data, it is robust to accents, noise, and 99 languages without fine-tuning. Variants like whisper.cpp, faster-whisper, and Distil-Whisper enable real-time and edge deployment.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Speech Recognition — The ability of a machine to identify spoken words and convert them into human-readable text.
• Multimodal Model — A model that processes and/or generates more than one modality — for example text, images, audio, and video — within a single architecture.