Intermediate · Generative AI

Voice Cloning

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Synthesizing speech in a target speaker's voice from a short reference sample, often just a few seconds long.

Technical Definition

Synthesizing speech in a target speaker's voice from a short reference sample, often just a few seconds long.

How it works

Modern systems like ElevenLabs, OpenVoice, XTTS-v2, and Tortoise use a speaker encoder to produce a voice embedding from reference audio, which conditions a neural TTS decoder (often a diffusion or autoregressive Transformer over neural audio codec tokens). Quality now reaches near-indistinguishable from real speakers, fueling both creative uses and serious deepfake/fraud concerns.

Related Concepts

• Generative AI — An AI field focused on creating models that can generate novel and complex content, such as text, images, audio, and video, that is coherent and original.
• Deepfake — Synthetic media in which a person in an existing image or video is replaced with someone else's likeness.
• Whisper — OpenAI's open-source speech recognition model trained on 680k hours of multilingual audio, handling transcription, translation, and language ID.