Intermediate · Generative AI

Multimodal Model

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A model that processes and/or generates more than one modality — for example text, images, audio, and video — within a single architecture.

Technical Definition

A model that processes and/or generates more than one modality — for example text, images, audio, and video — within a single architecture.

How it works

Modern multimodal LLMs (GPT-4o, Gemini, Claude 3.5, Qwen-VL) typically encode non-text inputs into tokens that share the same Transformer backbone as text. Training mixes paired data (image-caption, audio-transcript) with interleaved web data. Native multimodal models (Gemini, GPT-4o) train from scratch on all modalities; bolt-on variants graft a vision encoder onto an existing LLM.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• CLIP — A multi-modal model connecting images and text in a shared embedding space for zero-shot visual classification.