Intermediate · Generative AI

Multimodal LLM

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An LLM extended to process and understand multiple types of data, such as images, audio, and text, beyond just language.

Technical Definition

An LLM extended to process and understand multiple types of data, such as images, audio, and text, beyond just language.

How it works

Multimodal LLMs build upon the linguistic capabilities of traditional LLMs by integrating other sensory modalities. They can, for instance, analyze an image and then generate a descriptive caption, or answer questions about content shown in a video. This enables a more holistic understanding of information and more versatile applications.

Related Concepts

• Computer vision — Computer vision enables computers to interpret and understand information from digital images and videos, automating tasks typically done by human sight.
• Multimodal AI — AI systems capable of processing and understanding information from multiple data types, such as text, images, audio, and video, simultaneously.