Advanced · Generative AI

ControlNet

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A neural add-on that lets diffusion models be steered by extra conditioning images such as edges, depth maps, poses, or scribbles.

Technical Definition

A neural add-on that lets diffusion models be steered by extra conditioning images such as edges, depth maps, poses, or scribbles.

How it works

ControlNet (Zhang et al., 2023) clones the encoder of a frozen Stable Diffusion model and adds zero-initialized convolutions that inject a control signal at each level. Training only the trainable copy preserves the base model's quality while learning to follow Canny edges, OpenPose skeletons, depth maps, segmentation masks, and more. It became the foundation of structured image generation workflows in ComfyUI and Automatic1111.

Related Concepts

• Diffusion Model — A generative model that learns to create data by reversing a gradual noising process, producing high-quality images and audio.
• LoRA (Low-Rank Adaptation) — A parameter-efficient fine-tuning method injecting small trainable matrices into frozen pre-trained layers.
• Text-to-Image — A generative task that produces images from natural-language descriptions.