Advanced · Neural Networks

Quantization

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.

Technical Definition

Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.

How it works

Quantization maps high-precision values to fewer discrete levels. INT8 halves memory vs FP16 with negligible quality loss. INT4 (GPTQ, AWQ) enables running 70B models on consumer GPUs. Methods include post-training quantization and quantization-aware training.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Knowledge Distillation — Compressing a large teacher model into a smaller student model by training the student to mimic the teacher's outputs.