Advanced · Systems

AWQ (Activation-aware Weight Quantization)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A 4-bit LLM quantization method that protects the most salient weight channels identified by activation magnitudes, beating GPTQ on many models.

Technical Definition

A 4-bit LLM quantization method that protects the most salient weight channels identified by activation magnitudes, beating GPTQ on many models.

How it works

AWQ observes that a small fraction of weights, those multiplied by large-magnitude activations, dominate output error after quantization. It scales these salient channels up before per-group quantization and back down afterward, achieving better accuracy at 4 bits than GPTQ with no calibration-set overfitting. It is the default quantization scheme in vLLM and TensorRT-LLM serving paths.

Related Concepts

• Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.
• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.
• GGUF — A single-file binary format from the llama.cpp project for distributing quantized LLM weights together with tokenizer and metadata.
• GPTQ — A one-shot post-training quantization method that compresses LLM weights to 3–4 bits using approximate second-order information.