Advanced · Systems

GPTQ

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A one-shot post-training quantization method that compresses LLM weights to 3–4 bits using approximate second-order information.

Technical Definition

A one-shot post-training quantization method that compresses LLM weights to 3–4 bits using approximate second-order information.

How it works

GPTQ (Frantar et al., 2022) quantizes weights one column at a time, using the inverse Hessian of the layer's calibration-set loss to update remaining columns and minimize output error. It produces 4-bit models that are nearly lossless and an order of magnitude smaller than FP16. AutoGPTQ and ExLlama provide widely used inference kernels.

Related Concepts

• Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.
• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.
• GGUF — A single-file binary format from the llama.cpp project for distributing quantized LLM weights together with tokenizer and metadata.
• AWQ (Activation-aware Weight Quantization) — A 4-bit LLM quantization method that protects the most salient weight channels identified by activation magnitudes, beating GPTQ on many models.