Intermediate · Systems

GGUF

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A single-file binary format from the llama.cpp project for distributing quantized LLM weights together with tokenizer and metadata.

Technical Definition

A single-file binary format from the llama.cpp project for distributing quantized LLM weights together with tokenizer and metadata.

How it works

GGUF replaced the older GGML format in 2023. A GGUF file packages weights at quantization levels like Q4_K_M, Q5_K_S, or Q8_0 alongside the tokenizer, chat template, and architecture hyperparameters, so a model can be loaded by llama.cpp, Ollama, or LM Studio with no extra config. It is the dominant local-LLM distribution format on Hugging Face.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.
• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.