Beginner · Fundamentals

Inference

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Using a trained model to make predictions on new data — the deployment phase of machine learning.

Technical Definition

Using a trained model to make predictions on new data — the deployment phase of machine learning.

How it works

Inference is the forward-pass computation on new inputs. Unlike training, it doesn't involve backpropagation. Optimization techniques include quantization, pruning, batching, KV-caching, and hardware acceleration. Latency, throughput, and cost are key metrics.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.