Intermediate · Systems

Model Serving

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Hosting a trained model behind an interface so applications can request predictions in real time.

Technical Definition

Hosting a trained model behind an interface so applications can request predictions in real time.

How it works

Serving systems load model weights, batch incoming requests, run inference on GPUs/CPUs, and return responses. Tools like Triton, vLLM, TGI, and TorchServe handle scaling, batching, and quantization. Latency, throughput, and cost-per-request are the core trade-offs.

Related Concepts

• Deployment — The process of moving a trained model into a production environment where it serves real users.
• Inference Endpoint — A network address that accepts inputs and returns model predictions.
• Latency — The time between sending a request and receiving the first (or final) response.