Beginner · Systems

Inference Endpoint

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A network address that accepts inputs and returns model predictions.

Technical Definition

A network address that accepts inputs and returns model predictions.

How it works

Typically an HTTPS URL that takes JSON, runs the model, and returns JSON. Endpoints can be hosted (OpenAI, Anthropic, Bedrock), self-managed (vLLM on a GPU box), or on-device. Authentication, rate limits, and cost monitoring are critical for production endpoints.

Related Concepts

• Model Serving — Hosting a trained model behind an interface so applications can request predictions in real time.
• Deployment — The process of moving a trained model into a production environment where it serves real users.
• API — An application programming interface — the contract by which software components communicate.