Intermediate · Systems
Batch inference
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. Processing multiple data examples at once to generate predictions, leveraging parallelism for efficiency.
Technical Definition
Processing multiple data examples at once to generate predictions, leveraging parallelism for efficiency.
How it works
Batch inference involves making predictions on a collection of data examples simultaneously, often processed in subsets called batches. This approach is highly efficient, especially on hardware with parallel processing capabilities like GPUs. By processing data in batches, systems can achieve higher throughput and lower latency compared to processing examples one by one.
Related Concepts
- Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.
- Latency — The time between sending a request and receiving the first (or final) response.
- Throughput — The number of requests or tokens a system can process per unit of time.