Home › Glossary › Systems › Edge AI

Intermediate · Systems

Edge AI

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Running AI models directly on user devices instead of cloud servers.

Technical Definition

Running AI models directly on user devices instead of cloud servers.

How it works

Edge inference (on phones, cars, cameras, IoT) reduces latency, preserves privacy, and works offline. Requires model compression (quantization, distillation, pruning) and runtimes like Core ML, TensorFlow Lite, ONNX Runtime, and llama.cpp.

Related Concepts

  • Knowledge Distillation — Compressing a large teacher model into a smaller student model by training the student to mimic the teacher's outputs.
  • Quantization — Reducing numerical precision of model weights (e.g., 32-bit to 4-bit) to shrink size and speed up inference.
  • Pruning — Removing redundant weights or neurons from a network to reduce size and improve inference speed.