Intermediate · Systems

Local Decision Models

Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Compact on-device models — like Google's Gemma 4 and Xiaomi's MiMo-V2.6 — that act as traffic controllers, solving tasks locally and calling cloud APIs only for frontier-level reasoning.

Technical Definition

Compact on-device models — like Google's Gemma 4 and Xiaomi's MiMo-V2.6 — that act as traffic controllers, solving tasks locally and calling cloud APIs only for frontier-level reasoning.

How it works

Local decision models invert the classic architecture: instead of a big cloud model deciding everything, a small resident model triages each request. Easy tasks — classification, extraction, rewriting, routing — are handled entirely on-device; only requests that genuinely need frontier reasoning are escalated to a cloud endpoint, with the local model choosing which endpoint and shaping the prompt. This cuts cost and latency, keeps private data local by default, and degrades gracefully offline. The design risk is routing error: a local model that under-escalates silently returns weaker answers, so escalation thresholds are tuned on evals, not vibes.

Visual Explanation (flowchart)

Request → Local decision model → [solve on-device] or [escalate → cloud frontier model] → response

Related Concepts

• Personal AI Routers (PAIR) — NVIDIA technology introduced at IFA 2026 that lets a local PC dynamically split heavy AI inference across multiple consumer-grade GPUs in the home.
• On-device AI — AI models that run directly on edge devices (e.g., smartphones, IoT) rather than relying on cloud servers for inference.
• Small Language Models (SLMs) — Compact language models that run efficiently on-premise, on edge devices, or on a laptop.