Advanced · Systems

Computer Use

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An agent capability where an LLM controls a computer GUI by viewing screenshots and emitting mouse/keyboard actions, automating arbitrary software.

Technical Definition

An agent capability where an LLM controls a computer GUI by viewing screenshots and emitting mouse/keyboard actions, automating arbitrary software.

How it works

Anthropic launched Computer Use with Claude 3.5 Sonnet (Oct 2024), followed by OpenAI's Operator and Google's Project Mariner. The model is shown a screenshot, reasons about the next action (click coordinates, type text, scroll), executes it in a sandbox, and re-observes. It enables automation of any app — not just those with APIs — at the cost of latency, fragility, and serious prompt-injection risk.

Related Concepts

• Agentic AI — AI systems designed to independently plan, execute, and adapt actions to achieve a given goal, often involving multiple steps and external tools.
• Tool Use — The ability of an AI model, especially an LLM agent, to autonomously identify, select, and utilize external tools and APIs to accomplish tasks.
• Prompt Injection — An attack where untrusted input (a webpage, email, tool output) contains instructions that hijack an LLM's behavior in an agentic context.