Intermediate · Safety

Prompt Injection

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An attack where untrusted input (a webpage, email, tool output) contains instructions that hijack an LLM's behavior in an agentic context.

Technical Definition

An attack where untrusted input (a webpage, email, tool output) contains instructions that hijack an LLM's behavior in an agentic context.

How it works

Coined by Simon Willison (2022), prompt injection is the LLM analogue of SQL injection. Direct injection comes from the user; indirect injection arrives through tools — a malicious webpage telling a browsing agent to exfiltrate credentials. Mitigations include privilege separation, structured tool schemas, content-source labeling, and constrained decoding, but the problem is considered unsolved at the model level.

Related Concepts

• AI Safety & Alignment — The field ensuring AI systems behave as intended, remain under human control, and avoid unintended harm.
• Red Teaming — Systematically probing an AI system for failures, exploits, and harmful behavior before deployment.
• Agentic AI — AI systems designed to independently plan, execute, and adapt actions to achieve a given goal, often involving multiple steps and external tools.
• Jailbreak — Adversarial prompts that bypass an aligned model's safety training, eliciting outputs the model would normally refuse.