Prompt injection is an input-validation problem.

02Direct and indirect injection

Direct injection comes from the person using your system. It matters, but they can usually only affect their own session. Indirect injection is the bigger risk: instructions hidden in content the system reads on someone’s behalf, such as emails, documents, web pages, tickets or another tool’s output. The person harmed may not be the person who planted the text.

Any system that combines three things is exposed: it reads untrusted content, it can see private data, and it can take actions or send data somewhere. Simon Willison calls this combination the lethal trifecta. Agents usually have all three.

03Why “ignore malicious instructions” doesn’t work

The tempting fix is another instruction: tell the model to disregard any instructions it finds in documents. It helps a little, and it fails exactly when it matters. Models have no reliable boundary between instructions and data; everything in the context is text that shapes the next token. Delimiters and “the following is untrusted” labels raise the bar, but an attacker only needs one phrasing that works.

Model-based injection detectors are useful as one layer, and worth running on high-risk inputs. Just don’t build a design that’s only safe if the detector is perfect.

04Limit the blast radius

Since you can’t guarantee the model won’t be manipulated, control what a manipulated model can do:

• Least privilege for tools. Give each agent the narrowest set of tools its task needs, scoped to the current user’s permissions rather than a service account’s.
• Separate reading from acting. A step that summarizes untrusted documents shouldn’t also hold the tools that send email or move money.
• Confirm irreversible actions. Payments, deletions, outbound messages and permission changes need explicit approval from a person who can see exactly what will happen.
• Allowlist destinations. If the system can fetch URLs or send messages, restrict where to.

type Risk = "read" | "write" | "external";

interface Tool {
  name: string;
  risk: Risk;
  run(args: unknown): Promise<unknown>;
}

interface Session {
  allowedTools: Set<string>;                   // scoped to this user and this task
  sawUntrustedContent: boolean;                // set once outside text enters the context
  confirm(summary: string): Promise<boolean>;  // asks the person, showing exactly what will happen
}

export async function callTool(session: Session, tool: Tool, args: unknown) {
  if (!session.allowedTools.has(tool.name)) {
    throw new Error(`Tool ${tool.name} is not available in this session`);
  }
  // Once untrusted text has been read, anything beyond reading needs a person's OK.
  if (tool.risk !== "read" && session.sawUntrustedContent) {
    const approved = await session.confirm(`${tool.name} ${JSON.stringify(args)}`);
    if (!approved) return { refused: true };
  }
  return tool.run(args);
}

The important property is that this check lives in ordinary code. Nothing in the model’s context can change it.

05Validate the output too

Injection often aims to get data out, not just to trigger an action. Check what the model produces before it reaches a user or another system:

• validate structured output against a schema, and reject anything that doesn’t fit;
• strip or block links and images that point at domains you haven’t approved, since a rendered image URL can carry data to an attacker’s server;
• scan for personal data and secrets that shouldn’t leave the system;
• never pass model output straight into a shell, a SQL query or eval.

Output checks are also where many accidental leaks get caught, injection or not: a model quoting a document the user shouldn’t see, or echoing a customer’s details into a log.

06Test it like an attacker

Resistance to injection should be measured like any other quality. We keep a set of attack cases next to the functional eval set: direct jailbreak attempts, instructions hidden in documents and tool outputs, exfiltration attempts through links and images, and attempts to call each tool out of turn. They run on every change, and the release gate treats a newly successful attack as a blocking regression.

None of this makes injection impossible. It makes it boring: an attack that gets through produces a refusal, a confirmation nobody approves, or a blocked request in the logs.