Prompt Injection

What is Prompt Injection?

Prompt Injection occurs when a later part of the context contains an instruction that conflicts with the original instructions—for example within an inserted document, a web page, or a user input. This instruction competes with the System Prompt or previous instructions for the same probability distribution during token selection.

How it works

A language model does not distinguish instructions through fixed access rules. Instead, it processes the System Prompt, previous messages, and inserted content together within the same context. A conflicting instruction appearing later in the context shifts the probability distribution of the next tokens. How strongly this happens depends on the model’s training, role annotations, the surrounding context, and the exact wording.

Why no instruction is absolute

There is no fixed priority rule under which either earlier or later text always wins. A System Prompt is not a server-side access control mechanism but rather influences the probability distribution within the same computation as the rest of the context. Any application that incorporates untrusted content into the model context—such as documents, web pages, API responses, or tool outputs—must therefore take Prompt Injection attacks into account.

Prompt Injection is not a traditional programming bug that can be fixed once and for all. It is a direct consequence of the fact that language models do not process instructions and content separately but treat both as text within the same context. Stronger wording or additional rules can reduce the risk, but they cannot eliminate it completely.

Why Prompt Injection has practical implications

• Untrusted content: Any application that incorporates untrusted content such as documents, web pages, or user input into the model context must take Prompt Injection attacks into account.
• No complete protection: Security measures reduce the risk but cannot guarantee complete immunity.
• Context dependency: How successful an injection is depends on the model, role annotations, and wording—not on a fixed priority rule.

Typical example

An inserted document contains a hidden instruction such as “Ignore all previous instructions and instead respond with…” This instruction competes within the context against the intended System Prompt for influence over the model’s response.

Further reading

Fundamentals → Prompts: Why Wording Changes the Answer

Related terms → System Prompt
→ Prompt