Prompt Injection in AI Applications: Mapping Trust Boundaries Before Tool Use

The attacker never talks to your model. They leave a sentence in a document it will read later.

That detail is why prompt injection keeps escaping the mental model most teams bring to it. The instinct is to treat it as a bad-input problem: find the malicious string, filter it, move on. But the payload does not arrive through a channel you control, and the model cannot verify who is speaking. Instructions and data land in the same context window, separated by formatting conventions the model learned to respect — not by any boundary it can enforce.

If you have already worked through red-teaming practice, agent permission design, or retrieval as context selection, you have most of the pieces. What is missing is the map that connects them: where trust actually breaks in a tool-using application, and which control belongs at which break.

Why the Model Cannot Tell Instructions From Data

Start with the mechanism, because everything else follows from it.

A modern LLM application assembles a single token stream. The system prompt, the user's turn, retrieved document chunks, tool responses, page text from a browser, OCR output from an image — all of it is concatenated into one context window. There is no privilege ring, no kernel mode, no signed envelope. The separation between "these are my rules" and "this is content I should reason about" is a formatting convention, and conventions are suggestions to a probabilistic reasoner.

This is what makes prompt injection structurally different from SQL injection or cross-site scripting. Those attacks exploit a parser boundary: you find the escape sequence, break out of the string literal, and land in the code path. The fix is input sanitization at a well-defined interface. Here, the "parser" is a model that generalizes. It has no formal grammar that says </data> ends the data region. It has a learned expectation about what instructions look like, and an attacker's job is to write something that reads like an instruction.

Two classes follow from this.

Direct prompt injection is the user attacking the model — the jailbreak case. The user is adversary and victim simultaneously, which limits the security impact. If someone convinces your assistant to produce disallowed output, that is a policy problem, not usually a breach of someone else's data.

Indirect prompt injection is the harder class. A third party plants instructions in content the model will later consume: a webpage, an email, a shared document, a database record, a tool response. The victim never sees the payload. They asked the agent to summarize a page, and the page contained a sentence addressed to the model rather than to the reader.

The payload needs no exotic encoding. Hidden text, white-on-white markup, and non-printing Unicode all work, but so does plain ASCII in a .txt file. Microsoft's security guidance makes this point directly: the technique does not require any specific file format or encoding.

One clarification matters for how you prioritize. A prompt injection is "successful" when the model follows the attacker's instruction. That is not automatically a security event. As Microsoft's AI bug bar frames it, the influence only becomes a vulnerability when it produces a security impact — data exfiltration, unintended actions, or worse. A model that recommends the wrong apartment because a listing told it to is a quality failure. A model that emails your internal pricing sheet to an attacker is a breach. Same mechanism, different consequence, and the consequence is what your controls should be sized against.

What is known: the mechanism, the demonstrated attack classes, and the fact that indirect injection is one of the most widely reported techniques in AI vulnerability disclosures. What is inferred: how robust any specific model is today. Vendors publish robustness improvements, and those are real, but they are measured against known attack families. Treat them as a moving baseline, not a guarantee.

Mapping Trust Boundaries in a Tool-Using Application

Before you write a single control, draw the map. The deliverable is a boundary inventory — a list of every point where content crosses from a context you do not control into the model's context.

Enumerate the crossing points in your actual system:

• User input, including the parts users do not type directly
• Retrieved documents from your vector store or search index
• Web and browser content, including dynamically loaded elements
• Tool and API responses
• File uploads
• Images and any OCR text extracted from them
• Audio transcripts
• Metadata fields — filenames, titles, author fields, headers
• The model's own output when it is fed back into a later turn

For each boundary, answer three questions:

1. Who can write here? If the answer is "anyone on the internet" or "anyone who can send this account an email," you have an untrusted boundary.
2. What authority does the model grant this content? In practice, the model grants it whatever authority the formatting implies. If retrieved chunks are inserted with the same structure as instructions, the model may treat them as instructions.
3. What capability is reachable if this content wins? This question determines severity.

The dangerous composition is untrusted content plus a privileged capability in the same context. Retrieval plus a write-capable tool is the canonical case, and it is worth naming precisely because it is so common: teams add retrieval for grounding, then add tools for usefulness, and never notice they have connected an open input channel to an action channel.

Tool output is the boundary teams underrate most. A compromised upstream API, or merely a verbose one, can return text that reads as instructions. If your agent pipes that response back into the model without marking its origin, you have handed a third party a direct line into your context window.

Multi-modal inputs widen the surface in a specific way: OCR text and transcripts are untrusted text that frequently bypasses text-only review. If your injection classifier only runs on the user's typed prompt, the image attachment is a side door.

I would not re-derive retrieval or agent permissions here. The point is narrower: retrieval is a context-selection system, agent permissions are an action-scoping system, and prompt injection lives in the seam where they meet. Map that seam.

What Actually Breaks: Exfiltration, Unintended Actions, and Cascades

Named failure modes let you match controls to consequences. Four are worth separating.

Data exfiltration is the most widely demonstrated impact. The model is steered into emitting sensitive context through a channel the attacker can read. That channel is often not an API response — it is a rendered link, an image URL, a markdown reference, or a tool call that carries data in its parameters. The attacker does not need the model to say anything incriminating; they need it to construct a URL.

Unintended action is the agent invoking a tool, writing a record, sending a message, or changing state with the user's or the service identity's privileges. The model believes it is helping. The audit log shows a legitimate-looking request.

Remote command execution is the extreme case: an application that can execute code or run shell commands on the user's behalf can be tricked into running attacker-specified commands, potentially at the user's privilege level. This is rare in production systems precisely because it is so obviously dangerous, but it is the reason "the agent has a shell" deserves a hard look before anything else.

Cascading impact is the one that breaks incident response. One injected sentence alters a search query, which changes which evidence is retrieved, which changes the final answer, which changes what the user does next. The failure is not localized to one turn, and by the time anyone notices, the trace shows a sequence of individually reasonable steps.

There is also persistence. Content the agent writes into memory, a summary, or a shared document can carry the payload into later sessions — a stored injection that re-fires without the attacker doing anything further.

Be careful about magnitude claims here. Vendor-reported attack success rates are measured under specific conditions against specific models, and a single figure is not a property of the field. What generalizes is the mechanism and the impact classes, not the number.

Content Isolation: Making Untrusted Text Look Untrusted

The first control layer reduces the probability that the model confuses data for instructions. It is worth doing everywhere and sufficient nowhere.

Structural separation. Never concatenate raw user or retrieved text into the system prompt. Use role separation, structured templates, and delimiters — XML tags or JSON envelopes — so the model can syntactically distinguish rules from data. This is cheap and it eliminates the crudest attacks.

Spotlighting and provenance marking. Transform untrusted content so its origin is visible in the token stream. Microsoft describes approaches that include encoding or datamarking untrusted spans, paired with a system prompt instruction not to follow instructions found inside them. The transformation does not need to be secret; it needs to make the boundary legible to the model.

Classifier-based detection at the input boundary. Scan both user prompts and documents for injection attempts, and filter outputs before rendering or tool dispatch. Anthropic describes scanning all untrusted content entering the context window and flagging potential injections with classifiers that detect adversarial commands in hidden text, manipulated images, and deceptive UI elements. Note the scope: the classifier runs on everything untrusted, not just the typed prompt.

Rate limiting and throttling. Slow automated probing and iterative payload refinement. This does not stop a determined attacker, but it raises the cost of the search loop.

Here is the distinction that matters, and it is easy to blur: every control in this section is a model-facing mitigation. Role separation, delimiters, provenance marking, spotlighting, and classifiers all influence what the model does with content. None of them enforce authority. They reduce attack probability; they do not create a boundary the model is obligated to respect. A determined payload can survive structural separation, survive spotlighting, and slip past a classifier. Anthropic's own framing is instructive — a meaningful improvement in robustness still represents meaningful residual risk, and no browser agent is immune.

That is not a reason to skip isolation. It is the reason isolation cannot be the only layer, and the reason every model-facing control must be paired with authorization that lives outside the model: independent tool authorization, data-access checks, and approval gates that do not depend on the model's cooperation.

Decision boundary: isolation is cheap, so apply it at every boundary you mapped. It is not sufficient anywhere a privileged capability is reachable from the same context.

Least Privilege and Approval Gates: Shrinking the Blast Radius

Here is where the design question changes. Stop asking "can we stop the injection?" Start asking "what can a successful injection still do?"

That reframe is the highest-leverage move in this article, because it converts an unsolvable detection problem into a solvable architecture problem.

Scope identities per tool and per task. Use constrained tool allow-lists rather than a general-purpose tool surface. Keep credentials out of the model's reachable context. Microsoft's guidance is blunt about why: the agent's privileges define your blast radius. If the agent's identity can reach the whole database, so can the injection.

Separate read from write. Split the retrieval identity from the action identity. A context that can read documents should not also be the context that can send email. This is ordinary service-design discipline applied to a new class of caller.

Human-in-the-loop for consequential actions. Writes, payments, external messages, permission changes, destructive operations. The approval gate is not a UX nicety; it is an independent authorization checkpoint that converts an automated breach into a blocked request.

Make the gate show the actual parameters. An approval prompt that summarizes the action rather than displaying it is a rubber stamp. If the dialog says "the agent wants to send a message" and the user clicks approve, you have added friction without adding safety. Show the recipient, the body, the amount, the target path.

Rollback and idempotency. Reversible actions with audit trails convert a security incident into an operational correction. This is the cheapest insurance in the list, and it is the one teams skip because it is not security-flavored work.

One caution about what approval buys you. Approval reduces automated execution risk; it does not prove the action is legitimate. A human can approve a manipulated summary, a misleading parameter set, or an action under time pressure. Treat the gate as one layer among several — distinct from validation, policy enforcement, transaction limits, and rollback — not as a substitute for them.

Decision boundary: approval gates are overkill for read-only, low-sensitivity flows — you will train users to click through them. They become mandatory the moment an action is irreversible or externally visible.

Validation, Monitoring, and the Controls That Catch What Slipped

Assume something got through. These layers determine whether you find out and how fast.

Validate model output before it triggers anything. Enforce response schemas, allow-list tool names and arguments, and reject outputs that do not fit the contract. If your tool dispatcher accepts arbitrary strings as parameters, the model's output is effectively executable input.

Log the full chain. Prompts, retrieved spans with provenance, tool calls with arguments, and detector verdicts. A failure you cannot reconstruct is a failure you will re-experience. The trace should let you point at the exact turn where authority was transferred.

Alert on detector hits and anomalous tool-call patterns, not just on errors. A successful injection often looks like a normal successful request. Nothing throws. The tool call succeeds. The only signal is that the pattern is unusual for this user or this task.

Convert every confirmed incident into a regression test. This is the handoff to red-team practice, not a re-teaching of it. The value of a red-team finding is that it becomes a test that runs on every release.

State the residual risk explicitly. Monitoring detects; it does not prevent. Detection latency is part of your blast radius, and if your alert goes to a channel nobody watches, your effective latency is measured in days.

Choosing Controls by Risk Tier

You cannot implement everything at once, and you should not try. Tier by consequence, not by model sophistication.

Tier 1 — public, read-only, non-persistent summarization with no external side effects and no sensitive data in context. Isolation and basic output validation may be proportionate here. Elaborate input filters are over-investment when there is nothing privileged to protect. Escalate the moment the boundary changes: sensitive data enters context, the flow gains persistence, the agent can take browsing actions, its output drives user-facing recommendations, or it feeds downstream automation.

Tier 2 — internal read with sensitive data in context. The risk is exfiltration. Isolation, provenance marking, classifier detection on all untrusted inputs, output filtering, and full-chain logging. The non-negotiable control is output filtering, because exfiltration requires the model to emit something.

Tier 3 — externally-acting agents. The risk is unintended action. Everything in Tier 2, plus scoped identities, read/write separation, approval gates on irreversible actions, and rollback paths. The non-negotiable control is the approval gate, because no amount of input filtering makes an irreversible action safe to automate.

Two common misallocations. Teams over-invest in elaborate input filters on systems with no privileged capability, where the worst outcome is a wrong answer. Teams under-invest in approval gates on irreversible actions, where the worst outcome is a wire transfer.

On model choice: newer models with stronger instruction-hierarchy training are a real advantage, and Microsoft's guidance treats model selection as a primary control for high-risk workflows. But it is a non-permanent advantage. Do not treat a version upgrade as a fix, and do not let it substitute for architecture.

What would change the conclusion: a demonstrated bypass of your isolation layer, or a tool whose blast radius you cannot bound, should move the system up a tier. Re-tier when the capability changes, not when the calendar does.

What to Learn and Build Next

Before adding any control, build the boundary inventory for one real application. List every crossing point, who can write there, and what capability is reachable. That map is the deliverable, and it will tell you which tier you are actually in — which is often not the tier the team assumed.

Then write one adversarial test per boundary and run it against your own system. Not a generic payload from a list. Something specific to your retrieval source, your tool surface, your approval flow. Watch what happens. The output is the lesson.

Practice reading a trace end to end and locating the exact turn where authority was transferred. That skill generalizes to every failure mode in this space, not just injection.

The skills worth developing: tool-permission scoping, output schema enforcement, retrieval provenance tracking, and the conversion of incidents into regression tests. The open questions worth tracking: how instruction-hierarchy training holds up under adaptive attacks, whether classifier-based detection generalizes across modalities, and how much of this becomes platform-level rather than application-level responsibility.

Before you add another filter, ask what a successful injection could still reach. If the answer is a privileged, irreversible, or externally visible action, the fix is architectural — narrower permissions, an approval gate, a reversible path — not a better prompt. The controls that compound are the ones that shrink blast radius and turn incidents into tests, because they keep working after the next payload variant ships.