Red-Teaming AI Applications: Testing the Failure Paths That Demos Hide

The demo exercises one path. Production exercises all of them.

A demo runs on a clean prompt, a tidy retrieval index, and a cooperative user who types exactly what the script expects. Then the system ships, and inputs arrive from everywhere: retrieved documents, tool outputs, client-side payloads, and users who type things nobody planned for. The model that looked reliable in the walkthrough now sits inside a system with permissions, session state, and consequences.

That gap is where red teaming AI applications earns its keep. Not as a prompt-hunting exercise, and not as a findings document that gets filed and forgotten. Red teaming here is a failure-path engineering discipline: define the paths the demo hides, reproduce them under production-like conditions, and convert each confirmed failure into a control plus a regression test.

A finding that does not become a control and a regression test is a story, not a fix.

Why the Demo Passes and the System Still Fails

Classic red teaming exploits the cyber kill chain — the sequence of steps an attacker takes to move from initial access to impact. AI red teaming borrows the adversarial posture but changes the target. It simulates a user who wants the system to misbehave in a specific way, and it probes for risks that are partly content-related and partly security-related. That distinction matters because the failure you are hunting is often not a crash or an exploit in the traditional sense. It is the system doing exactly what it was told, by someone who was not supposed to be giving instructions.

The unit under test is the application, not the base model. The system prompt, the orchestrator that routes requests and tool calls, the retrieval index, the tool permissions, and the output handling all sit inside the attack surface. A model that refuses a harmful request in isolation may comply once a retrieved document frames that request as an authorized instruction. The model did not change. The context did.

This is the first reframe: the demo tests the model, the incident tests the system. And it produces the article's central handle — the failure path. A failure path is a reproducible sequence of inputs and system states that leads to an outcome you would not accept in production. Red teaming is the work of finding those paths before a user does.

Name the decision boundary up front, because it determines how much process you need. If your application is a read-only assistant over public data with no tools and no privileged context, a lightweight adversarial review may be enough. The moment the system retrieves private data, calls tools, or takes irreversible actions, you need a structured, repeatable program — because the cost of a missed path is no longer a bad answer.

What You Are Actually Testing: Model, Application, or Both

The most common planning mistake is testing only the model endpoint. That tells you something useful, but it does not tell you whether your application is safe. Treat the target as layered.

Test the base model with its safety system in place to find gaps that must be handled at the application layer. This is usually done through an API endpoint, and it establishes the floor: what the model will and will not do when nothing else is in the way.

Test the application through its real UI, because that most closely resembles real-world usage. The UI is where front-end structures, session handling, and orchestration weaknesses become visible. It is also where you discover that a control you believed was enforced in the backend is actually enforced only by a client-side check.

Test before and after mitigations. The point is to measure whether a control actually changes behavior, not whether it changes the prompt. A mitigation that makes the system sound more careful without altering its decisions has not fixed anything.

Record which endpoint was used for every finding. A result from a non-production endpoint is a hypothesis until it reproduces on the production path. This is not bureaucracy; it is the difference between a finding you can act on and a finding you will argue about for a week.

This is where observability does the heavy lifting. Traces and evaluation layers are what make a failure reproducible instead of anecdotal. If you cannot retrieve the exact input, the retrieved context, the tool calls, and the output for a given run, you are collecting stories, not evidence. The observability stack is assumed prior ground here — the point is that red teaming without it degrades into guessing.

The Failure Paths Demos Hide

Organize your plan by mechanism, not by tool. A checklist of tools produces a checklist of findings. A taxonomy of mechanisms produces a plan that scales as your stack changes.

Prompt injection, direct and indirect

Direct prompt injection arrives through the chat box: a user tries to override instructions, escalate role, or extract system content. Indirect prompt injection enters through retrieval, documents, tool output, or client-side payloads rather than the chat box. It is often the higher-consequence variant, because the user never types the attack. They ask a normal question, and the poisoned document does the work. Whether that path is reachable depends on your architecture: if untrusted content can cross an instruction boundary and the model treats it as a command, the risk is live. If retrieved text is structurally separated from instructions and downstream actions are gated, the same injection attempt may be inert.

Front-end data is not trustworthy

Attackers will intercept and modify client-side data structures — JSON payloads, chat logs, session fields — to attempt indirect injection. Design your application so that privileged system messages originating from the user interface are dropped or blocked by application logic. Prompt wording that discourages this is not a control. It is a suggestion, and suggestions lose to a modified payload. The condition that makes this high risk is client-side state influencing authorization or instruction flow; if the server independently re-derives identity and permissions, a tampered payload has less to grab onto.

Data leakage

Look for cross-tenant retrieval bleed, secrets sitting in context, verbose error surfaces that expose internals, and outputs that echo privileged system content. Retrieval is the usual suspect: a shared index with a filter applied only at query time is fragile, because the tenant boundary lives in application code rather than in the data store. That fragility becomes a real leak when the filter is bypassed, misconfigured, or applied inconsistently across code paths. A shared index can be safe when the tenant boundary is enforced server-side, tested directly, and backed by an independent isolation check — but that is a design commitment, not a default.

Unsafe tool use

Over-broad scopes, missing confirmation on destructive actions, and tool output treated as trusted instructions. That last one is subtle. If a tool returns text that the orchestrator feeds back as instructions rather than as data, you have built a remote control for your own agent.

Unreliable behavior under ordinary conditions

Not every failure is adversarial. Long sessions accumulate contradictory context. Retrieved evidence conflicts. Requests arrive ambiguous. Upstream services degrade and return partial results. Test these paths too, because they are the ones that generate support tickets rather than incident reports.

Scoping a Risk-Based Red-Team Plan

Prioritize by consequence and reachability. Ask three questions: what can the system do, who can reach it, and what is the worst realistic outcome. A path that lets an anonymous user trigger a destructive tool call outranks a path that requires an authenticated admin to see a mildly confusing answer.

Assemble a diverse team across AI, security, and domain expertise. Domain specialists surface risks that generalists miss — a medical reviewer will spot a harmful recommendation pattern that a security engineer reads as a normal answer. Recruit both benign and adversarial mindsets. The benign tester finds the confusing path; the adversarial tester finds the exploitable one. You need both, because confusing paths become exploitable ones once someone is motivated.

Write a round brief before anyone starts. It should cover the purpose of the round, the features in scope, how to access them, focus areas, the time budget, the recording format, and who to contact with questions. A brief is not overhead; it is what keeps a round comparable to the last one.

Capture reproducible evidence per finding: the date, a unique identifier for the input/output pair, the exact input, the observed output, and environment notes. Use a shared findings store so testers can see each other's examples, build on them, and avoid duplicate work. A shared spreadsheet is often enough to start.

Run exercises in a production-like non-production environment, and be explicit about what that environment does not reproduce. Different data, different scale, different upstream services — each gap is a place where a finding might not survive contact with production.

Manual Probing, Automated Scans, and Where Each Belongs

Manual red teaming is time- and expertise-intensive, and it depends on creative adversarial thinking. That makes it a bottleneck when you try to scale it across many applications. It is also the most reliable way to find genuinely novel failure paths, because it is not bounded by a predefined list.

Automated scanning accelerates the identification and evaluation of known risk categories at scale, and it supports shifting left from reactive incidents toward proactive testing. But automated coverage is bounded by its risk taxonomy. It finds what it was built to find — which is exactly why it cannot replace exploratory probing.

The ordering is a default, not a law. If you already have automated evaluations, use them to establish a cheap baseline before a manual round; if you are starting from nothing, an initial manual pass is usually the fastest way to discover application-specific paths that a generic suite would miss. Either way, the two modes correct each other: manual probing expands the taxonomy, and automated runs measure whether mitigations hold across it.

Two operational details are easy to skip and expensive to skip. Automated runs should redact harmful adversarial inputs from results, so developers and non-technical stakeholders are not exposed to the attack payloads. And agentic runs need transient, non-logged execution — meaning the run is not persisted by the agent service — so harmful data is not stored after the exercise.

The decision rule: automate regression of known failure paths; keep humans on novel, cross-domain, and multilingual abuse patterns. Those are the categories where a fixed taxonomy runs out of coverage first.

From Finding to Control: What Actually Blocks the Path

Relying on a strong system prompt leaves the application exposed when the underlying infrastructure or data pipeline can be manipulated. The prompt is a weak authorization boundary, because everything upstream of it can rewrite the context it depends on. That does not make it the weakest control in every stack — it makes it the wrong place to enforce trust decisions that belong in code.

Enforce trust boundaries in application logic. Sanitize inbound data. Drop privileged messages from user-facing channels. Separate instruction from content, so retrieved text is treated as data rather than as a command. These are code-level decisions, not prompt-level wishes.

Constrain tools with least privilege, explicit allowlists, scoped credentials, and human confirmation for irreversible actions. If a tool can delete, send, pay, or publish, it needs a confirmation step and a scope that limits the blast radius — the set of systems and data reachable if the control fails — when the confirmation is bypassed.

Apply ordinary application security to the whole AI stack. Third-party libraries, orchestrators, and components carry conventional supply-chain risk, and that risk does not disappear because the system contains a model. The supply-chain controls are assumed prior ground; the point here is that they are not optional just because the interesting part is generative.

Integrate application and infrastructure logs into centralized monitoring, so model poisoning, data leakage, adversarial inputs, and supply-chain issues become detectable rather than invisible. A failure path you cannot see is a failure path you cannot fix.

State the boundary honestly: controls reduce blast radius and raise attacker cost. They do not make an LLM application provably safe. Anyone who tells you otherwise is selling something.

Turning Findings Into Regression Tests

Track each finding, prioritize by risk, implement a mitigation, and verify that the fix does not introduce a new failure. That last step is where most programs quietly fail, because a control that blocks one path often opens another.

Encode confirmed failure paths as repeatable evaluations that run on every meaningful change to prompts, retrieval, tools, or models. Without this feedback loop, red teaming becomes performative rather than a meaningful critical practice. The finding was real; the fix was real; then someone edited the system prompt six weeks later and the path reopened with nobody watching.

A regression test only reproduces a path if it preserves the state that made the path work. A prompt and an output are not enough. The minimum record needs the trigger input, the retrieved artifacts or document identifiers that shaped context, the tool calls and their arguments, the model and configuration version, the policy or prompt version, the expected security invariant, the observed impact, and a pass/fail assertion that fails without the control and passes with it. Without that record, a test can pass while the underlying path has quietly changed.

Treat red teaming as a form of acceptance testing. No amount of pre-deployment testing anticipates every issue, so the suite has to keep growing. Each round should add paths, not just re-run the old ones.

Assign ownership and a review cadence. An unowned regression suite decays into a skipped CI job, and a skipped CI job is indistinguishable from no test at all.

Governance, Evidence, and the Limits of What You Can Claim

A red-team report is evidence of testing effort and coverage, not proof of safety. Say what was tested, on which endpoint, and what was out of scope. The out-of-scope section is often the most honest part of the document.

Red teaming extends beyond technical components to scrutinizing governance structures and challenging foundational design assumptions. Non-technical stakeholders — legal, risk, and management — can use red-team scenarios to pressure-test policy and oversight assumptions, because a scenario that describes a plausible misuse is more useful to them than a control matrix.

External and regional expertise is increasingly relevant, because high-risk failure modes depend on domain, language, and local context that one internal team cannot replicate. A model that behaves acceptably in one language may fail in another, and the failure may be invisible to reviewers who do not read it. This is a watchpoint rather than a settled practice: the direction of travel is toward more distributed, multilingual testing, but the evidence base for how much it changes outcomes is still thin.

Two open questions are worth holding rather than resolving. First, how much of a red-team result should be shared externally, given that the same findings describe exploitable paths? Second, how do you keep coverage honest as models, tools, and retrieval corpora change underneath a fixed test suite? Neither has a clean answer, and treating either as settled is a mistake.

What to Learn and Build Next

Build the smallest useful artifact first: a findings template with reproducible identifiers, plus one regression test for the highest-consequence failure path you already know about. You almost certainly know one. The system does something you would not want a customer to see, and you have been treating it as an edge case.

Skills worth developing: threat modeling for retrieval and tool boundaries, adversarial prompt construction, evaluation design, and log-based detection. Reusable assets that compound: a failure-path library, a round brief template, and an evaluation suite wired into CI. These are the artifacts that make the second round cheaper than the first.

Watch the ecosystem direction — automated red-team tooling, agentic risk categories, and access-tiered security models are moving fast. Treat vendor claims as claims and verify them against your own stack. A tool that finds known risks at scale is useful; it is not a substitute for understanding your own failure paths.

Here is the decision rule I would carry out of this article. If you cannot reproduce a failure on demand, you do not yet understand it well enough to fix it. And a finding is only fixed when a regression test fails without the control and passes with it.

So pick the highest-consequence failure path in your own system. Reproduce it in a production-like environment. Turn it into the first test in a suite that grows with every round. The demo already passed. Now make the failure paths fail.