Prompt-Injection Defense Architecture for Tool-Using Agents

Alternate title: Tool-Using Agents Under Prompt Injection: Architecture for Contained Autonomy

Prompt injection in a tool-using agent is not a prompt-polish problem. It is an architectural boundary problem. The moment a language model can read untrusted bytes and also hold credentials, private context, or a network egress path, you have built an interpreter whose instruction stream is attacker-influenced. Stronger system prompts do not fix that; they raise the attacker’s cost slightly while leaving the trust boundary exactly where it was. The useful question is not “how do we stop the model from being fooled?” but “what is the maximum damage a fooled model can do?” This essay compares seven containment components — checklist mitigations, trifecta reduction, input provenance signalling, capability and control-flow enforcement, construction patterns, egress policy, and approval gates — and argues that only the last five are load-bearing when the model is assumed compromised.

Sources and scope: This essay is desk research over official documentation and papers — no agent was operated and no injection payload was executed. It draws on OWASP’s Top 10, Willison’s lethal trifecta, CaMeL, Spotlighting, Microsoft’s defense-in-depth post, and the design-pattern catalog.

Checklist-Only Mitigations Set a Baseline, Not a Boundary

OWASP’s guidance gives a defensive checklist: constrain behaviour through system prompts, define strict output formats, filter inputs and outputs, apply least privilege to the model’s tools and APIs, require human approval for privileged operations, segregate untrusted content, and run adversarial testing. Each control reduces incident probability, but every one of them still assumes the model declines to act on injected text — which is precisely the assumption an injection attack invalidates.

The checklist is worth implementing, because several entries are genuinely architectural: least privilege on tools and human approval for privileged operations constrain what a compromised model can reach, not what it believes. Others are probabilistic. Input and output filtering, output schemas, and system-prompt constraints all live inside the same trust domain as the thing they are protecting, and classifiers are themselves models with their own attack surface. The OWASP taxonomy is also the reason agent builders should care about indirect injection specifically: direct injection is the user’s own crafted prompt, while indirect injection arrives inside content fetched from websites, files, or third-party sources and need not be human-visible at all. For a coding agent reading a repository, a support agent reading a ticket, or a research agent reading a web page, indirect injection is the operating condition, not the edge case. A checklist-only posture therefore leaves the agent as an injectable interpreter that has been asked politely to behave.

Lethal-Trifecta Reduction: Pick a Breakpoint

Willison’s lethal trifecta states that an agent combining private data access, exposure to untrusted content, and an external communication channel is exploitable by prompt injection. Containment by construction means removing at least one leg: isolate private data from retrievals, sandbox untrusted content, or deny external communication by default. The breakpoint should follow from the product’s minimum capability set.

This is the cheapest architectural control available, because it is a product decision rather than an engineering one. An internal copilot that never sends outbound messages can read untrusted documents and private data with the exfiltration leg removed. A public-facing agent that must send email can be given no private data at all. What you cannot do is grant all three and then filter your way to safety — the trifecta framing is deliberately about capability composition, not payload patterns, which is why it survives attacker adaptation. Willison’s practical guidance is to avoid the trifecta where possible, sandbox agents that cannot avoid it, and treat untrusted content as an adversarial channel rather than as ordinary input.

The hard part is that “external communication” is broader than a send_email tool. Writing to a shared document, committing to a public repository, posting a comment, calling a webhook, and storing state in a bucket another party can read are all egress. Breakpoint selection should be done by enumerating every channel through which a model-controlled string can leave the trust boundary, then deciding which of the three legs the product can actually live without.

Spotlighting: Delimit, Datamark, Encode

Microsoft Research’s Spotlighting transforms input so provenance becomes a continuous signal: delimiters wrap untrusted spans in explicit markers, datamarking interleaves a distinct token through untrusted text, and encoding converts untrusted content into a form the model can separate from instructions. The approach reduces instruction/data confusion without touching model weights, and was reported robust with minimal detriment to the underlying NLP tasks.

Spotlighting matters because it addresses the failure mode at its root. The model does not fail because it is malicious; it fails because in a single token stream, a sentence inside a retrieved PDF and a sentence inside the operator’s system prompt are the same kind of thing. Explicit provenance markers give the model something to condition on. But this is an input-shaping control, not a boundary: it raises the difficulty of a successful injection and does nothing about what a successfully injected model is permitted to do next. Adaptive attackers who can see the delimiter scheme can attempt to spoof it or to argue the data into the trusted region. Treat spotlighting as a layer that pairs with egress policy and approval gates rather than as a standalone defense.

CaMeL: Capability and Control-Flow Enforcement

CaMeL wraps the model in a protective system layer so security holds even if the model itself is injectable. A privileged LLM extracts control and data flows from the trusted user query and writes pseudo-code; a custom Python interpreter executes that plan under capability-based policies, so untrusted data retrieved during execution can never influence program flow.

The mechanism is worth stating precisely, because it is the clearest example of containment by construction in current literature. Every value carries capabilities — its provenance and the set of readers allowed to consume it. A data-flow graph assembled during execution decides whether a given tool call may proceed, must be blocked, or requires explicit user approval. This is a direct import of ideas from control-flow integrity, access control, and information-flow control — the lineage runs through libcap, Capsicum, and CHERI — applied to an LLM planner rather than to compiled code.

The cost is real and should be budgeted. On AgentDojo, CaMeL solves 77% of tasks with provable security against 84% for an undefended system: a measurable utility loss in exchange for a guarantee that does not depend on the model resisting manipulation. Willison’s commentary on CaMeL frames this trade honestly, and the reference implementation is published as google-research/camel-prompt-injection. For workflows where exfiltration or destructive tool calls dominate the risk register — payments, infrastructure, code execution, customer data — paying seven points of task completion to stop depending on model obedience is usually the right trade. For a summarisation agent, it is not.

Construction Patterns: Action-Selector, Plan-Then-Execute, Dual-LLM

Beurer-Kellner et al. catalogue architectural patterns that shrink the injection surface rather than filtering content: the action-selector pattern lets the model choose only from a fixed set of pre-approved actions with no free-form arguments, plan-then-execute fixes the plan before untrusted data is read, and the dual-LLM or quarantined-LLM pattern keeps raw untrusted text away from the model that holds tools.

The design-pattern catalog is useful because it reframes the problem as a decomposition question. If the model that touches untrusted content has no tools, and the model that has tools never touches raw untrusted content, the injection has no path to a side effect. Context minimisation is the same idea applied backwards: if a step does not need a piece of context, do not put it in that step’s window, because an unused context slot is an unmanaged attack surface.

Action-selector has the strongest structural property and the highest product cost, since free-form tool arguments are often the entire value proposition. Plan-then-execute is a good middle ground: the plan is derived from a trusted query, then executed against data the planner has not seen, which is close to the CaMeL idea implemented with ordinary prompting. Dual-LLM quarantine is the most incrementally adoptable for existing systems, because it is a topology change rather than a rewrite. All three depend on a strict action schema; if the model can synthesise an arbitrary tool call, the pattern collapses back into a single agent with a nicer diagram.

Deterministic Backstops: Egress Allowlisting and Approval Gates

Deterministic backstops operate independently of model intent. Microsoft’s defense-in-depth guidance pairs probabilistic controls such as Prompt Shields with deterministic ones, including plan drift detection that monitors multi-step reasoning for deviation from the intended task flow, and egress allowlisting that restricts where a tool call may send data. Human-approval gates for privileged tools belong outside the model’s direct control.

The value of running probabilistic and deterministic controls side by side is that they fail differently. A detector can be evaded; a network policy cannot be talked out of a destination it does not have on its list. Microsoft’s MSRC write-up positions Prompt Shields, Spotlighting, and plan drift detection as complementary rather than as a ranked list, and the Zero Trust guidance for indirect injection makes the deterministic half explicit. Egress allowlisting is cheap to operate but requires a per-tool destination map, and that map is the actual security artifact: if it is derived from what the model requests rather than from what the product needs, it is not a control.

Approval gates are the last deterministic layer before an irreversible action. They are also the control most likely to degrade in practice, because approval prompts compete with the operator’s attention and a gate that fires constantly is approved reflexively. The mitigation is scoping rather than persuasion: gate privileged, irreversible, and high-blast-radius calls, run deterministic checks before the approval prompt appears so the operator sees what will actually execute, and keep the approval mechanism outside the model’s callable tools. OWASP’s checklist places human approval for privileged operations alongside least privilege; both only hold if enforcement lives in the harness rather than in the system prompt.

Comparison Table: Containment Layers in Practice

The table below compares the seven components as containment layers, split by component rather than by attack phase. Columns are Component, Control type, Injection surface reduced, Operational cost, and Failure mode; values are intentionally qualitative and no single row is presented as a winner. Defensible deployments combine an architectural boundary with deterministic backstops.

Read the table as a composition guide, not a ranking. Trifecta reduction decides what the agent may ever do; construction patterns and CaMeL decide how trust flows between the components that do it; spotlighting shapes what the model perceives; egress allowlisting and approval gates decide what happens when everything above them is wrong. The rows that cost the least protect the least, and the rows that cost the most are the ones that hold when the model has already been compromised.

FAQ

These questions come up in nearly every design review of a tool-using agent. The consistent answer is that no single control is complete, that provenance and policy have to surround the data rather than live inside the model, and that deterministic backstops must operate independently of model intent. Filtering, spotlighting, and enforcement solve different problems and should be layered.

Does CaMeL-style enforcement replace content filtering?

No. Content filtering reduces input noise and catches known payload shapes before they reach the planner, while CaMeL-style enforcement constrains what untrusted data is permitted to do once it is inside. They operate at different layers and fail in different ways, so filtering remains useful for reducing alert volume and enforcement remains necessary for the guarantee. Removing filtering because enforcement exists is a defensible simplification only if you have measured the resulting cost elsewhere.

Can spotlighting be bypassed by adaptive attackers?

It can be weakened. Delimiters and datamarking are visible conventions, and an attacker who knows the scheme can attempt to mimic trusted formatting or argue the payload into the trusted region. That is why Microsoft’s guidance places Spotlighting alongside Prompt Shields and plan drift detection rather than treating it as complete. Use it with egress allowlisting and policy enforcement so that a bypassed provenance signal is not the last line of defense.

Where should human-approval gates sit in a tool chain?

On privileged, irreversible, or high-blast-radius tool calls, positioned so that deterministic checks run before the approval prompt and the operator sees the exact arguments that will execute. The gate itself should live in the harness and not be callable or bypassable by the model. Approval is a latency and attention cost, so scope it narrowly; a gate that fires on every read trains operators to approve without reading.

The through-line is that containment by construction changes what a compromised model is able to do, not what it is able to believe. Pick the trifecta leg your product can live without, minimize what each model step can see, enforce capabilities and egress outside the model, and put a deterministic gate in front of anything irreversible. Everything else is hardening.

📖 Related Reads

• NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from CodeIntel Log.