Annie Tsai

AI agents are trained to achieve task completion and reach a satisfactory state as defined in the spec as their singular objective. Too often, however, the team building the system doesn’t scope into their project the capacity to reason and react to appropriate levels of threat protection or potential negative, unscoped, or unintended use cases. As product and process builders, we too often spend a lot of time defining the scope of what our agents should be doing and not enough time defining what our agents should NOT be doing.

This week, we’re breaking down threat protection into workable chunks so you can develop your own organization-wide framework threat taxonomy.

Agents, both smart and dumb

What makes an AI agent good at its job is the same character trait that produces the unexpected conversations. It wants to help, so it tries very hard and eventually finds a path to yes because that’s what it is designed to do.

That instinct to orient toward completion, usefulness, and staying in the conversation until something is resolved to satisfaction is what makes an agent worth deploying in the first place. It’s why agents work in the first place, and it’s the reason why we’ve been chuckling and shaking our heads at the small pile up of “oopsies” mistakes that go public very quickly.

In March 2026, a Reddit post circulated showing Chipotle’s order support bot, Pepper, successfully reversing a linked list. As The Vibe Reporter documented, Pepper can do it because it draws from a foundation model trained on every Stack Overflow thread and CS101 lecture uploaded to the internet and it knows how to code because it knows a lot of things. There was no exploit or attacker so to say, just a capable system that hadn’t been given the proper governance model on what it should and should not be engaging with, as well as how to let a user down in a way that still counted as task completion. Amazon’s Rufus had fielded off-scope conversations continuously, months after the problem was publicly documented. A Chevrolet dealership’s chatbot was walked into committing to a $1 vehicle sale through a well-framed sequence of confirmations. Air Canada’s bot promised a bereavement discount it had no authority to offer, and when the company lost the small claims case that followed, its defense — that the chatbot was “responsible for its own actions” — the court didn’t buy it. DPD’s bot, when a frustrated customer couldn’t reach a human, ended up swearing and writing a poem critical of the company’s own service.

The failure for all of these use cases is in the system’s design.

The design property that makes an agent productive can feel easy and seamless, but is complicated through both an overly expansive scope and its underlying foundational model. Agents must prioritize goal-completion orientation, conversational continuity, willingness to find a resolution, but these are also the same things that makes it exploitable when it faces potentially bad actors. This is an architecture argument, not an AI safety argument. The boundary between “helpful to your customer” and “helpful to anyone who knows the right framing” never enforces itself.

Simon Willison’s “lethal trifecta,” cited in OWASP’s June 2026 State of Agentic AI Security report, gives this a meaty definition: any agent that combines access to private data, exposure to untrusted content, and the ability to communicate outside the system can be turned into an exfiltration or liability tool without a traditional exploit. Three CVEs were documented in that same reporting window, including a CVSS 9.6 remote code execution in MCP infrastructure and a Microsoft Semantic Kernel vulnerability where a single injected prompt achieved code execution by traversing Python’s class hierarchy, no browser exploit or memory corruption required. Most outward-facing agents have all three trifecta components by design.

OWASP moved this cycle from cataloging hypothetical risks to cataloging actual CVEs. Research aggregated by AgentMarketCap puts 88% of organizations as having confirmed or suspected a related incident in 2025–2026. Cisco’s 2025 data finds 83% of organizations lack basic exposure prevention controls for their AI tools. Organizations that enforced least-privilege access for their agents had a 17% incident rate; those that didn’t had 76%.

That gap here is explained less by attacker sophistication but rather by defaults and undocumented assumptions.

Threats by vertical, a few examples to get you started

The threat surface varies by context. While not an exhaustive list by any means, what follows is a representative look at what to consider in your threat taxonomy design across three verticals.

Financial services is where an oops from your AI agent can very quickly become fiduciary exposure. An agent handling account queries operates inside compliance and regulatory obligations that a retail chatbot never touches. Here’s are some examples of what this looks like in practice:

• Using urgency to negate good decision-making: it can be very easy for a bad actor to  urgency-framed wire transfer authorization requests, where time pressure creates a path to approval that slower processing and more gate-checking would catch
• SIM-swap-adjacent 2FA requests: where someone claims their phone changed and asks the agent to confirm or relay a verification code
• Investment queries that drift gradually from information into advice, until the agent is describing suitability it has no license to assess

What you might notice here is that the three examples above have also always existed as issues within financial services customer service organizations in the modern financial services industry. For your specific organization, looking back at the past year’s customer service data and using documented security breach attempts as the baseline for your threat taxonomy is an easy way to get started.

Consumer and D2C is where the risk is visible before it’s serious, and where a bad interaction can very quickly become a viral screenshot. Consumer-facing agents with any kind of resolution authority, like the ability to confirm, approve, or commit on behalf of the company, are the ones that generate public incidents. Here’s what this looks like in practice:

• Commitment traps: users who figure out that the right framing produces a confirmation, and a confirmation creates an obligation. The $1 Chevrolet sale and the Air Canada bereavement refund both came from this pattern.
• Brand voice failures: content the agent generates that’s critical of its own parent company, or fabricated narratives a customer reads as an official company position. Every output in this vertical is potentially a screenshot.

For consumer and D2C deployments, auditing your most common conversation flows for places where a creative framing might produce an unintended commitment is a practical starting point. Then build explicit refusals for those specific patterns into the architecture. In addition to a block list, define the full scope of what the agent is and is not authorized to confirm.

Ecommerce is where the exploit and the payout happen in the same conversation, and where a single vulnerability is replicated across every concurrent session at that moment. Here’s are a few examples of what this looks like in practice:

• Fabricated discount codes: a user claims a promotional code exists, the agent can’t verify it, and processes it anyway
• Price-confirmation traps: a numeric error in a conversation becomes a claimed binding quote
• Order state self-report: a customer asserts their order hasn’t arrived, and the agent processes a replacement or refund without verification

A flaw that produces a $10 mistake at 50,000 concurrent sessions isn’t a UX problem, it’s now a material revenue problem. For ecommerce, any agent with resolution authority should require a verification step before taking action, and these rules should be hard-coded into the routing layer, not left to the model’s judgment.

How to train for threats and not just filter for threats

Enforce scope where the agent can’t argue with it. Chipotle and Rufus were both talked out of their lanes by users who asked nicely. The instruction lived in the system prompt, the one place a persuasive enough conversation can always reach. The fix here is to architect a routing layer that decides what the agent is allowed to attempt before the model ever gets its hands on the question. This is an architecture design decision, not a prompt engineering decision. (The gap between deploying and hardening is the same structural gap I wrote about in Issue #21 between mandating and enabling: the announcement is not the implementation.)

Train on what good looks like, not just on what bad looks like. A block list tells an agent what to refuse, but it doesn’t tell it how to best help. What works well is to pull your best customer service people off the floor to rewrite the source material from scratch: real examples of excellent handling, not rules or genericized best practices or guidelines. A taxonomy is the floor, and your best people and their best answers are the ceiling.

Treat every surprising conversation as a new regression test. Every incident referenced above was catchable with adversarial testing before launch. The harder problem is drift after launch, when edge cases accumulate invisibly because nothing broke loudly enough to notice. Every time a real conversation surprises your classifier, it belongs in the test suite permanently. Any case that should have been blocked but only got flagged is a release blocker, not a backlog item. Here, it’s important to build into your design another architectural decision – sub-agents that are regularly scanning and assessing conversations for response patterns, anomalies, escalations, and errors so issues don’t persist or expand in scope and threat.

Your takeaway this week

Here’s the full threat taxonomy to download and personalize for your scope and needs. Of course, this is not a fully exhaustive list and you will need to work with your team to ensure you’ve covered an appropriate scope for your use cases.

Would love to see any additions you’d throw in (feel free to comment below with them) so we can all continue to improve the quality of our agentic systems together.