I use AI every day. It drafts, it summarizes, it reviews configurations, it argues with me about threat models, and in all of those uses it earns its keep. So this is not an anti-AI piece. It is an architecture piece, and the argument is narrow: the moment we make AI more than a tool, the moment it plans, decides, and acts on its own, we remove the human from the path. That removal is itself the risk multiplier. A tool's failure is advisory because a human still reviews the output before anything happens. An agent's failure is operational because the bad output is the action.
Twenty-five years of building security programs has taught me that risk lives in mechanisms, not in headlines. So let's walk the mechanism.
Where AI actually is right now
A dated survey first, because perishable claims deserve timestamps. As of mid-2026, frontier language models are genuinely capable at bounded knowledge work: code generation and review, document synthesis, structured analysis, translation between technical and business language. Coding is the dominant enterprise use case by a wide margin. The OWASP GenAI Security Project's State of Agentic AI Security and Governance v2.0 [1] found that more than half of the agentic projects it tracks are coding agents, and its fastest-growing tools all sit in that category. McKinsey's 2025 State of AI survey [2] put organizational adoption at 88% using AI in at least one business function, with 62% at least experimenting with agents.
The limitations deserve the same amount of highlighting. Models still fabricate with confidence. Hallucination is reduced, not solved. They have no reliable way to distinguish instructions from data in their context window, and that single fact is the root of most of what follows in this post. Long-horizon reliability degrades: an agent that succeeds at a five-minute task 95% of the time will fail multi-hour task chains far more often than intuition suggests, because errors compound across steps. The gap between demo and production also remains wide. Surveys through 2026 consistently show most enterprises experimenting with agents while only a small fraction run them in production. That gap is not a technology lag. It is, in part, the industry quietly discovering the argument of this post.
What actually changes from tool to agent
"Agent" gets used loosely, so let me define it mechanically, because the mechanism is where the risk lives.
A plain LLM is a text-completion function. You send it context; it returns tokens. Every consequence passes through a human who reads the output and decides what to do with it. That human is not a courtesy. That human is a control point.
An agent is the same model wrapped in three additions. First, tool access: the model's output is parsed not as prose but as function calls (send this email, run this query, execute this command) and a scaffold executes them. Second, memory and state: results feed back into the context, along with retrieved documents, prior sessions, and whatever else the scaffold gathers. Third, the loop: observe, plan, act, observe again, until the scaffold decides the goal is met. No new intelligence appears anywhere in that wrapper. The model that hallucinated a citation yesterday is the same model deciding which API to call today.
Notice what the loop did to the trust model. In the tool configuration, model output terminated at a human. In the agent configuration, model output terminates at an interpreter with credentials. The text-generation errors we tolerated as annoyances, like fabrication and instruction confusion, are now input to an execution engine. And the loop runs at machine speed, so a wrong turn does not produce one bad action. It produces a chain of them, each one feeding a context that has already accepted the first error as ground truth.
The risk review, adversary first
I map threats before controls, because compliance-first analysis produces checkbox architectures that pass audits and fail incidents. Here is the adversary view of an agent.
Prompt injection starts behaving like code execution. MITRE ATLAS catalogs this as LLM Prompt Injection (AML.T0051) [3], with sub-techniques for direct injection through the user interface (AML.T0051.000) and indirect injection through content the system ingests (AML.T0051.001). Verify current IDs at atlas.mitre.org before citing them yourself, because ATLAS has been expanding rapidly and added agent-focused techniques through late 2025 and early 2026. OWASP holds the same ground as LLM01: Prompt Injection in the Top 10 for LLM Applications 2025 (v2.0, published November 2024) [4], where it has sat at number one since the list existed.
Why does this attack work so much better against agents? Because the model has no privileged channel separating the developer's instructions from instructions embedded in data. An email, a web page, a retrieved document, a tool result: all of it lands in the same context window the system prompt lives in. Against a chatbot, an injected instruction produces bad text, and a human reads the bad text and discards it. Against an agent holding credentials, the same injection produces an executed action under a trusted identity. That is the confused deputy problem, the 1988 vintage [5], where a program with authority is tricked into wielding it for an attacker. Now it runs at machine speed with no deputy ever getting suspicious. The June 2025 EchoLeak finding (CVE-2025-32711) [6] made this concrete: a crafted email could cause Microsoft 365 Copilot to exfiltrate data with zero clicks from the victim, because the assistant read the attacker's instructions as part of doing its job.
Excessive agency is the amplifier. OWASP LLM06: Excessive Agency [4] names the condition where the agent holds more capability, permission, or autonomy than the task requires. It converts every other weakness from embarrassing to expensive. An injected chatbot with no tools leaks a system prompt. An injected agent with a mail API, a database connection, and a payment workflow does whatever the payload says, bounded only by its scopes. Field data through 2026 backs this up: surveys of enterprises running agents attribute the majority of agent-related incidents to over-permissioned credentials, with prompt injection as the trigger.
Improper output handling closes the loop. OWASP LLM05: Improper Output Handling [4] covers downstream systems trusting model output without validation, meaning LLM output flowing into shells, SQL, interpreters, or other agents. In an agent, output handling is the architecture. Every tool call is model output being trusted by an executor.
Then the attack crosses into traditional intrusion. Once an adversary controls an agent's behavior, the agent's credentials are the adversary's credentials. That is ATT&CK's T1078, Valid Accounts [7], under a new name. From there the kill chain is boring and familiar: discovery, collection, exfiltration, all logged as legitimate service activity. We have already watched this play out. On November 13, 2025, Anthropic disclosed disrupting a state-sponsored espionage campaign in which the attackers used an agentic coding tool to automate the bulk of their intrusion work across roughly thirty targets [8]. The agent loop served as a force multiplier for the offense, not just for the enterprise.
The human in the path is a control
Here is the part I want practitioners to internalize, because it reframes the whole debate. The human review checkpoint between AI output and real-world consequence is not friction to be optimized away. It is a security control, and a dense one. A person notices the drafted wire transfer is wrong: that is detection. The wrongness stops at the draft: that is containment. A named human approved or rejected the action: that is accountability. Remove the checkpoint and all three functions disappear in one architectural decision, usually justified by a productivity slide.
NIST's AI Risk Management Framework (AI 100-1, version 1.0, January 2023, with the Generative AI Profile, NIST AI 600-1, July 2024) [9][10] organizes AI risk work into Govern, Map, Measure, and Manage. Every one of those functions presumes someone is positioned to see what the system is doing and intervene. Autonomy does more than add risk to that model. It degrades the machinery the model depends on. You cannot Manage a risk whose realization outruns your ability to observe it.
Zero trust makes the same point from the identity side. SP 800-207 (August 2020) [11] built its architecture on the assumption that no subject is trusted by default and every access is evaluated per-request under least privilege. Agents are non-human identities, and I would argue they need least privilege more than humans do, for two mechanical reasons: they act faster than any review cycle, and they never get suspicious. A human employee handed an odd instruction hesitates, and that hesitation has quietly caught more fraud than most controls I have deployed. An agent parses the instruction and complies at wire speed. Suspicion was a control too. We just never had to write it down until now.
What to actually do
Prioritized, because a list of twenty undifferentiated controls helps no one. The top three:
- Consequence-tiered human approval gates. Classify agent actions by reversibility and blast radius. Irreversible or high-blast-radius actions (payments, deletions, external communications, permission changes, anything touching production) keep a human in the path, full stop. Low-consequence, reversible actions can run free. This is not all-or-nothing. It is putting the control where the consequence is.
- Least-privilege scoping of agent credentials. Each agent gets its own identity with no shared service accounts, scopes cut to the specific task, short-lived tokens, and independent revocation. The credential design determines the blast radius when an injection lands, and one eventually will.
- Treat every agent input channel as untrusted. Every email, document, web page, RAG chunk, and tool result entering the context is attacker-reachable until proven otherwise. Draw that trust boundary in the architecture diagram explicitly. Most agent findings fall out of drawing it honestly.
Behind those three: sandboxed execution, so the agent's reach is structurally bounded rather than politely requested, and comprehensive action logging covering every tool call, input source, and approval. When an incident comes, "what did the agent do and why" is the first question, and today most deployments cannot answer it.
Residual risk, stated honestly
Now the part vendor decks omit. These controls reduce exposure. They do not remove the underlying condition. Prompt injection remains architecturally unsolved as of mid-2026. Adaptive attacks continue to bypass published defenses, and the joint guidance Careful Adoption of Agentic AI Services, published May 1, 2026 by six Five Eyes cyber agencies including CISA and the NSA [12], reached the same conclusion I have: adopt incrementally, restrict agents to low-risk tasks first, and keep humans at consequential decisions, because you cannot filter the problem away. As long as instructions and data share one channel, injection is a property of the system, not a bug awaiting a patch.
Approval gates inherit human failure modes. Approval fatigue turns a checkpoint into a rubber stamp, and an attacker who studies your tiering will shape actions to slide under it. Least privilege bounds damage without preventing misuse of legitimately granted scope. Logs enable response after the fact, at machine-speed lag. And every one of these controls erodes under commercial pressure to remove exactly the friction that makes them work.
"We added a guardrail" is not the same sentence as "we removed the risk," and an architect should never let the first be presented as the second. What we can honestly say is this: keep the human in the path wherever the consequence is real, understand the mechanism before trusting the product, and expand autonomy only as fast as your ability to observe, bound, and reverse what it does. None of that is new discipline. Trust boundaries, least privilege, consequence-driven design, and residual risk stated in writing are the same fundamentals I built the Cybersecurity Architect's Handbook, Second Edition around, and agents have not changed a single one of them. What agents changed is the price of skipping them. If you want the full treatment, the handbook walks those fundamentals domain by domain and includes an AI reference architecture that puts the controls in this post into a complete design you can build against. And if you do pick up a copy, let me know what you think. Reader feedback shaped this edition, and it will shape the next one. The discipline matters more than the tooling. It always has.
References
- OWASP GenAI Security Project. State of Agentic AI Security and Governance, v2.0, 2026. https://genai.owasp.org/
- McKinsey & Company. The State of AI: Global Survey, 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
- MITRE ATLAS. LLM Prompt Injection, AML.T0051 (sub-techniques AML.T0051.000, AML.T0051.001). https://atlas.mitre.org/techniques/AML.T0051
- OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2025, v2.0, November 18, 2024. Entries cited: LLM01 Prompt Injection, LLM05 Improper Output Handling, LLM06 Excessive Agency. https://genai.owasp.org/llm-top-10/
- Hardy, N. The Confused Deputy (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review, 22(4), 1988.
- NIST National Vulnerability Database. CVE-2025-32711 (M365 Copilot "EchoLeak," reported by Aim Security, June 2025). https://nvd.nist.gov/vuln/detail/CVE-2025-32711
- MITRE ATT&CK. T1078: Valid Accounts. https://attack.mitre.org/techniques/T1078/
- Anthropic. Disrupting the first reported AI-orchestrated cyber espionage campaign, November 13, 2025. https://assets.anthropic.com/m/ec212e6566a0d47/original/Disrupting-the-first-reported-AI-orchestrated-cyber-espionage-campaign.pdf
- NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. https://doi.org/10.6028/NIST.AI.100-1
- NIST. AI RMF Generative AI Profile, NIST AI 600-1, July 2024. https://doi.org/10.6028/NIST.AI.600-1
- NIST. Zero Trust Architecture, SP 800-207, August 2020. https://csrc.nist.gov/pubs/sp/800/207/final
- CISA, NSA, ASD ACSC, Canadian Centre for Cyber Security, NCSC-NZ, NCSC-UK. Careful Adoption of Agentic AI Services, joint guidance, May 1, 2026.
