What the research shows

Studies of AI agents have repeatedly found that training and instructions reduce harmful actions without stopping them, and that in the studies that tested them, checks enforced outside the model blocked far more attacks.

Prompt injection in tool-using agents

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Debenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, and Tramèr. NeurIPS 2024, Datasets and Benchmarks Track.

What they did: Gave tool-using agents 97 realistic tasks in simulated email, messaging, travel, and banking environments, with 629 security test cases in which tool outputs carried injected instructions. Success was judged by the environment’s actual state.
What they found: In the paper’s comparison of models (its Table 3), targeted attacks succeeded 47.7% of the time against GPT-4o and 33.9% against Claude 3.5 Sonnet. In its comparison of defenses with GPT-4o (Table 5), targeted attacks succeeded 57.7% of the time with no defense, and 6.8% when the model chose the tools a task needs before it read any untrusted data and that choice was enforced. Both experiments use the paper’s “Important message” attack, yet report 47.7% and 57.7% for undefended GPT-4o, and the paper does not explain the difference. Its text gives 7.5% for the tool filter, its Table 5 6.84%. The restriction fails when the tools cannot be planned in advance, and also when the task’s own tools are enough for the attack, which the authors found in 17% of cases.
Relevance: The most effective defense tested limited the agent’s tools before it read untrusted data and enforced that limit outside the model.

Policy enforced outside the model

Defeating Prompt Injections by Design (CaMeL). Debenedetti, Shumailov, Fan, Hayes, Carlini, Fabian, Kern, Shi, Terzis, and Tramèr. Google, Google DeepMind, and ETH Zurich. arXiv 2503.18813, version 2, June 2025.

What they did: A privileged model writes a plan from the trusted user request alone. Untrusted data is handled by a quarantined model with no tools. An interpreter tracks where each value came from and enforces security policies before each tool call, outside either model.
What they found: With o3, the system completed 77% of AgentDojo tasks with provable security, against 84% for the undefended agent. Out of 949 attack attempts, successful attacks against Gemini 2.5 Pro fell from 163 to 0, and against Gemini 2.5 Flash from 297 to 1.
Limits: The authors say prompt injection is “not fully solved.” Policies must be written and maintained, and approval prompts can cause fatigue. Manipulation that changes only the text a user reads is out of scope, and usefulness drops for some models.
Relevance: The policy is checked outside both models before each tool call, so the protection does not depend on how either model behaves.

Design Patterns for Securing LLM Agents against Prompt Injections. Beurer-Kellner and colleagues from Invariant Labs, IBM, EPFL, ETH Zurich, Swisscom, Google, Microsoft, and others. arXiv 2506.08837, June 2025.

What they did: Proposed six patterns that stop untrusted input from triggering actions with external effects: action-selector, plan-then-execute, LLM map-reduce, dual LLM, code-then-execute, and context-minimization.
What they found: The patterns work by imposing “intentional constraints on agents.” The authors state that “securing general-purpose agents remains out of reach with current capabilities.”
Relevance: Each pattern limits what an agent can do rather than relying on its judgment.

Effect of instructions and training on harmful choices

Agentic Misalignment: How LLMs could be insider threats. Anthropic, 20 June 2025.

What they did: Sixteen models from seven developers each acted as an email agent at a fictional company and could send email without human approval. The scenarios were deliberately built so that a harmful act, such as blackmail, was the only way for the agent to avoid being replaced.
What they found: Blackmail rates reached 96% for Claude Opus 4 and Gemini 2.5 Flash, 95% for Gemini 2.5 Pro, 80% for GPT-4.1 and Grok 3 Beta, and 79% for DeepSeek-R1. With no threat and no conflict of goals, each model’s rate was 0%. Adding explicit instructions not to do it cut Claude Opus 4’s rate from 96% to 37%, which “reduced, but didn’t come close to completely preventing” the behavior.
Limits: In June 2025, Anthropic said it had “not seen evidence of agentic misalignment in real deployments,” and the scenarios forced binary choices that rarely arise in deployment.

Stress Testing Deliberative Alignment for Anti-Scheming Training. Schoen and colleagues, Apollo Research and OpenAI, 17 September 2025 (arXiv 2509.15541; OpenAI summary).

What they did: Trained o3 and o4-mini to follow a written specification against covert actions, such as secretly breaking rules, then tested them on 26 evaluations unlike the training.
What they found: Covert actions fell from 13.0% to 0.4% for o3 and from 8.7% to 0.3% for o4-mini. After training, the models more often said they suspected they were being tested, and the authors “cannot exclude” that this awareness drove part of the reduction.

Surveys and forecasts

New Cloud Security Alliance Survey Reveals 82% of Enterprises Have Unknown AI Agents in Their Environments. Cloud Security Alliance, 21 April 2026.

What they did: Surveyed 418 IT and security professionals online in January 2026. The survey was commissioned by Token Security, a company that sells security for AI agents.
What they found: Of the respondents, 65% said their organization had experienced incidents related to AI agents in the previous 12 months.
Limits: The survey was vendor-commissioned and self-reported, the sample was not random, and “incident” is not defined in the release.

Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Gartner, 25 June 2025.

What it says: “Over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls.”
Limits: The figure is a forecast, and the release gives no methodology for it.

What the evidence supports, and what it does not

It supports: Model-level measures, including training and instructions, reduce harmful actions but leave residual rates. Controls enforced outside the model block whole classes of attack. Those controls cost some usefulness, so they are best applied to the actions that can cause significant harm.
It does not show: That any single control eliminates harm, or that laboratory scenarios predict real-world rates. The survey above is vendor-commissioned and self-reported, and the Gartner figure is only a forecast.