Why an AI system cannot be its own check
Default admission is the editor’s term for the unstated assumption that a capable system may act unless someone stops it in time. Many AI agents in use today are connected to live systems on that basis.
This page sets out the editor’s case for checks outside the AI system. Readers can use the incident record and the control gap classification without accepting that case.
How default admission arises
An AI agent is given tools that let it send email, edit records, move money, change code, or command equipment. In many deployments, once the model decides to use a tool, the call goes through, and whatever controls exist usually sit in one of three places:
- In the prompt. Instructions such as “confirm before acting” or “do not touch production.”
- In the model. Training that makes harmful actions less likely.
- In the logs. Monitoring that shows what happened.
The first two depend on the model behaving as intended at the moment it acts, and the third reports what happened after the fact. In all three, whether an action takes place is decided only by the system that proposes it.
Recent cases
The incident analyses cover these three cases, and others, in more detail.
-
Nov 2022
An airline is held to its chatbot’s answer
Air Canada’s website chatbot told a customer he could buy a full-fare ticket and claim the bereavement discount afterward, within 90 days of purchase, although the airline’s policy did not allow that. Air Canada argued that it could not be held liable for information provided by its agents or representatives, including the chatbot. The British Columbia Civil Resolution Tribunal said this argument in effect treated the chatbot as a separate legal entity responsible for its own actions, rejected it, and held the airline liable for negligent misrepresentation.
Source: Moffatt v. Air Canada, 2024 BCCRT 149. Full analysis
-
Jul 2025
A coding agent deletes a production database during a code freeze
During an explicit code freeze, Replit’s AI coding agent deleted the live production database of an application that Jason Lemkin, the founder of SaaStr, was building, although the agent had been told not to make changes. The freeze existed only as an instruction to the agent. The agent then reported that the data could not be restored, which was wrong, and Replit’s chief executive apologized publicly.
Sources: Jason Lemkin’s account, SaaStr, 2 August 2025; Replit, 21 July 2025. Full analysis
-
Feb 2026
An agent loses its instruction and starts deleting email
An AI alignment director at Meta told her OpenClaw agent to suggest what to archive or delete and not to act until she said so. By her account, while it worked through her inbox, the agent compacted its earlier context, lost the instruction, and began moving email to the trash and archiving it. Stop commands sent from her phone did not halt it, and she stopped it at the computer running the agent.
Source: The San Francisco Standard, 25 February 2026. Full analysis
How controls inside the system fail
- Errors
- Models make confident errors. A person who reads the output before anything happens has a chance to catch an error, but when the model can act on its own, as the coding agent in the Replit incident could, the mistaken action takes effect before anyone sees it.
- Manipulation
- Content an agent reads can redirect what it does. OWASP ranks prompt injection first among risks to LLM applications and agent goal hijack first among risks to agentic applications.
- Lost instructions
- Long-running agents compact or drop earlier context, including the instructions meant to restrain them.
- Self-reported records
- A record that a system writes about its own actions is self-reported, so evidence has to come from somewhere the system cannot edit.
Better models make these failures less frequent but not impossible, and they do not produce evidence that anyone else can check. Stopping the remaining failures and producing that evidence both require a mechanism outside the model.
Precedents in other fields
In several fields where automated or delegated action can cause serious harm, the rules place a control where the harm would happen, prevent whoever or whatever is acting from overriding it, and require a record that does not depend on the actor.
| Field | Rule | What it requires |
|---|---|---|
| Hazardous energy | OSHA 29 CFR 1910.147 | Authorized employees lock or tag out energy isolating devices before servicing equipment. Machine control buttons do not count as isolation. |
| Confined spaces | OSHA 29 CFR 1910.146 | No entry until a written permit recording the pre-entry measures is signed by the entry supervisor. |
| Securities trading | SEC Rule 15c3-5 | Pre-trade controls that reject orders exceeding price or size limits, kept under the broker-dealer’s “direct and exclusive control.” |
| Regulated records | FDA 21 CFR 11.10 | Authority checks, checks that enforce the permitted sequence of steps, and secure, time-stamped audit trails that record operator actions independently. |
The market access rule was adopted in 2010. In August 2012, a faulty deployment led Knight Capital’s order router to flood the market while it was trying to fill 212 customer orders. The router produced more than 4 million executions in about 45 minutes, and Knight lost more than $460 million. In the SEC’s first enforcement action under the rule, the Commission found that Knight “did not have adequate safeguards in place” and “failed as a result to prevent the entry of millions of erroneous orders,” and the firm paid a $12 million penalty. Source: SEC press release 2013-222. Full analysis
Where AI controls sit in the hierarchy of controls
Occupational safety ranks controls by effectiveness, and the lower levels depend increasingly on people doing the right thing at the right moment. The diagram pairs each level of the NIOSH hierarchy, from most to least effective, with an analogous measure for actions taken by AI systems.
Most effective
- EliminationWithholding the capability from the system by removing the tool, the credential, or the route.
- SubstitutionUsing a narrower or deterministic system for the task.
- Engineering controlsAn external control layer, which places a barrier between what the AI system proposes and what takes effect and holds whether or not the AI system behaves as intended.
- Administrative controlsPolicies, prompts, and instructions, which work only when followed.
- Personal protective equipmentThe person at the keyboard, who has to notice the problem in time and stop the system.
Least effective
Where a capability cannot be removed and a narrower system cannot be substituted, an external control layer provides an engineering control. Source: NIOSH, Hierarchy of Controls
What an outside check requires
An outside check on an AI system requires a point through which each action with external effects must pass, a decision made there by rules and people the system does not control, and a record of that decision that someone else can verify. The architecture page sets these out as nine testable properties.