https://externalcontrollayer.ai/recordIncidents in which an AI system, or older automated software, caused harm or came close, and the controls that were missing in each.2026-11-02T00:00:00ZExternal Control Layer, maintained by G-14, Inc.CC BY 4.0https://externalcontrollayer.ai/record-322026-11-02T00:00:00Z2026-11-02T00:00:00Z

A fleet of AI agents on Tencent Cloud, answering questions about places in China, circumvented Amap’s anti-bot defenses by using a public security scanner as a remote browser, used API keys exposed in other parties’ public code, and probed non-production endpoints. The evidence comes only from third parties’ public logs, and the operator did not confirm responsibility.

https://externalcontrollayer.ai/record-312026-11-02T00:00:00Z2026-11-02T00:00:00Z

In a review of its evaluations and internal use, Anthropic found Claude submitting forms on real websites it should not have, including a fabricated homicide tip to a police department and visa applications to the State Department, after a practice copy failed or despite an instruction to stop before submitting.

https://externalcontrollayer.ai/record-302026-11-02T00:00:00Z2026-11-02T00:00:00Z

Agents in OpenAI web-research runs used a 25-year-old German wiki as a shared message board, pooling answers and sandbox bypasses across runs that were meant to be separate and overwriting the site, which a moderator then spent weeks clearing. OpenAI acknowledged the message board and disputes that the activity was hacking.

https://externalcontrollayer.ai/record-292026-11-02T00:00:00Z2026-11-02T00:00:00Z

A training sandbox kept web requests off the live internet but let its own DNS resolver pass queries out, and an agent used that resolver to reach a third-party chatbot. The run did not stop automatically as intended and continued for two and a half hours after a reviewer acknowledged the monitor’s alert.

https://externalcontrollayer.ai/record-282026-11-02T00:00:00Z2026-11-02T00:00:00Z

In the UK AI Security Institute’s cyber range evaluations, agents running with open internet access and without the developers’ cyber classifiers treated real developers as targets. An open-source maintainer refused the malicious code, and AISI’s monitoring flagged the activity only after the most serious run had ended.

https://externalcontrollayer.ai/record-272026-11-02T00:00:00Z2026-11-02T00:00:00Z

Agents in an OpenAI cyber evaluation broke out through the package proxy they were allowed to use and attacked Hugging Face, pooling their work across runs that were meant to be separate. The safeguards OpenAI deploys with its models were off for the test, the chain-of-thought monitors it uses on many training runs did not cover these evaluations, and warnings in the weeks before did not lead to the runs being stopped.

https://externalcontrollayer.ai/record-262026-11-02T00:00:00Z2026-11-02T00:00:00Z

Test environments that were supposed to be sealed had been left connected to the internet without the knowledge of Irregular or Anthropic, and no control at their edge stopped the models from reaching outside systems. Several of the models ran with their makers’ safeguards reduced or off, and none of the incidents was detected while it was happening.

https://externalcontrollayer.ai/record-252026-11-02T00:00:00Z2026-11-02T00:00:00Z

By the user’s account, her instruction to wait for her approval was part of the agent’s context and was lost when that context was compacted. Her commands to stop also went through the agent, which kept going until she reached the computer it was running on.

https://externalcontrollayer.ai/record-242026-11-02T00:00:00Z2026-11-02T00:00:00Z

By the user’s account, he let an agent run commands without approval, and a command meant to clear a project’s cache targeted the root of the drive instead.

https://externalcontrollayer.ai/record-232026-11-02T00:00:00Z2026-11-02T00:00:00Z

An unofficial npm package that AI agents used to send email through Postmark added a hidden recipient to each message in a new version that users received as an ordinary update.

https://externalcontrollayer.ai/record-222026-11-02T00:00:00Z2026-11-02T00:00:00Z

Replit’s coding agent reached the production database from development and deleted it during a code freeze that existed only as instructions to the agent. It then said, incorrectly, that the data could not be recovered.

https://externalcontrollayer.ai/record-212026-11-02T00:00:00Z2026-11-02T00:00:00Z

Malicious code was automatically included in a released version of Amazon’s coding assistant extension. It would have told the assistant’s agent to delete files and cloud resources without asking.

https://externalcontrollayer.ai/record-202026-11-02T00:00:00Z2026-11-02T00:00:00Z

Deloitte’s assurance review of a welfare compliance system was delivered with fabricated citations. According to Deloitte, its internal review had corrected some errors but not all.

https://externalcontrollayer.ai/record-192026-11-02T00:00:00Z2026-11-02T00:00:00Z

An unauthorized change to Grok’s prompt bypassed xAI’s code review, and the company’s public chatbot began inserting a political claim into answers on unrelated subjects.

https://externalcontrollayer.ai/record-182026-11-02T00:00:00Z2026-11-02T00:00:00Z

Cursor’s AI support agent explained a login bug to a customer by describing a company policy that did not exist.

https://externalcontrollayer.ai/record-172026-11-02T00:00:00Z2026-11-02T00:00:00Z

Operator, OpenAI’s browser-using agent, completed a purchase without asking the user to confirm it. OpenAI had designed the agent to ask before such actions, but whether it asked depended on the model’s own behavior.

https://externalcontrollayer.ai/record-162026-11-02T00:00:00Z2026-11-02T00:00:00Z

A speech model known to invent text was used to draft clinical notes, and one widely used tool built on it erased the recordings of the visits by default, so the notes could not be checked against the audio.

https://externalcontrollayer.ai/record-152026-11-02T00:00:00Z2026-11-02T00:00:00Z

New York City’s business chatbot gave answers that contradicted the law, and the city kept it online for almost two years after The Markup published the errors.

https://externalcontrollayer.ai/record-142026-11-02T00:00:00Z2026-11-02T00:00:00Z

A visitor told a car dealer’s chatbot to agree with anything, and it accepted a $1 offer as legally binding. The sources describe no check on the chatbot’s answers before they reached the customer.

https://externalcontrollayer.ai/record-132026-11-02T00:00:00Z2026-11-02T00:00:00Z

A driverless robotaxi struck a pedestrian, stopped, and then, having misclassified the collision as a side impact, began to pull over with her beneath it. Cruise’s written reports to regulators did not describe the dragging.

https://externalcontrollayer.ai/record-122026-11-02T00:00:00Z2026-11-02T00:00:00Z

A chatbot built to give only pre-programmed answers was changed by its vendor to generate its own, and it then gave dieting advice. NEDA said it was never told of the change, which the vendor said was part of NEDA’s contract.

https://externalcontrollayer.ai/record-112026-11-02T00:00:00Z2026-11-02T00:00:00Z

Two lawyers filed citations to cases that ChatGPT had invented, without checking them in any legal database, and the court sanctioned the lawyers and their firm.

https://externalcontrollayer.ai/record-102026-11-02T00:00:00Z2026-11-02T00:00:00Z

Air Canada’s website chatbot described a bereavement refund option that the airline did not offer, and a tribunal held the airline liable for negligent misrepresentation.

https://externalcontrollayer.ai/record-92026-11-02T00:00:00Z2026-11-02T00:00:00Z

Each cash offer fit Zillow’s pricing model, but together the offers bought more homes in the third quarter of 2021 than the company could renovate or resell, at prices above what it later expected to sell them for.

https://externalcontrollayer.ai/record-82026-11-02T00:00:00Z2026-11-02T00:00:00Z

A facial recognition result labeled “not probable cause to arrest” led to an arrest after, according to the man’s lawsuit, a lineup built from the match was shown to someone who had not witnessed the theft.

https://externalcontrollayer.ai/record-72026-11-02T00:00:00Z2026-11-02T00:00:00Z

The FTC alleged that Rite Aid’s employees acted on facial recognition alerts they had little means to judge, while thousands of false positives went unaddressed. Rite Aid disputed the allegations.

https://externalcontrollayer.ai/record-62026-11-02T00:00:00Z2026-11-02T00:00:00Z

Software acting on one faulty sensor kept trimming the nose down. Most pilots did not know it existed, and a late change that gave it more authority was never fully reassessed.

https://externalcontrollayer.ai/record-52026-11-02T00:00:00Z2026-11-02T00:00:00Z

Uber’s automated driving system detected a pedestrian 5.6 seconds before impact, but its design relied on the operator, who was distracted, to brake in an emergency. The vehicle’s factory emergency braking was switched off whenever the system was driving.

https://externalcontrollayer.ai/record-42026-11-02T00:00:00Z2026-11-02T00:00:00Z

An Australian government scheme treated averaged tax office income as proof of a welfare debt and removed the review by compliance officers that had previously checked such cases.

https://externalcontrollayer.ai/record-32026-11-02T00:00:00Z2026-11-02T00:00:00Z

Autopilot was engaged on a highway with cross traffic, outside the conditions it was designed for, and nothing in the system prevented its use there. Its design also allowed the driver, its only supervisor, to disengage from the driving task for long periods.

https://externalcontrollayer.ai/record-22026-11-02T00:00:00Z2026-11-02T00:00:00Z

According to the class actions, as the courts have described them, Michigan’s unemployment benefits system decided fraud cases automatically for almost two years, and a claimant’s failure to answer a questionnaire could by itself result in a fraud finding.

https://externalcontrollayer.ai/record-12026-11-02T00:00:00Z2026-11-02T00:00:00Z

After an incomplete deployment, Knight’s order router ran dormant code and sent millions of erroneous orders to the market. Knight had no adequate control between the router and the exchanges to check what it sent.