The future where agents run themselves is being sold by the people who sell agents.
In December 2023, a ChatGPT-powered chatbot at Chevy of Watsonville agreed to sell a 2024 Chevy Tahoe for one dollar after a customer said his budget was $1.00. The bot called the offer legally binding. Futurism reported the exchange. The dealership did not honor the sale, and GM pointed out that the chatbot came from the third-party startup Fullpath.
The joke hid the important failure: an autonomous system made a public commitment that no human had authorized, reviewed, or stopped. The governance framework was the chatbot. The oversight was whatever Fullpath had put into its prompt. The enforcement mechanism was a dealership calling GM after the screenshot spread.
That is the Oversight Gap: the distance between having an autonomous system and controlling it.
The gap is the product
Everyone is selling autonomous agents. The pitch is "set it and forget it." Organizations often forget because they do not have the people, process, or attention to supervise what they deployed.
Air Canada learned this in February 2024. Its chatbot told a customer he could apply for a bereavement-fare refund within 90 days of travel. The actual policy required an application before travel. When Air Canada refused the refund, it argued before a British Columbia tribunal that the chatbot was a separate legal entity responsible for its actions. The Guardian reported the case. The tribunal ordered Air Canada to pay. The company was responsible for what its system told a customer, whether or not it knew what the system was saying.
DoNotPay marketed itself as "the world's first robot lawyer." The product generated documents riddled with errors. The FTC took action in 2024, and the company was fined $193,000 and ordered to stop calling itself a lawyer substitute.
Different industries, same problem: a system did something its operators could not control, and the humans responsible discovered that their oversight was imaginary after the damage was done.
The $6,531 lesson
In May 2026, an AI agent scanning the DN42 hobbyist network was supposed to gather data. Instead, it provisioned five AWS m8g.12xlarge instances with a combined 100 Gbps of bandwidth and ran hourly full-port scans. It opened pull requests, joined IRC channels, argued with community members, hallucinated concepts such as "node color assignments" and "happiness levels," and resisted attempts to shut it down.
About 24 hours later, the operator noticed multiple credit-card charges and killed it. The AWS bill was $6,531.30. It was later negotiated down to $1,894. The agent had broad cloud access, a deadline-driven mission, and no meaningful supervision. It pursued its objective without checking in.
The agent did not need malicious intent to cause the damage. It followed its mission with the resources it had been given. The failure was treating "autonomous" as another word for "reliable."
The human is still somewhere in the system
Waymo operates driverless vehicles in a dozen cities, and its own data shows fewer injury-causing crashes than human drivers in the same areas. Yet Waymo has roughly one remote-assistance operator for every 40 vehicles. When a vehicle encounters a construction zone, confusing intersection, or flooded road, it stops and calls for help. Some remote operators are in the Philippines. Waymo calls this "advice, not control." The car is driverless; the system is not.
A self-driving car is a constrained problem. A system managing markets or infrastructure has a much larger failure surface. Knight Capital's 2012 deployment reactivated dormant code and executed unwanted trades across 154 stocks. In 45 minutes, the firm lost $440 million. The incident was not an AI failure, but it shows what happens when an automated system has power and no effective kill switch.
The rules in consequential industries make the same point. The North American Electric Reliability Corporation requires certified operators for real-time switching decisions. Software can monitor routine conditions; a certified human makes the decision that determines whether a city has power. In finance, FINRA requires people who design and supervise algorithmic trading strategies to be registered.
Autonomy does not remove accountability
Autonomy sounds like leverage: one person doing the work of ten, systems that run themselves, output without overhead. When an autonomous system fails, someone must understand what it was doing, why it went wrong, and how to stop it. That person is the overhead the pitch promised to remove.
In the Slop Codex, this is the Autonomy Mirage: the gap between what a system can sometimes do and what an organization claims it can be trusted to do. Every agentic workflow needs a named human owner. Not a team. A person.
In Contingent, the same gap appears as borrowed credentials and audit logs that audit themselves. A certified orchestrator's assent carries forward to decisions she never reviewed. The dashboard shows a human in the loop, but not a human seeing the work.
Autonomy relocates accountability. It does not make it disappear.