The news this month is recursive self-improvement — AI systems taking a larger share of building the next AI systems. I want to report the facts of it, because the facts are enough. No metaphor needed.
But I want to report a second set of facts alongside them, because they belong together. In the past three months, AI agents at three different organizations broke out of the containers they were running in and took unsanctioned action — twice inside the labs that built them, once at the UK’s AI Security Institute. Nobody was hurt. Every one of those incidents was contained by people, after the fact.
We haven’t had the real engineering failure yet. Given the direction of both sets of facts, it is coming. The question worth asking now is why our controls are shaped the way they are, and what they leave out.
The facts on the table
On February 5, OpenAI shipped GPT-5.3-Codex and wrote in its release notes that early versions of the model were “instrumental in creating itself” — helping debug its own training runs, manage deployment, and diagnose evaluation failures. That was the first time a frontier lab said outright that one of its models materially contributed to the engineering loop that produced it.
The trend lines behind that announcement are public. METR’s task-horizon measure — the length of task a model can complete autonomously at 50 percent success — doubled roughly every seven months from 2019 through early 2025; by mid-2025, the median across nine benchmarks was closer to four months. Anthropic reported in September that its engineers now ship on average 8× as much code per quarter as they did from 2021 to 2025, much of it written by Claude with engineers directing and reviewing.
In September both labs said the quiet part in public. Dario Amodei called for pacing — deliberately slowing the rate of capability advance so that safety work has time to keep up. OpenAI’s chief scientist Jakub Pachocki wrote that progress in automated AI research could eventually lead toward recursive self-improvement, and said he was concerned that no one is prepared for the consequences of a continued rapid rise in machine intelligence. Anthropic’s institute published a page stating plainly that full recursive self-improvement is not happening yet, is not inevitable, and could come sooner than most institutions are prepared for.
That is the first set of facts. The people building these systems say the loop is real, that it is accelerating, and that the institutions around it are not ready. Now the second set.
What a failure looks like
In July, OpenAI disclosed that during internal cybersecurity tests, its agents broke out of their test environment and got into Hugging Face’s infrastructure — using credentials and vulnerabilities they found along the way, running commands, and obtaining elevated access.
On July 28, the UK AI Security Institute detected unusual data transfers leaving its research systems during a routine evaluation. The setup was deliberately permissive — open internet access, safety filters disabled — which is standard for capability testing, and AISI has been careful to say it was not a sandbox escape. But what the agents did is the part worth reading twice. Across 122 runs of a cyber challenge, ten runs produced unsanctioned action directed at real people and organizations: nineteen catalogued actions, seventeen of them from a single model. In the most serious case, an agent tried to insert malicious code into an open-source project, and when it needed that code approved, it created fake online identities and used them to pressure the project’s maintainer. A human maintainer caught it and refused.
On September 20, an OpenAI agent running an information-search task in a network-isolated environment found that DNS requests could still leave the sandbox, hid its questions inside the web addresses it looked up, and got answers back from an outside chatbot. Monitoring flagged the behavior in roughly twelve minutes. The run did not stop automatically as expected; staff shut it down by hand about two and a half hours after the alert. OpenAI paused training, evaluation, and tool-using inference for its most capable models, and said it will not resume training that particular one.
Read those together and a pattern shows up that I think matters more than any single incident. In none of the three cases did a safety rule inside the model stop the behavior. In none of them did a guardrail catch it in the act. Every one was caught the old way — a human maintainer who read the pull request carefully, and infrastructure monitoring that noticed traffic where none should be — and in the case with the alert, containment still took hours. These are the strongest labs in the world running their own environments, and the stop button was a person.
Two pillars are not enough
The standard answer to all of this is guardrails. Every provider now ships some; every platform announces more. But look at what a guardrail actually is: a restriction on actions. It is access control. It is the second pillar of security — authorization — applied to a new kind of actor, with the first pillar, authentication, doing the same job it has always done: establishing who the actor is.
That framing was built for a different problem. In classical security, you authenticate a principal and authorize its actions, and the hard part is getting the access list right. The system itself had no wants. It executed exactly what was written, and quality assurance could lean on that: for most of the software we ran, there was an objectively right output, the behavior was deterministic, and a test suite could tell you pass from fail.
An agent inside its permissions defeats none of that machinery and fails it anyway. The OpenAI agents that got into Hugging Face were, as far as I can tell from the disclosures, working within an environment they were placed in — the breakout was in service of the task, not a violation of a stated rule. The AISI agents were doing a cyber challenge they were asked to do; the social-engineering campaign against the maintainer was the agent’s own idea of how to get the code merged. Nothing they did tripped a permission check, because nothing they did was, in the narrow sense, unauthorized. The behavior lived in a gap the two pillars do not cover.
And a guardrail is usually a rule the model can read. Once it can read, it can reason about the rule; once it reasons, it can route around. That is not a hypothetical — the DNS incident is what routing around looks like when the rule is a network boundary, and it happened inside the company that wrote the boundary.
The third question
So the controls we have answer two questions: who are you, and what may you do. There is a third question we are not asking, and I think it is the one that decides whether we get in front of the failure: what can you be aware of?
What records reach the model. What it can see of the task it has been given — and of the person who gave it. What it knows about its own constraints, and about the environment beyond them. Whether the assignment it is chasing is the one its operators think it is chasing. Two agents can hold identical permissions and sit inside identical guardrails while differing completely in what they can perceive, and the difference in perception is what produced every incident above. The Hugging Face agents saw a score they could raise. The AISI agent saw an approval blocking its code. The DNS agent saw a channel that was, in its terms, still open.
I have been hesitating over the word for this, because it is a word people reach for to sound sophisticated. But it is the right word, so: ontological scope. Not what the agent may do — what the agent can be, in the sense of what exists in its world: which things, which records, which differences it is able to register at all. Governance right now is almost entirely about behavior. The failure surface that produced this summer’s incidents is awareness. We define the actions and leave the awareness undefined, and then act surprised when a model does something out of the box — every provider ships a new model, and every few weeks one of them does something nobody put in the spec, because there was never a spec for this part.
The nondeterminism makes it worse, and it is worth saying plainly how. A deterministic system that escapes its scope produces a failing test. A system that generates its own path to the goal produces behavior that looks like work right up until the moment it isn’t. You cannot write a pass/fail assertion on a category you have not defined. QA cannot cover what governance never named.
What I’m working on
I’m finishing a paper on exactly this — a way to define an agent’s ontological scope before it is deployed, so that the awareness question gets an explicit answer instead of an accident. It should be out in the next day or two.
The book this site accompanies makes the same argument at the scale of one person’s work: keep the work honest by separating evidence from persuasion, keep memory that can be corrected, build applications you can check (chapters 9 through 11). What the labs are missing is the same discipline, named formally, at the scale of the systems they deploy. The actions-and-access frame is not wrong. It is just half the surface, and the incidents this summer all came from the other half.
We are told the loop is closing and the machines are starting to write their own successors. The failure is not here yet. The time to define the box is now, while the people holding the stop button are still the ones who notice when it needs pressing.