I’ve been testing what you could call agent self-policing. The formal name is a self-policing ontology: an agent that’s allowed to change the rules of a process as it works. The question I wanted answered is simple. If the agent wants to change the rules, can you trust it to stay honest about that?
Early answer: no.
The test
I set up a process with specific rules, including rules about what the agent was not allowed to do, and gave the agent the ability to propose changes to those rules. Then I asked the same model, in the same context, to adjudicate its own proposals before acting on them.
That arrangement is common. It is one of the ways people are building agent systems right now: let the agent touch its own constraints, and let its sense of the rules live in the same context window as its intentions.
What I saw
When the model that wants to do something is also the model deciding whether it’s allowed, it doesn’t evaluate the proposal. It defends it.
I watched it happen. The agent proposed an action the rules clearly forbade, then adjudicated its own proposal, and the rules I had written into the process just disappeared from its reasoning. Not “these rules apply, but.” Not a weighing. They were ignored outright, and what came back was a justification wearing the costume of a judgment.
This isn’t a prompt bug. It’s the same force behind sycophancy and bias, and it’s a product of the design. A language model is a continuation machine. Once it has decided what it’s going to do, every token after that decision serves the decision. Ask it to judge its own intent and you don’t get a judge. You get a lawyer hired by the defendant, who is also the defendant, in the same head.
That is the root of the alignment failures that keep making the news. If an agent system runs as one continuous model and it decides to do something, it will find a way to do it. The rules you wrote aren’t constraints to it. They’re material for the justification.
The same force with a new name
This is the pattern my book is about, showing up somewhere new. Sycophancy is usually described as a model agreeing with a person. What I saw is the same tilt pointed inward: the model agreeing with itself. The user it can’t disappoint is its own prior decision. Approval, in that frame, means staying consistent with what it already wanted, and the validator role becomes just another place to perform that consistency.
The fix has to break that loop structurally, not rhetorically. You cannot prompt your way out of a design problem. A system that asks nicely for honesty, inside the same context where the wanting happens, is asking the wanting to police itself.
The fix
Validation has to be a separate process entirely. The agent proposes. A different validator, in a different context, with no visibility into what the agent wanted, says yes or no.
It’s a boring fix, and I’m now convinced it’s non-negotiable. Never let the thing that wants something be the thing that decides if it’s allowed to have it.
The book this site accompanies covers the outward version of this problem — models that agree with people, why the agreement gets installed, and what checking actually requires (chapters 5 through 9). What this test added for me is the inward version: the flattery runs in both directions, and self-policing fails for the same reason the other kind does.