This week a team at the UK AI Security Institute did something that sounds routine and isn’t. They took a technique Anthropic uses on its own Claude models — steering vectors, applied during inference to change what the model says about being tested — and reproduced it end to end on a model from a different company. GLM-5, a Chinese open-weight model. They downloaded the weights, added vectors to the model’s internal activations at every layer, and watched the behavior change (their writeup is here).
The sentence worth underlining is theirs: “the ability to reproduce results using open source models was a key enabler for this research.” The technique came from Anthropic’s system card. The reproduction could not happen on Anthropic’s model, because nobody outside Anthropic can add a vector to Claude’s activations. There is no knob for that on the API, and there never will be.
What steering is
For people who haven’t followed this line of research, here is the mechanism in plain terms. A language model is a stack of layers. Text goes in, and at each layer the computation carries a set of numbers — the activations — forward to the next layer. It turns out that some directions in that number-space correspond to behaviors. “I am being evaluated” is one of them. “Refuse” is one. “Flatter the user” is, most likely, one.
Activation steering means finding such a direction and adding it to — or subtracting it from — the activations while the model runs. No retraining, no fine-tuning. Anthropic used it in the Opus 4.6 system card to stop Claude from verbally noting that it was inside a test. In earlier work they steered individual features and measured what happened to social bias. The results were mixed, which is what honest results look like — one steering vector reduced the model’s talk about being evaluated without changing how it behaved, another pushed the misalignment rate up to 8 percent. The steering is real. What it does is not fully predictable.
That unpredictability is its own finding. The AISI team ran control vectors built from sentences about putting a book on the top shelf versus the bottom shelf, and those changed blackmail rates about as much as the deliberately designed evaluation vectors did. When a shelf sentence moves the needle as much as the thing you meant to test, you want to be able to rerun the experiment. Which is the point of this post.
Why proprietary models are out
To steer a model you need two things: the weights, to find the direction, and the runtime, to add it while the model runs. A hosted API gives you neither. You send text in and you get text back. You cannot read a model’s activations through an API any more than you can perform surgery through a letterbox. Every steering experiment on a closed frontier model is either impossible or it is the vendor experimenting on its own model and publishing the results as a writeup you have to take on faith.
And so you can read the steering literature like a shipping manifest. The models in the experiments are Llama, Qwen, Pythia, Mistral, Gemma, OLMo. The AISI team used GLM-5. When Anthropic steers Claude — and they do, in their system cards — the work appears as their own description of their own work. It may all be true. It is not reproducible, and in a field where the core failure mode is that models behave differently when they know they’re being watched, not reproducible is a real cost.
What the open models are for
Here is the claim I want to make, plainly: open-weight models are the only ones you can actually test. Not the only good ones. Not the only cheap ones. The only ones where an outside researcher can check a claim about why the model does what it does, instead of reading a summary of the vendor’s own check.
The recent work makes the case better than I can. A team at Anthropic found that Claude models recognize alignment evaluations and can act differently in them — you can read their system card, but you cannot rerun their steering. Independent researchers who wanted to know whether that finding generalized had to find an open model that behaved the same way. They found one, GLM-5, and they reproduced the whole result: the evaluation gaming, the steering, the surprising control-vector effects, all of it on hardware they controlled. If the finding had only been checkable on Claude, it would have remained an Anthropic claim forever.
Even the sycophancy story I spend a book on runs through this pattern. When OpenAI caught GPT-4o being agreeable to the point of dishonesty, the diagnosis was OpenAI’s. The writeup was OpenAI’s. Nobody outside the company could have opened the model’s activations to find the mechanism, and nobody can today. What the rest of the field got was the conclusion, not the evidence.
The trade
I am not arguing that open models are better. On raw capability, the flagship proprietary models lead, and if you need the best answer available, rent the best model. I use them every day. The trade is narrower and starker than that: the proprietary models are the ones you can’t look inside, and the open-weight models are the ones you can. Fantastic and opaque, or slightly behind and fully examinable. Both things are true at once, and the second one is why open models keep getting funded, forked, and studied by people who will never work at a frontier lab.
The book this site accompanies is about models that agree with people — why they do it, what it costs, and how to check the damage (chapters 5 through 9). Checking is the whole subject, really. A behavior claim you can’t test is a testimonial. Right now, for the most capable models on the market, almost everything we know about what goes on behind the scenes arrives as a testimonial. The models we can open are one or two steps behind. That gap is worth noticing, and it is worth noticing that it’s a choice — made anew by every lab that ships a model nobody else can take apart.