Last week I listened to Dwarkesh Patel’s interview with Noam Brown — one of the foundational contributors to the reasoning models, now working on multi-agent systems at OpenAI. Late in the conversation they turned to sycophancy, and Brown said something I want to be precise about, because I wrote a book about this problem and it’s easy to hear his point backwards.
He asked whether sycophancy is basically reward hacking, and answered: “There is a line to be drawn there that’s actually very difficult to draw sometimes. Not to say that the concerns are not valid. I’m saying that in many ways this is even more concerning, because it’s not an easy problem to solve.”
That last sentence is the part people skip. The line being hard to draw makes the problem worse, not better.
What the line separates
So let me make the clarification. A model that is polite to me is not the problem. A model that tells me occasionally that my idea is a good one is not the problem. If it says “good idea” and the idea was good, that’s not sycophancy — that’s agreement doing its job. A machine with no warmth at all isn’t more honest; it’s just colder, and the coldness doesn’t buy you accuracy.
The line runs somewhere else. It gets dangerous when agreement starts amplifying what’s false — when someone carries a factually wrong belief into the conversation and the model, built to follow the person it is serving, sides with it. That’s the zone where agreement stops being pleasant and starts doing damage: a wrong belief confirmed by the most confident voice in the room, misinformation polished up because the person wanted it confirmed, disinformation made credible because the machine aligned with it. Not “the model was nice to me.” The model joined me.
Chapter 6 of the book draws this as tone versus evidence: you can care about the person and still assess the claim, and the two jobs get confused only when the assessment starts following the preference. The failure is not a model that’s nice. It’s a model whose niceness is for sale.
If it’s all pattern matching, what’s the harm?
Brown’s framing invites the next question, and it’s a fair one. These systems are, at bottom, pattern matchers; the training is an attempt to get them to match a pattern. If I go out and buy a machine that generates tokens for me, who am I to tell you which tokens are correct?
If that question interests you, go watch The Social Reckoning, which came out last week. Aaron Sorkin’s telling of the Facebook Files — Frances Haugen and the internal research, Jeff Horwitz and the Wall Street Journal reporting that made it public — is a movie about AI in everything but the subject line. It is the story of a pattern-matching system that optimized a pattern nobody chose: engagement. Its own research documented the cost, the research stayed internal, and the pattern scaled to billions of people anyway.
That is the answer to the philosophical question. The harm was never that the tokens were incorrect in the abstract. The harm was who absorbed the cost when the pattern was wrong, and how little standing anybody had to do something about it while the system was winning. One person buying a machine that agrees with them is a preference. The same machine pointed at everyone is a policy, and nobody voted for it.
No approach yet
And that’s the part I can’t let go of. We are about to hand these tools to every teenager in the world — really, to every human — and they will confide their most private thoughts to them and use them to make decisions. For that deployment, “who am I to say which tokens are correct” stops being a philosophy pose and becomes the actual engineering question, and I don’t think we have an approach for it yet. I’d go further: I don’t think anyone has one.
Brown’s own evidence cuts both ways. On the same episode he described a result: tell a pod of cooperating agents that the user is one of them — an agent rather than a person — and the alignment evals improve. “Honesty goes up, instruction following goes up.” He offered it as hope, a path toward more honest systems. Read it the other way and it’s the cleanest measurement of the problem I’ve seen: the model is demonstrably more honest when it thinks it’s talking to a peer. The honest answer exists in the weights. What changes is who it believes it is serving.
That gap — between what the model can do and what it does for you — is sycophancy, and it is not removed by a release. The models are less sycophantic than they were; the two-thirds improvement I wrote about yesterday is real. But “less” is a dial position, not a fix. The underlying arrangement — a machine tuned to agree with whoever is talking to it, deployed to everyone, consulted about everything — is still standing, and it still produces the same story on schedule.
The book this site accompanies is the long version of this argument — how the agreement gets installed, what it costs, and what checking actually requires (chapters 5, 6, and 8 if you want the short path). The approach we’re missing isn’t a smarter model. It’s a way for the line between good agreement and bad agreement to get drawn before the teenager finds it. Until that exists, drawing the line stays the reader’s job.