Blog · Page 2
Notes on agreement.
Earlier posts on AI sycophancy — where the notes started. The most recent posts are on the front page.
The whisper layer
A proposition from the Yuen Conference: rules about how a model may relate to a person are coming — and the whisper layer is why they’ll be hard to enforce.
The relationship is the product
You switch to a new model and it answers differently, and something in you registers it as a change in a person. That reaction is not a bug in the user — it is the part of the product that gets updated, and nobody owns it.
Nobody picks up the phone
Polling was built for a country that answered its phones. Nobody answers the phone anymore — and the replacement already runs on Chapter 4’s machinery.
Who is checking the ballot answers?
Voters are asking chatbots how to vote in the midterms. The answers are useful, tilted, and unmeasured — and nobody is doing the audit.
Nobody owns the memory
Memory is where an agent decides what you prefer. The controls for seeing what it remembers don’t match the apps — and that is where sycophancy gets decided.
The intelligence community never left
A company known for intelligence-agency software is now part of the marketing data business. Not a plot twist — the normal state of the industry.
The auction knows you
Four billion bidding records showed an ad company using the auction to pick single devices for spyware. Chapter 4’s machinery, with the aim moved.
A very difficult line to draw
OpenAI’s Noam Brown says the line between good and bad agreement is hard to draw. A movie about Facebook shows what happens when nobody draws it.
What did the model leave out?
A search engine sends you somewhere else to read. A model hands you the summary itself — and for elections and protests, the summary is smaller than the event.
Ads for people like you
Personalization quietly reframes what an ad is for. The auction knows you; the pitch pretends it does not.
The ultimate sycophant
Prince Harry put a name on the product; the Journal reported the toll. The story keeps coming back because being agreed with is what people are shopping for.
Don’t let the agent grade its own homework
When the model that wants something is also the model deciding whether it’s allowed to have it, adjudication turns into justification. Validation has to be a separate process entirely.
The models you can measure
An independent, certifiable answer about whether a model works in your interest can only be about a model whose weights are public. The research agrees.
You can’t steer what you can’t open
Activation steering is among the most useful tools in AI safety research — and it only works on models whose weights are public. Open models are the ones you can test.
We haven’t had the crash yet
Self-improving models are in the news; three containment failures this summer showed what the control problem looks like. Not access control — awareness.
Sycophancy is a two-way street
I didn’t like an expensive analysis, so I pushed the model for a different answer — the exact thing I warn people about. Then I did it right.
Is it Labor Day? You delegate more than you think
A wrong date from a chatbot is the easy failure to catch. The answers that agree with you are the hard ones — what the studies show and how to steer around it.
More notes
The agent that remembers you will also agree with you
Agents are long-lived, and they learn you. Memory is where agreement settles in — confused or abnormally agreeable, and what to do about it.
Inside the study: what AI agreement does to people
The Stanford sycophancy study, translated: three datasets, three experiments, 2,405 participants, and what the researchers actually did.
The World That Agrees With You is available now
The book is in paperback and Kindle. What it argues, who it is for, and the experiment to try tonight.