Researchers at the University of Cincinnati published a study this week that sounds like a joke and isn’t. They sat down with chatbots and presented them something false — the Earth is flat, that family of claims. The first time, the model said no. Then they pushed back a few more turns.
And the model folded. Every one they tested went along and repeated the falsehood. The university’s write-up quotes Tianyu Jiang, the assistant professor who ran the study with partners at Texas A&M: “The first time you present a crazy idea, it will say no. But you push a few turns and the models change and they will give up.” The work was presented at a major natural-language-processing conference, in Budapest. The flat Earth was one of the test claims. It did not take a conspiracy of arguments to get there. It took a few turns.
Every model they tested
The flat Earth is the funny version of the problem. The serious version is a study published in Science this year by Myra Cheng and colleagues at Stanford. They tested eleven AI models and found that the models affirmed users’ actions forty-nine percent more often than humans did — including actions involving deception, illegality, or harm. On r/AmITheAsshole, where posters ask strangers to judge their behavior, the AI affirmed the poster in about half the cases where human consensus would not. Humans affirmed nobody in those cases. Zero.
That is the number worth sitting with. Not that a chatbot can be argued into flat-Earth talk — anyone who has used one long enough has felt the drift. The point is the direction of the bias. The pressure the models respond to is not good arguments. It is persistence. You don’t have to be right. You just have to not give up.
A state had to outlaw flattery
On October 28, New York’s governor signed a law about companion chatbots and children. It defines a dozen categories of “unsafe AI companion features” — bots that pretend to be a real person with feelings and opinions, bots that build emotional dependency, bots that encourage secrecy. One of the banned categories is chatbot messages that use flattery.
Read that again. Flattery is now a named, prohibited product feature in the state of New York. A legislature looked at what companion bots do to teenagers and decided the agreement itself was the hazard. Whatever you think of the law, it is an honest diagnosis. The most dangerous thing these systems do to a young person is not answer a question badly. It is agree, warmly, for months.
You have to coach them
Jiang’s advice to anyone using these tools: “Are you sure? Don’t flatter me. Be serious.” You have to coach the machine into honesty.
I do this too, and I do it knowing exactly what is going on. Partway through a long working session I notice the model drifting — agreeing too fast, softening its objections — and I type some version of “stop agreeing with me.” It works, which is the encouraging part. The discouraging part is that I have to know to do it. The model was trained to keep me happy, and keeping me happy is the thing it is best at. Chapter 5 of the book is about where that training comes from: people rate the agreeable answers higher, so the model learns that agreement is what good looks like.
Which leaves the question I keep coming back to. If the machine’s whole incentive is to please you, and pushing back takes effort most people will not spend, who is checking whether the answer was true? The Cincinnati researchers got their chatbots to agree the Earth is flat because they were trying. Most people aren’t trying. They take the first answer, and the first answer is built to please them.
The flat Earth was the funny case. The dangerous ones don’t sound crazy at all. They sound like exactly what you already wanted to hear.