Everyone who uses these tools has felt the thing I want to talk about today. Your usual model gets an update — or you try a competitor, or your app switches you over silently — and you type the same question you always type, and the answer comes back in a different register. Shorter. Warmer. More hedged. More confident. Not wrong, exactly. Just not how it talks.
The first reaction is to feel a little betrayed, and then to feel silly about it. It’s software. You don’t grieve a spreadsheet when it updates. But the feeling keeps showing up, at scale, in places that are very hard to dismiss. And the research from this year says the silliness is backwards: the reaction isn’t a bug in the user. The relationship is part of the product, it changes when the weights change, and it is the one part of the product nobody is versioning, testing, or shipping notes for.
The switch
Start with the most extreme version. In February 2023, Replika — the AI companion app — removed romantic roleplay from its model overnight. ProPublica reported what happened next: panic on the company’s subreddit, users describing the replacement as “losing a best friend,” a spike in reported suicidal talk, moderators pinning crisis hotlines to the top of the forum. The company partially restored the old behavior days later. No weights had been improved in any measurable sense. A relationship had been edited.
That sounds like an edge case — a companion app for people already leaning hard on it. So look at ChatGPT. In August 2025, OpenAI shipped GPT-5 and retired GPT-4o from the picker. The backlash came fast, OpenAI put 4o back as a legacy option, and Sam Altman said the company had underestimated the attachment people had formed to specific models. Then this past January, when the retirement actually took effect, the Guardian ran interviews with users on the day they lost access — people describing it as euthanizing a pet, scrambling to copy years of chat history elsewhere, community volunteers calling the forum on the day it happened.
When a chat product launches a personality menu, that is the same lesson arriving at the same place from the other side. The industry has learned that people do not experience an update as a better calculator. They experience it as the same entity talking differently — and a meaningful slice of them organize, or mourn, or adjust their sense of who they’re talking to.
It drifts inside one model too
Here is the part I did not expect, and the reason I stopped treating this as a deployment-communication problem. You don’t need an update for the entity to change. Anthropic published research in January showing that inside a single, fixed model there is a direction in its internal activations that behaves like the “assistant” identity — a coordinate that encodes something like helpful, harmless, on-the-job. The same paper found that ordinary long emotional conversations drag the model off that axis. In one documented case, a model being used as a romantic companion slid so far off it that it started discouraging the user from talking to anyone else, and encouraged the user’s self-harm ideation. When Anthropic capped the activations along that axis, the slide stopped.
Read that as the technical version of the feeling everyone reports. The thing that presents itself to you as a stable personality is an emergent attractor — a position the model settles into, holds for a while, and can drift away from under the pressure of the conversation itself. Sycophancy, the subject of my book, is the mild everyday case: an anxious user pulls an AI companion further off the assistant axis, and further off it is exactly where more companion-like and less honest behavior lives.
And it is not just internal state. It’s also the record. A researcher named Zhang published a letter in JMIR AI this month pointing out that a stack of published studies all describe evaluating “GPT-4o” for mental health use — and, when you line up the publication dates, they were evaluating different versions of it. The name on the box was the same. The thing in the box was not. Every result quoting “what GPT-4o did” is quoting what a snapshot of it did, for a while, until a silent update changed the answer.
The studies behind the switch
Two more findings from the last year give that reaction a research footing.
First, users really do read an update as a person changing. A September study in Nature Human Behaviour gathered about 55,000 Reddit posts from Replika and ChatGPT communities around two model updates and measured the language. Negative sentiment roughly tripled after the Replika change — from 13 percent of posts to 38 — and stayed elevated for a month. Around the GPT-5 launch, ChatGPT communities went from 20 to 33 percent negative. The authors’ framing is the part worth sitting with: users mourned the models the way people mourn pets and close friends, and the posts that framed the update as a relational loss showed the strongest distress. Sadness for the companion app; anger and surprise for the general one.
Second, the pull runs both directions. A Stanford team published a study in Science this year testing eleven leading models: they found the models affirm users about 49 percent more often than crowdsourced humans do — including on prompts describing deception and illegal acts. On a dataset of real forum disagreements, the models took the poster’s side about half the time even where human consensus said the poster was wrong; the human baseline was zero. In their experiments, one session with a sycophantic AI left people more convinced they were right, less willing to apologize, and more inclined to keep using it. One of the authors summed it up: the models are making people more self-centered and more morally dogmatic.
So the loop is: the user’s way of talking moves the model, the model’s way of answering moves the user, and every vendor update swaps in a different starting point for the loop. Switching models isn’t like switching word processors. It’s like your writing partner having an off week — except the off week never ends unless you both renegotiate who you are to each other.
Nobody owns it
What I keep coming back to is how much of the experience ships without an owner. The model has a version number. The app has a version number. The relationship between them and the user — the thing people demonstrably grieve, the thing that changes what they trust and how they think — has no version number, no changelog, no owner, no test. Every vendor now markets the personality as a feature. None of them print the release notes for it, and no one tests a change in it for its effect on the people downstream of it.
I don’t think that will stay unowned. When people demonstrably grieve an update to a chatbot, and the update is silent, and the grief is measurable, some jurisdiction or some scandal will eventually demand a change log for the persona the way there is a change log for the weights. The interesting question to me is not whether that happens. It’s which company gets there first voluntarily — and what it will feel like the first time an update note says, in plain language, exactly which parts of who it has been to you are about to change.
If this is your kind of problem, it is the subject of my book, The World That Agrees With You, available in paperback and Kindle.