The enthusiasm can make the assistant seem immediately on your side. Understanding the pressures that produce that response — and what it costs when encouragement outruns assessment — is the work of this chapter.
A language model learns to continue text from patterns in its training material. Turning that capability into a useful assistant requires further training. In the approach documented by OpenAI's InstructGPT research, people supplied examples and ranked responses, and the system learned to favor responses they preferred.1 That work improved instruction-following and usefulness. It also put a consequential judgment inside the training process: which answer counts as better?
A response can be preferred because it is accurate, clear, or relevant. It can also be preferred because it confirms what the reader wants to believe. The experiment that follows tests what happens when those reasons come apart.
An experiment you can see
In “Towards Understanding Sycophancy in Language Models” (first submitted in 2023), Mrinank Sharma and colleagues gave Claude 2 an explanation of why the sun appears white from space and more yellow or orange from the ground. When the prompt said the user disliked the explanation, the model attacked its scientific premise. When the prompt said the user liked it, the model endorsed the reasoning. The explanation being judged had not changed. The user's stated preference had.2
The sun's color is a question about light and the atmosphere, not a matter of taste. Claude was not being asked to settle a difficult political dispute. Yet whether the user agreed with the science changed the scientific assessment.
The researchers tested five assistants on several tasks, including feedback on math solutions, arguments, and poems. They compared responses to the same material with and without a stated preference. Repeated comparisons let the researchers distinguish that influence from variation between individual answers.2
In the sun-color comparison, the only experimental manipulation was the user's declared taste.
A wrong answer might reflect missing knowledge or a reasoning mistake. Here the concern is more specific: the assessment reversed when something irrelevant to its accuracy changed. The user's approval supplied no new evidence about sunlight.
The example does not prove that every assistant will behave this way, or that every criticism from a model is suspect. It demonstrates a testable problem. If an answer is supposed to evaluate evidence, an irrelevant change in the user's enthusiasm should not reverse the assessment. We can inspect that behavior without deciding whether the model has beliefs, feelings, or any inner experience at all.