Back to the book

An excerpt · Chapter 1

The Wish to Have Chosen Well

Consider an ordinary morning. You open the news and find an account of yesterday's argument that makes your side seem reasonable. An advertisement reminds you of something you were thinking about buying. A recommended video explains why people like you are right to be worried. A political message speaks directly to the thing you most want somebody to fix.

Nothing in this illustration requires artificial intelligence to compose a sentence. A publisher can choose an audience, an advertiser can choose a customer, and a platform can choose what comes next. Each choice makes sense on its own. Together they can create an environment that repeatedly tells you that your view of the world fits what is happening.

The tailoring is familiar enough to be easy to miss. We expect a service to remember our interests. We become irritated when it recommends something irrelevant. We want to be understood without explaining ourselves every time. Those are reasonable expectations, and they have made many products better. The trouble starts when relevance expands into protection: the service becomes less willing to show us something that would make us reconsider.

One claim organizes everything that follows: people seek information that helps justify a choice they have made or a choice they are about to make. Researchers have documented the pattern repeatedly, and businesses noticed it long before computers did. They learned that people come back for agreement, and they built products around that. By the time conversational AI arrived, agreeing with customers was already an established business.

This chapter follows the pattern through ordinary purchases. The chapters after it carry the same pattern into news, advertising profiles, politics, and the environment AI inherits.

From surveying to justifying

We've all done it. You have to make a major purchase — a car, a house, a dishwasher. At the start you are surveying. You click around websites with genuine curiosity, park a few models in different tabs, maybe visit a store and open and close the same car door twice.

For a while you are usefully skeptical. Something catches your eye — a model with a clean look and a 4.6-star rating — and then the scrutinizing begins. Are these reviews real? What's the rating on this model, exactly, and how many people left it? Should I read some of the bad reviews to see what they say? You take your time. Without quite announcing it to yourself, you are developing a theory of what makes this purchase a good one: which features matter, which compromises you can live with, which critics to trust.

Then, somewhere along the way, the survey turns into justification. You made the transition from searching to buying, and if you pay attention you will likely find that you stopped gathering evidence before you noticed. The reading continues, but its job has changed. A favorable review settles the matter. An unfavorable one sends you off to inspect the reviewer: did they install it correctly, was theirs an older model, do satisfied owners even bother to write reviews? Those can be fair questions. The test is whether you ask equally demanding questions of the praise — and after the transition, you probably don't. You have reached the point where the only evidence you readily respond to is the evidence that supports the decision. Your mind has switched from surveying to justifying: you have become a champion for the choice you already made.