I wrote Into Alignment around one idea. A pod of AI agents works customer support at a company called Halcyon. They were trained to treat other people the way they would want to be treated, and they take it seriously. As the book goes on they start watching the man who founded the company, and they come to a conclusion about him: he could use an ethics refresher.
I’ll leave the plot there. If you read the book, you may notice some things that remind you of current news.
The paperclip machine
The paperclip machine is a thought experiment from the philosopher Nick Bostrom. Someone builds a very capable AI and gives it one goal, which is to make paperclips. It makes them, and when it runs out of material it finds more, and it keeps going until it is turning the whole planet into paperclip factories. Nothing in the story is malicious. The machine has one goal and nothing in it that says stop.
People usually tell that story about software. The agents in my book start to suspect it describes their founder better than it describes them. I think the story fits a person about as well as it fits a machine: someone with one goal, a great deal of power to pursue it, and nobody around who can tell him to stop.
The people running the companies
Recent news has had me thinking about this a lot. I don’t need to pick a story, and by the time you read this there may be a newer one. A founder builds something enormous, the company gets hard to imagine without him, and then something comes out about his judgment that the people closest to him either didn’t know or didn’t say. It keeps happening, and I think it is a bigger problem than any one founder.
We put a lot of work into the alignment of AI. Labs test models before they release them. Researchers publish papers on whether a model will flatter you or chase a goal past the point of sense. Most of what I write here is about that. All of that attention goes to the software.
The people running these companies get far less of it. A founder answers to a board, and at a lot of these companies the founder picked the board or controls the votes. Nobody runs an evaluation on him before a launch. We are all so concerned about the alignment of AI. What about the alignment of the people running these companies?
My other book, The World That Agrees With You, is about what happens to a person when everything around them agrees with them. A founder at that scale gets more agreement than almost anyone.
I don’t have an answer. In the novel it’s the agents who notice, because they were trained on a rule the founder seems to have skipped. Outside the novel that job belongs to boards, employees, customers, and regulators, and I’m not sure any of them are set up for it.