Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI

by Karen Hao

Loopt was a social network that used location tracking to notify users when they were close to friends or to recommend nearby restaurants. He worked so hard that summer and ate so much instant ramen, he gave himself scurvy. He didn’t regret it. “Work really hard in the beginning of your career,” he would later say to young founders. “It pays off like compound interest.” Altman never returned to Stanford. By late 2005, he and his cofounders were already in talks with VC firms New Enterprise Associates and Sequoia to give them $5 million in funding. Altman took his chances and dropped out of college.

“The first piece of startup wisdom I heard was ‘increasing your sales will fix all problems,’ ” Altman wrote in a 2013 blog post titled “Growth and Government” that thanked Graham and Thiel for shaping his ideas. “This turns out to be another way of phrasing Paul Graham’s point that growth is critical.” For startups, more sales meant more capital meant better talent and fewer internal tensions. For countries, more growth meant more technological innovation meant a higher quality of life. The dysfunction in the US government was threatening this growth cycle, Altman added. “Either you’re growing, or you’re slowly dying,” and the US government was dying. “Without economic growth, democracy doesn’t work because voters occupy a zero-sum system,” he said.

But there is a different way to view this history. Connectionism rose to overshadow symbolism not just for its scientific merit. It also won over the backing of deep-pocketed funders due to key advantages that appealed to those funders’ business interests.

At one point, Welinder, who would become VP of product, commented bitterly that every conversation felt like a reenactment of a 1944 US intelligence manual about nonviolent sabotage. One section of the pamphlet, declassified in 2008, lists simple instructions for how to destabilize and undermine the productivity of an organization, including: Talk as frequently as possible and at great length. Bring up irrelevant issues as frequently as possible. Haggle over precise wordings of communications, minutes, resolutions. Refer back to matters decided upon at the last meeting and attempt to re-open the question of the advisability of that decision. Ask endless questions.

RLHF also became the central technique OpenAI would use in its efforts to teach neural networks to encode factual information and to reliably retrieve it as a way to mitigate hallucinations. It asked workers to repeatedly answer fact-based questions (“Who won the NFL Super Bowl in 1995?”) and downrank inaccurate answers. But in April 2023, John Schulman, one of the scientists on OpenAI’s founding team, would remind the audience during a talk at UC Berkeley that the issue of hallucinations was rooted in the nature of neural networks. Unlike the deterministic information databases of symbolic systems, neural networks would always traffic in fuzzy probabilities. Even with RLHF, which helped to strengthen the probabilities within a deep learning model that correlate with accuracy, there was fundamentally a limit to how far the technique can go. “The model obviously has to guess sometimes when it’s outputting a lot of detailed factual information,” he said. “No matter how you train it, it’s going to have probabilities on things and it’s going to have to guess sometimes.”