Summer Frost (Forward Collection, #2)

by Blake Crouch

Knowledge is just information, which is subjective.”

In the end, all we have is math.”

The question at hand is—what would an idealized version of humanity want?

We have to program the AI to act in our best interests. Not what we tell it to do, but what we mean for it to do. What the ideal version of our species should want.

“Roko’s basilisk. Have you heard of it?” I shake my head. “It’s an arcane info hazard first posed sixty-four years ago.” “What’s an info hazard?” “A thought so insidious that merely thinking it could psychologically destroy you.”

“The human mind is just patterns of information in physical matter, patterns that could be run elsewhere to construct a person that feels like you. It’s no different from running a computer program on a multitude of hardware platforms. A simulation of you is still you.”

Pascal’s wager, the famous eighteenth-century philosophical argument that humans gamble with their lives on whether or not God exists. Pascal posited that we should conduct our lives as if God were real and try to believe in God. If God doesn’t exist, we will suffer a finite loss—degrees of pleasure and autonomy. If God exists, our gains will be infinitely greater—eternal life in heaven instead of an eternity of suffering in hell.

“Without pain, there’s no beauty, Max. The beauty is worth the price.”