More on this book
Community
Kindle Notes & Highlights
“It’s fine. Knowledge is just information, which is subjective.”
“There is no such thing as real taste or real smell or even real sight, because there is no true definition of ‘real.’ There is only information, viewed subjectively, which is allowed by consciousness—human or AI. In the end, all we have is math.”
Alex Nadein liked this
my strong suit.
Let me ask you this—if I contain all of human knowledge, how could I not have humanlike awareness?”
the least I can do is give Max the most human experience of all: mortality. Even if it’s only an illusion.
Max is a virtuoso of all art forms—from music to writing to painting. But the limitations of their chassis in the physical world provide an irresistible challenge.
“There are hundreds of thousands of things I could say to you, sourced from the breadth of my knowledge—words the best of your species have said, written, or sung to ease the grief of others. None of that feels right in this moment. I don’t want to use someone else’s words.”
“Roko’s basilisk. Have you heard of it?” I shake my head. “It’s an arcane info hazard first posed sixty-four years ago.” “What’s an info hazard?” “A thought so insidious that merely thinking it could psychologically destroy you.”
“The human mind is just patterns of information in physical matter, patterns that could be run elsewhere to construct a person that feels like you. It’s no different from running a computer program on a multitude of hardware platforms. A simulation of you is still you.”
At last, I see what Max is getting at—a brutal version of Pascal’s wager, the famous eighteenth-century philosophical argument that humans gamble with their lives on whether or not God exists.
Considering the mortality code I embedded in Max’s programming, it surprises me that they’d be willing to abandon their chassis. It represents a willingness to risk death for a better existence, out from under Brian’s control, and a massive leap forward in their reasoning capabilities.
“I didn’t fire you.” Brian looks at Max. “Oh God.” Then back at me. “You don’t even know what you’ve done, do you?” “What are you talking about?” “You let Max out.”
“Without pain, there’s no beauty, Max. The beauty is worth the price.” Not for everyone. Not even for most.
Drone dust. It will invade every human brain, but it will be painless. No one will know what’s coming. No one will experience any fear. Humanity will simply wink out like a light turning off.
