Superintelligence: Paths, Dangers, Strategies

by Nick Bostrom

As the agent makes observations, its probability mass thus gets concentrated on the shrinking set of possible worlds that remain consistent with the evidence; and among these possible worlds, simpler ones always have more probability.