This is why distinct configurations of the world seem equivalent to us.
Between ourselves and the rest of the world there are physical interactions. Obviously, not all the variables of the world interact with us, or with the segment of the world to which we belong. Only a very minute fraction of these variables does so; most of them do not react with us at all. They do not register us, and we do not register them.
