I think that the answer lies in the intimate connection between time and heat. There is a detectable difference between the past and the future only when there is the flow of heat. Heat is linked to probability; and probability in turn is linked to the fact that our interactions with the rest of the world do not register the fine details of reality.
