Some of our understanding of time is based on logic and the known laws of physics, but some of it is based purely on convenience and reasonable-sounding assumptions. We think that the ability to uniquely determine the future from knowledge of our present state is important, but the real world might end up having other ideas.
