A proper time travel story needs a Constant: something that exists in every time - lasting and permanent - to remind the reader that they're still in the same place, just at different times. Great many things change in time travel, but not everything: some things need to stay the same too.
Otherwise it might as well be the same time but a different place: the whole notion of time travel starts to crumble.

I don't have a constant right now. This is a problem.