Positing A Question

So this one is for all of you time-travel junkies out there.


There is an interesting thought posed by John Scalzi's Red Shirts:

Does time travel to the past violate the conservation of matter and energy? (Since the atoms and energy that make up you, or the object, etc. are already in the past- time travel to the past would be introducing matter);

-OR-

Does the conservation of mass and energy apply not from moment to moment, but holistically across all-time space so matter borrowed from the future isn't actually removing or introducing matter from the system as a whole?

I'm actually fascinated by this idea (since I'm of the mind that time travel to the past is pure fantasy or requires so much energy it's impractical).

I'm curious, what do you think?