The Memory: Does Daydreaming Create a Black Hole?

Recently I returned to Calgary to visit my family. Over a delicious curry dinner, my sister and I had a conversation about memory.

My memory is very bad and for many years I believed it was because of some sort of deficiency in my brain or because of stress or perhaps my life wasnt exciting enough to remember. But my sister brought up something interesting.

Maybe I can't remember my parts of my life because I spend so much of it daydreaming.

I mean, when I'm dreaming in sleep,  I can't remember anything. Perhaps it applies to daydreaming. As writers we are often in our worlds and every moment we are in that world we can't consciously be part of our own.

Could it be that simple? What do you think? How's your memory?