It’s perfectly all right for a character to go into a reverie about other times and places if the memory contains important information—and if the present scene does finally take place. If the memory or fantasy entirely supplants the present time, however, readers will wonder why you bothered to take us to a party in the first place if you weren’t going to let us talk to anyone or dance.
