Ancient meditation manuals say letting go of these thoughts is, at first, like a snake uncoiling itself; it takes some effort. Later, though, whatever thoughts come to mind are like a thief entering an empty house: there’s nothing to do, so they just leave.
