If we’re not talking, no one else is, either. Many of us experience our sexual fantasies in isolation (despite the public ubiquitousness of sex). Since we don’t know what others are thinking and doing, we have nothing to compare ourselves with, no way to gauge whether or not we’re normal. We’re afraid of being different and therefore deviant.
