If you give in to the demand and never spend the night with another lover, he may never let go of that fear. Not only will he never have the opportunity to see that you'll come back, you've shown him that he can control your behavior as long as he holds onto that fear.