Explain that you're going to start slowly and that you will choose whom to delegate to based on what the direct is good at, likes, wants, and needs to do in terms of performance and goals. Remind them again that when you ask them to accept the delegation, it's okay to say no.
