Since your hand interacts with the pan’s handle, you can’t apply the second law to the handle on its own. You must include both the handle and your hand (and, to be more precise, the entire pan, the stove, the surrounding air, and so on). And a careful accounting shows that the increase in the entropy of your hand outstrips the decrease in the entropy of the handle, ensuring that the total entropy does indeed go up.
