It’s not about nerfing him so much as it is about thinking “what’s the logical conclusion of X happening?”
In my opinion, draining happy memories pretty thoroughly equals draining XP from an adventurer.
I think giving him chocolate would cheer him up, but it wouldn’t bring his XP back. He has to do some combat and/or roleplaying and/or questing to do that.