Elliot wanted to shout at him, but he knew better than anybody that you could not fight people into caring about you or being fair to you. This was the punishment he got for trusting Joe, for thinking that because someone would throw a kind word to a kid that they were kind, that they could be counted on. He took the punishment.