We all can be more thoughtful about our own speech, but it is unjust to treat people as if they are bigots when they harbor no ill will. Doing so can discourage them from being receptive to valuable feedback. It may also make them less interested in engaging with people across lines of difference.
Yes - it's better to give a constructive comment on why there might be an issue with a statement. I suspect self-censorship as a deleterious consequence will be mentioned soon, and that of course prevents us from airing out potentially damaging (but well meaning/thought to be inconsequential) beliefs in open discussion.

