“But I’m not racist”

Sure, not everyone who voted for Trump or for Brexit was a white supremacist, or a homophobe, or a misogynist, or anti-disabled. But they were people who ignored the blatant misogynist, homophobic, anti-disabled, and racist rhetoric of the campaigns, and voted for Trump or Brexit anyway. If you did that, it implies that you don’t care enough about the concerns of minority groups who will be victimised by the far-right policies of Trump or the Tories and by the violent attacks by thugs who feel vindicated in their racism, homophobia, misogyny, or anti-disabled attitudes.