Congress denied the right to vote and hold office to any former Confederate who did not sign an oath of allegiance to the Union. Many refused and stayed away from the polls, diluting white conservative voting strength just as blacks were on their way to winning the right to vote.