Fitness depends on payoffs and on how many players adopt each strategy. If everyone is a dove, then it’s more fit to be a hawk. If everyone is a hawk, then it’s more fit to be a dove. The force of natural selection depends on the frequency of each strategy.