If we instead think of rights as a way to stop government from being too overbearing, then it becomes clear that the more rights that are recognized, the less power that the government has overall. When government has more constraints, all people get more freedom from government bullying.
