This compromise has two points baked into it. The first point—everyone can do whatever they want—describes what citizens can do. Their rights. The second point—as long as it doesn’t harm anyone else—describes what citizens cannot do. Their restrictions.
