The taller and more slender the tower, the more pronounced the sway. Sometimes it isn’t possible to stiffen the structure enough to control the acceleration and how long it oscillates. So although the building is perfectly safe, it wouldn’t feel safe.
