It is because the Pentagon is a relatively flat and unassuming building, whereas the World Trade Center was a tall phallic totem whose collapse created an immense audiovisual effect. Nobody who saw the images of its collapse could ever forget them. Because we intuitively understand that terrorism is theater, we judge it by its emotional rather than material impact.
