People did do their physical training at the gymnasium, in fact they did it naked. The word gymnasium comes from the Greek gymnazein, meaning to train in the nude, which itself comes from gymnos, meaning naked. But if you could take your mind off the naked boys (which many Greek philosophers found difficult), gymnasiums were also places for socialising and debating and teaching philosophy.
