At each stage, we pulled a bit of labor out of agriculture and into new industries that managed, leveraged, or improved the very agriculture the labor had originally come from. The steadily increasing levels of labor specialization and urbanization first gave us towns, then city-states, then kingdoms, and eventually empires.
