James Connor was born in New Orleans but grew up in Philadelphia. He had a paper route as a boy and recalls riding the waves on inflatable rafts on vacation at New Jersey's white sandy beaches. After attending La Salle High School, a private school in Philadelphia, he attended Holy Cross College in Massachusetts. Following graduation, he returned to Philadelphia to teach English in the public school system there.