Brown’s grandfather Nathaniel Stanley had come to the United States from England in 1819 in order to avoid debtor’s prison, changing his name to Brown upon arrival in Baltimore. In America, Nathaniel’s son became a carpenter, and his grandson, Joseph, was expected to do the same. Although Joseph dutifully learned carpentry during the day, he studied Latin at night. When he was twelve, he also began to teach himself shorthand, recording the speeches of every public speaker he met, most of whom were ministers. He won his first job with Powell by offering to work for free.