Growing up in Virginia, I thought Lee went with the Confederacy because all of his family, friends, and army colleagues pushed him in that direction. Wrong. Much of his extended family wanted to stay in the Union. Not just in 1861, but for generations past.