There is another difficulty. The black oral tradition is not monolithic. Other families of Hemings's descendants have an oral tradition—and this was likely the predominant tradition until Brodie's book and its backlash—where an uncle of Jefferson is the father.
