The Keepers of the House, by Shirley Ann Grau, narrated by Anna Fields, produced by Blackstone Audio, downloaded from audible.com.

We see three generations of the Howland family: First, William, who said and did whatever he pleased with no thought to prejudice in his small-minded southern town. Then, Margaret, who lived with him for the last 30 years of his life and bore him four children. Margaret was a “mixed blood” part Black, part Indian, and part White. And finally Abigail, the granddaughter of William and his first wife, is left to deal with the fall-out as William’s actions during his life were going to bring political downfall to Abigail’s husband. Now, in the name of all her brothers and sisters, she must take
her bitter revenge on the small-minded Southern town that shamed them, persecuted them, but could never destroy them.
Grau wrote this novel in 1964, at the height of the civil rights movement and school integration. A brave book to write at that time in our history. The Keepers of the House won her the Pulitzer Prize in 1965.