The Villain and the Shadow
“Shadow” is a term used most commonly in Jungian therapy and inner work. It means that part of our psyche that we have repressed, usually out of shame and the refusal to admit that such elements (the regret that we had children, say, or the rage we carry against “good” or honorable entities) are part of us.
We’re ashamed of our shadow. We don’t want to see it.

John Wayne as Ethan Edwards in “The Searchers”
We reject it. We deny its existence. We banish it from our self-conception.
“Shadow work” in the Jungian sense is the introspection that shines a light on these repressed parts of ourselves and allows us to see them, accept them, and integrate them into our greater consciousness.
Consider this definition of a hero in the narrative sense:
The hero is the individual who does his or her shadow work.
Have you ever read Richard Rohr? He’s a Franciscan monk and one of the writers and thinkers I respect most. I’m reading his book, Falling Upward, right now. The book’s subtitle is “A Spirituality for the Two Halves of Life.”
Shadow work, in Richard Rohr’s lexicon, is the interior labor of consciousness that we all must perform if we are to evolve successfully from what Rohr calls “first-half-of-life work” (constructing an identity, building a physical and financial personal establishment, defining boundaries, etc.) to that consciousness that should and does come to the fore in the second half of our time on earth (a sharing with others, generosity, contemplation, a “being” rather than a “doing.”)
Which brings us to our ongoing subject: villains.
Odysseus [writes Richard Rohr] learned from his shadow side too. Some call this pattern the discovery of the “golden shadow” because it carries so much enlightenment for the soul. The general pattern in story and novel is that heroes learn and grow from encountering their shadow, whereas villains never do. Invariably, the movies and novels that are most memorable show real “character development” and growing through shadow work. This inspires us all because it calls us all.
In John Ford’s 1956 Western, The Searchers, the John Wayne character, Ethan Edwards, carries his homicidal hatred of the Comanche Indians vividly in the forefront as he and his nephew Martin Pawley (Jeffrey Hunter) search over years for Ethan’s abducted niece Debbie (Natalie Wood), who was kidnapped as a child by the war chief “Scar” (Henry Brandon). It’s clear throughout the story that Ethan cannot and will not forgive Debbie for becoming, even if totally against her will, a participant in and member of that (to him) savage society.
ETHAN EDWARDS
Injun will chase a thing till he thinks he’s chased it enough. Then he quits. Same way when he runs. Seems like he never learns there’s such a thing as a critter that’ll just keep comin’ on. So we’ll find ’em in the end, I promise you. We’ll find ’em. Just as sure as the turnin’ of the earth.
When Ethan does eventually catch up to Debbie, now a young woman and one of the wives of “Scar,” and chases her on foot as she flees, we in the audience are certain he intends to kill her, or at least exact some form of violent vengeance.
(Spoiler alert: he doesn’t.)

Natalie Wood as Debbie in the fatal confrontation in “The Searchers”
The hero has encountered his own shadow and learned from it. He has changed. Somewhere along the trail, Ethan has become aware of his demons and been altered by them. His consciousness has evolved beyond eye-for-an-eye savagery to something kinder and nobler.
The Searchers is one of the greatest Westerns ever. In 2008 the American Film Institute named it #1 (I’d put it at #2 behind Shane), partly because of its moral structure. For a good portion of the story (screenplay by Frank Nugent from a novel by Alan Le May), Ethan is, if not the literal villain, then certainly no better than the villains he pursues (he mutilates a slain Comanche and is brutal to virtually every character he encounters including his own kin) in the sense that he cannot and does not change.
He becomes the hero only at the very end, by his ultimate act of clemency.
Richard Rohr put his finger on it.
. . . heroes learn and grow from encountering their shadow, whereas villains never do.

I am grappling with how to apply your ideas about villains and change in a memoir, where my father “character” is not a “pure” villain—he knew the difference, learned his lessons (sort-of), but stubbornly refused to make right decisions based on what he learned.
He evolves in his career, but his adaptations to changing personal and family circumstances are constrained by a refusal to own up to his violent abuse of his children. Call it Irish obstinacy, or revenge on us for fleeing from him? Also complicated by my desire to not make him a cartoon villain, to show him as human.
A related issue is how to damp down the repetition, since he repeated himself again and again. At times, near the end, he actually made behavior changes that showed he had evolved, and knew, understood, but when confronted directly reverted to stock, stonewalling responses. Some of the probs with my manuscript, according to advance readers, hinge on this.
I ultimately added a coda, in which I explain why I cannot forgive him, since he will not admit anything, but I do the next best thing: I imagine him at fourteen, before polio and unwanted children and all the rest. I put him on his parent's farm, racing with his sister and cousins across a field. As a standalone piece it was published in an anthology, and it full of grace notes, but it is not quite enough for the book.
Villains can be messy and still be villains, right? Certainly in real life.