The Phoenix Project

by Gene Kim

I also know that the workload in my organization is totally out of control. No amount of heroics on my part can make a big dent in the tidal wave of work that’s been allowed to get into the system. Because no one ever said no. Our mistakes were made long before it came to me. The mistakes were made by accepting the project and all the resulting shortcuts that Chris had to make before it reached me.

Looking at John, he continues, “Remember, Jimmy, the goal is to increase the throughput of the entire system, not just increase the number of tasks being done. And if you don’t have a trustworthy system of work, why should I trust your system of security controls? Bah. A total waste of time.”

Steve looks like he’s just swallowed something that isn’t agreeing with him, and he says eventually, “Everyone knows that in manufacturing, as wip increases, due-date performance goes down.

The sun shines down on me. It’s 11 a.m., and the air smells like autumn. The leaves on the trees are starting to turn orange and brown, and there are piles of them starting to form in the parking lot. Despite my fretting, I realize how refreshing it is to be able to think about what work we need to be doing and how to prioritize and release it. For a moment, I marvel at the lack of constant firefighting that dominated so much of my career in it. The types of issues we’re having to solve lately are so…cerebral. It’s what I thought management was all about when I got my mba. I’m convinced that if we do a good job thinking, we can make a real difference.

“Obviously,” he continues, “every work center is made up of four things: the machine, the man, the method, and the measures. Suppose for the machine, we select the heat treat oven. The men are the two people required to execute the predefined steps, and we obviously will need measures based on the outcomes of executing the steps in the method.”

I hurriedly write down, “Work center: machine, man, method, measure.”

I nod, unable to resist groaning. “You’re right. I’ve heard my managers complain that if Brent were hit by the proverbial bus, we’d be completely up the creek. No one knows what’s in Brent’s head. Which is one of the reason I’ve created the level 3 escalation pool.” I quickly explain what I did to prevent escalations to Brent during outages to keep him from being interrupted by unplanned work and how I’ve attempted to do the same thing for planned changes.

“Properly elevating preventive work is at the heart of programs like Total Productive Maintenance, which has been embraced by the Lean Community. tpm insists that we do whatever it takes to assure machine availability by elevating maintenance. As one of my senseis would say, ‘Improving daily work is even more important than doing daily work.’

It takes a future focused business, one that for instance creates value by building and maintaining resilient systems, to see the value in this. Daily work secures today. Improving daily work secures the future, so that daily work is more precise or requires less effort.

The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful.

“Sensei Rother calls this the Improvement Kata,” he continues. “He used the word kata, because he understood that repetition creates habits, and habits are what enable mastery. Whether you’re talking about sports training, learning a musical instrument, or training in the Special Forces, nothing is more to mastery than practice and drills. Studies have shown that practicing five minutes daily is better than practicing once a week for three hours. And if you want to create a genuine culture of improvement, you must create those habits.”

“A critical part of the Second Way is making wait times visible, so you know when your work spends days sitting in someone’s queue—or worse, when work has to go backward, because it doesn’t have all the parts or requires rework.

I stare at John’s binder, not quite believing he discarded it so carelessly. He’s been carrying it around for over two years. In front of where he was sitting is a single piece of paper, almost blank with a few lines scribbled on it. Wondering if it’s a suicide note or a resignation letter, I sneak a quick peek at what appears to be a poem. A haiku? Here I sit, hands tied Room angry, I could save them If only they knew

Patty says quickly, “Didn’t you hear what Erik told Bill? Improving something anywhere not at the constraint is an illusion. You know, no offense, but you sort of sound like John right now.”

“As part of the First Way, you must gain a true understanding of the business system that it operates in. W. Edwards Deming called this ‘appreciation for the system.’

In describing her responsibilities, she summarizes, “Ultimately, the way I measure our understanding of customer needs and wants is whether customers would recommend us to their friends. Any way you cut it, our metrics aren’t very good.”

“If you could wave a magic wand, what would you do instead?” I ask. “How big of a magic wand?” she asks. “It can do anything you want,” I reply, smiling. “That’s a big magic wand,” she says, laughing. “I want accurate and timely order information from our stores and online channels. I want to press a button and get it, instead of running it through the circus we’ve created.

The need to continually reduce cycle times is part of the First Way. The need for amplification of feedback loops, ideally from the customer, is part of the Second Way.

We’re building a body of tribal knowledge that’s helping us fix things faster than ever, and, when we do need to escalate, it’s controlled and orderly. Because of our ever-improving production monitoring of the infrastructure and applications, more often than not, we know about the incidents before the business does.

As Erik keeps reminding me, a great team performs best when they practice. Practice creates habits, and habits create mastery of any process or skill.

He continues, “Now you must prove that you can master the Second Way, creating constant feedback loops from it Operations back into Development, designing quality into the product at the earliest stages. To do that, you can’t have nine-month-long releases. You need much faster feedback.

He pauses. “That’s ridiculous, given all the investments you’ve made virtualizing your production systems. You still do deployments like they’re physical servers. As Sensei Goldratt would say, you’ve deployed an amazing technology, but because you haven’t changed the way you work, you haven’t actually diminished a limitation.”

They look at the entire flow of work, identify where the constraints are, and use every possible technology and bit of process knowledge they have to ensure work is performed effectively and efficiently.

“As part of the Second Way, you need to create a feedback loop that goes all the way back to the earliest parts of product definition, design, and development,” he says.

They also knew that until code is in production, no value is actually being generated, because it’s merely wip stuck in the system. He kept reducing the batch size, enabling fast feature flow. In part, he did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, just like the application that Development ships. That enabled him to create a one-step environment creation and deploy procedure, just like we figured out a way to do one-step painting and curing.

“So, we now know that Allspaw and Hammond weren’t so crazy after all. Jez Humble and Dave Farley independently came to the same conclusions, and then codified the practices and principles that enable multiple deployments per day in their seminal book Continuous Delivery. Eric Ries then showed us how this capability can help the business learn and win in his Lean Startup work.”

“Your next step should be obvious by now, grasshopper. In order for you to keep up with customer demand, which includes your upstream comrades in Development,” he says, “you need to create what Humble and Farley called a deployment pipeline. That’s your entire value stream from code check-in to production. That’s not an art. That’s production. You need to get everything in version control.

“I just came back from a meeting with him. He showed me a bunch of stuff and explained how they do single-minute exchanges of die at Toyota. He thinks we need to build the capability to do ten deploys per day. He not only insists this is possible but also that it supports the feature deployment cycles the business needs, not just to survive, but to win in the marketplace.”

Brent gets up and starts drawing boxes to indicate the packaging of the code for deployment; preparing new server instances; loading and configuring the operating system, databases, and applications; making all the changes to the networks, firewalls, and load balancers; and then testing to make sure the deployment completed successfully. I contemplate the entirety of the diagram, which surprisingly reminds me of the plant floor. Each of these steps is like a work center, each with different machines, men, methods, and measures. it work is probably much more complex than manufacturing work. Not only is the work invisible, making it more difficult to track, but there are far more things that could go wrong. Countless configurations need to be set correctly, systems need enough memory, all the files need to be put in the right place, and all code and the entire environment need to be operating correctly.

On a separate whiteboard, she writes down two bullet points: “environments” and “deployment.” Pointing to what she just wrote, she says, “With the current process, two issues keep coming up: At every stage of the deployment process, environments are never available when we need them, and even when they are, there’s considerable rework required to get them all synchronized with one another. Yes?”

When they both give me a blank look, I say with some exasperation, “You have a magic wand. Use it!” “How big is the magic wand?” William asks. I repeat what I said to Maggie. “It’s a very powerful magic wand. It can do anything.” William walks to the whiteboard and points at a box called “code commit.” “If I could wave this magic wand, I would change this step. Instead of getting source code or compiled code from Dev through source control, I want packaged code that’s ready to be deployed.” “And you know,” he continues, “I want this so much, I’d happily volunteer to take over responsibility for package creation. I know exactly the person I’d assign, too. She would be responsible for the Dev handoff. When code is labeled ‘ready to test,’ we would then generate and commit the packaged code, which would trigger an automated deployment into the qa environment. And later, maybe even the Production environment, too.”

In many ways, most of these guys are my temperamental opposites. I like people who create and follow processes, people who value rigor and discipline. These guys shun process in favor of whim and whimsy.

Even touching the production databases meant linking to their libraries, and any changes to them would require convincing the architecture team to approve it. Since the entire company could be out of business by that time, the developers and Brent decided to create a completely new database, using open source tools, with data copied from not only Phoenix but also the order entry and inventory management systems.

“Brent is very unique. Unicorn needs someone who has the respect of the developers, has enough deep experience with almost every sort of it infrastructure we have, and can describe what the developers need to build so that we can actually manage and operate in production. Those skills are rare, and we don’t have anyone else that can rotate into this special role right now.”

“It’s possible. We’re already using virtualization for most of our environments. It shouldn’t be very difficult to convert them so that they run on a cloud computing provider.”

“Sending our customer data to the cloud may have some risks like accidental disclosure of private data or someone unauthorized hacking into those compute servers.”

Ironically, one of the developers suggested turning off all the real-time recommendations, which we had worked so hard to build. Why recommend more products to buy, he argued, if customers can’t even complete a transaction? Maggie quickly agreed, but it still took the developers two hours to change and deploy. Now, this feature can be disabled with a configuration setting, so we can do it in minutes next time around, instead of requiring a full code rollout. Now that’s what I call designing for it Operations! It’s getting easier and easier to manage the code in production. We also kept optimizing database queries and moving the largest site graphics to a third-party content distribution network, offloading more traffic from our servers. By late Thanksgiving afternoon, the customer experience had improved to something tolerable.

I nod emphatically. “I suspect we’ll want to follow the Unicorn model for any new applications we develop internally. It’s easier to scale, as well as easier to manage, than any application we’ve supported in the past. We’re setting up the processes and procedures so that we can deploy at whatever rate it takes to quickly respond to customers. In some cases, we’re even enabling developers to deploy the code. The developer will be able to push a button and within several minutes, the code will be in the testing environment or in production.” “I can’t believe how far we’ve come in such a short time. I’m proud of all of you,” Steve says. “I want to commend you for truly working together and being worthy of one another’s trust.”

Of course, Wes tried to stop this. He insisted that we schedule penetration tests into predefined time frames. However, I convinced him this is the fastest means to institutionalize Erik’s Third Way. We need to create a culture that reinforces the value of taking risks and learning from failure and the need for repetition and practice to create mastery. I don’t want posters about quality and security. I want improvement of our daily work showing up where it needs to be: in our daily work.

“Hey, relax. Let me explain. My board holds me responsible for making the best use of company resources to achieve the goals that maximize shareholder value. My primary job is to lead my management team to make that happen.”

Understanding what technology can and can’t do has become a core competency that every part of this business must have. If any of my business managers are leading a team or a project without that skill, they will fail.” He continues, “I need each and every one of my business managers to take calculated risks, without jeopardizing the entire enterprise. People everywhere in the business are using technology, so it’s like the Wild West again—for better or for worse. Businesses that can’t learn to compete in this new world will perish.”

🎉🎉🎉

“A dysfunctional marriage assumes that the business and it are two separate entities. it should either be embedded into business operations or into the business. Voilà! There you go. No tension. No marriage, and maybe no it Department, either.”

🙌🏾

“I need you to help me elevate the state of the practice of how organizations manage technology. Let’s face it. Life in it is pretty shitty when it’s so misunderstood and mismanaged. It becomes thankless and frustrating as people realize that they are powerless to change the outcome, like an endlessly repeating horror movie. If that’s not damaging to our self-worth as human beings, I don’t know what is. That’s got to change,” he says passionately. “I want to improve the lives of one million it workers in the next five years. As someone wise once told me, ‘Messiahs are good, but scripture is better.’”

“I want you to write a book, describing the Three Ways and how other people can replicate the transformation you’ve made here at Parts Unlimited. Call it The DevOps Handbook and show how it can regain the trust of the business and end decades of intertribal warfare. Can you do that for me?” Write a book? He can’t be serious. I reply, “I’m not a writer. I’ve never written a book before. In fact, I haven’t written anything longer than an e-mail in a decade.” Unamused, he says sternly, “Learn.” Shaking my head for a moment, I finally say, “Of course. It would be an honor and a privilege to write The DevOps Handbook for you while I embark on what will probably be the most challenging three years of my entire career.” “Very good. It’ll be a great book,” he says, smiling.

There’s a term that we’re hearing more lately: something called “DevOps.” Maybe everyone attending this party is a form of DevOps, but I suspect it’s something much more than that. It’s Product Management, Development, it Operations, and even Information Security all working together and supporting one another. Even Steve is a part of this super-tribe. In that moment, I let myself feel how incredibly proud I am of everyone in this room. What we’ve pulled off is remarkable, and even though my future is probably less certain than anytime in my career, I feel incredible excitement at the challenges the coming years are going to bring.

When Kevin Behr, George Spafford, and I first started writing The Phoenix Project we never suspected how quickly DevOps would be embraced by technology professionals within all types of organizations.

Because we wanted to show both the problems and solutions in a recognizable and relatable form, we decided very early on that the only way we could describe with adequate fidelity the enormous complexity of this problem was in the form of novel, just like Dr. Eliyahu Goldratt did in The Goal, the seminal book he published in 1984. The Goal helped many of us have a giant and meaningful “aha” moment. It has been credited for helping make Lean manufacturing principles become mainstream, and, since its publication, The Goal has been integrated into almost every mainstream MBA curriculum and operations management course, influencing our next generation of leaders.

I’ve also come across otherwise smart [people] who are of the mistaken belief that if they hold on to a task, something only they know how to do, it’ll ensure job security. These people are knowledge Hoarders. This doesn’t work. Everyone is replaceable. No matter how talented they are. Sure it may take longer at first to find out how to do that special task, but it will happen without them.

Whether we are a John, a Brent, a Wes, a Patty, or a Bill, when we’re trapped in a system that prevents us from succeeding, our job becomes thankless, reinforces a feeling of powerlessness, and we feel like we are trapped in a system that preordains failure. And worse, the nature of technical debt that is not paid down ensures that the system gets worse over time, regardless of how hard we try. We now know that DevOps principles and patterns are what allow us to turn this downward spiral into a virtuous spiral, through a combination of cultural norms, architecture, and technical practices.

Ideally, small teams of developers independently implement their features, validate their correctness in production-like environments, and have their code deployed into production quickly, safely and securely. Code deployments are routine and predictable. Instead of starting deployments at midnight on Friday and spending all weekend working to complete them, deployments occur throughout the business day when everyone is already in the office and without our customers even noticing—except when they see new features and bug fixes that delight them. And, by deploying code in the middle of the workday, for the first time in decades IT Operations is working during normal business hours like everyone else.

By creating fast feedback loops at every step of the process, everyone can immediately see the effects of their actions. Whenever changes are committed into version control, fast automated tests are run in production-like environments, giving continual assurance that the code and environments operate as designed and are always in a secure and deployable state. Automated testing helps developers discover their mistakes quickly (usually within minutes), which enables faster fixes as well as genuine learning—learning that is impossible when mistakes are discovered six months later during integration testing, when memories and the link between cause and effect have long faded. Instead of accruing technical debt, problems are fixed as they are found, mobilizing the entire organization if needed, because global goals outweigh local goals.