The Phoenix Project

by Gene Kim

“Bill Palmer here,” I say, answering my cell phone on the first ring. I’m late, so I’m driving ten miles per hour over the speed limit, instead of my usual five. I spent the morning at the doctor’s office with my three-year-old son, trying to keep the other toddlers from coughing on us, constantly being interrupted by my vibrating phone.

“I bet,” Steve laughs. “I was in the Army for eight years myself, slightly longer than I was obligated to. But I didn’t mind. rotc was the only way I could pay for college, and they treated me well.” He adds, “They didn’t coddle us like they did you Marines, but I still can’t complain.” I laugh, finding myself liking him. This is the longest interaction we’ve had. I suddenly wonder if this is what politicians are like. I try to stay focused on why he summoned me here: He’s going to ask me to undertake some kamikaze mission.

To get Finance the data they need, we may have to cobble together some custom reports, which means bringing in the application developers or database people. But that’s like throwing gasoline on the fire. Developers are even worse than networking people. Show me a developer who isn’t crashing production systems, and I’ll show you one who can’t fog a mirror. Or more likely, is on vacation.

Wes and Patty are sitting next to each other at the conference table, so I walk behind them to listen in. Wes leans back in his chair with his arms crossed over his stomach. They don’t get all the way across. At six feet three inches tall and over 250 pounds, he casts a shadow on most people. He seems to be in constant motion and has a reputation of saying whatever is on his mind. Patty is the complete opposite. Where Wes is loud, outspoken, and shoots from the hip, Patty is thoughtful, analytical, and a stickler for processes and procedures. Where Wes is large, combative, and sometimes even quarrelsome, Patty is elfin, logical, and levelheaded. She has a reputation for loving processes more than people and is often in the position of trying to impose order on the chaos in it. She’s the face of the entire it organization. When things go wrong in it, people call Patty. She’s our professional apologist, whether it’s services crashing, web pages taking too long to load, or, as in today’s case, missing or corrupted data. They also call Patty when they need their work done—like upgrading a computer, changing a phone number, or deploying a new application. She does all of the scheduling, so people are always lobbying her to get their work done first. She’ll then hand it off to people who do the work. For the most part, they live in either my old group or in Wes’ group.

In a gruff voice, Wes says, “Okay, Bill, what’s so important that you think is worth interrupting a Sev 1 outage in progress?” That’s not a bad question. Severity 1 outages are serious business-impacting incidents that are so disruptive, we typically drop everything to resolve them. I take a deep breath. “I don’t know if you’ve heard, but Luke and Damon are no longer with the company. The official word is that they’ve decided to take some time off. More than that, I don’t know.”

I scratch my head. As a company, we’ve made a huge investment in virtualization. Although it looks uncannily like the mainframe operating environment from the 1960s, virtualization changed the game in Wes’ world. Suddenly, you don’t have to manage thousands of physical servers anymore. They’re now logical instances inside of one big-iron server or maybe even residing somewhere in the cloud. Building a new server is now a right-click inside of an application. Cabling? It’s now a configuration setting. But despite the promise that virtualization was going to solve all our problems, here we are—still late in delivering a virtual machine to Chris.

Situations like this only reinforce my deep suspicion of developers: They’re often carelessly breaking things and then disappearing, leaving Operations to clean up the mess. The only thing more dangerous than a developer is a developer conspiring with Security. The two working together gives us means, motive, and opportunity.

CI/CD 🤣

There’s a short silence on the phone before John replies, “No, we couldn’t test the change. There’s no test environment. Apparently, you guys requested a budget years ago, but…” I should have known. “Well, that’s good news,” Patty says after John hangs up. “It may not be easy to fix, but at least we finally know what’s going on.” “Was John’s tokenization change in the change schedule?” I ask. She laughs humorlessly. “That’s what I’ve been trying to tell you. John rarely goes through our change process. Nor do most people, for that matter. It’s like the Wild West out here. We’re mostly shooting from the hip.”

“We don’t have time to do interrogations every time something goes wrong,” I say, exasperated. “Get me a list of all the changes made in the past, say, three days. Without an accurate timeline, we won’t be able to establish cause and effect, and we’ll probably end up causing another outage.” “Good idea,” she nods. “If necessary, I’ll e-mail everyone in it to find out what they were doing, to catch things that weren’t on our schedule.”

I shake my head at his awful imagery, but he’s right. We’ve got a serious problem here. He continues, “We’re going to have to assemble a huge team, including Chris’ guys, to figure out how we’re going to pull this off. We have problems at every layer: networking, servers, databases, operating systems, applications, Layer 7 switching—the whole wad of crap. It’s going to be late nights for all of us for the next nine days.” I nod unhappily. This type of all-hands effort is just another part of life in it, but it makes me angry when we need to make some heroic, diving catch because of someone else’s lack of planning. I say, “Get your team assembled, and ask Chris to assemble his respective team as well. Stop trying to do this by e-mail and in the ticketing system. We need everyone in the same room.”

I always liked that phrase in Saving Private Ryan: “There’s a chain of command: gripes go up, not down.”

In another one of the endless Phoenix status meetings, I realize that the developers are even more behind than we feared. As Wes had predicted, more and more work is being deferred to the next release, including almost all of the testing. This means that we’ll be the ones finding the problems when they blow up in production, instead of the Quality Assurance (qa) Department. Great.

“a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.”

Later that afternoon, I’m back in the Phoenix war room when Patty calls. I run out to the hallway. “What’s up?” She sounds stressed. “We’ve got a problem. I was expecting we’d have fifty changes for us to review next week. But we’re already up to 243 submitted changes. I keep getting e-mails from people saying to expect more cards over the weekend… I think we’re looking at over four hundred changes being made next week!” Holy crap. Four hundred? How many of these four hundred changes are high risk, potentially affecting Phoenix, the payroll application, or worse? I suddenly remember Rangemaster duty in the Marines. As Rangemaster, I was responsible for the safety of everyone on the firing range. I have a horrifying vision of a mob of four hundred unsupervised eighteen-year-olds jumping out of trucks, running to the firing range, firing their rifles into the air, hooting and hollering…

When I get back to my desk, I’m looking for the Advil I usually keep on my desk when my cell phone rings. “Palmer here,” I say, rummaging through my drawers.

🤣 what?

I look up for a moment, trying to figure out how to describe the few corrective actions I’ve put in place after this week. I reply, “Honestly, I’m still trying to get some situational awareness. Mostly, I’m being whiplashed from one emergency to another. I do know that we need more rigor and discipline in how we work. I’m trying to figure out what processes we rely on to get work done around here. Based on what I’ve seen, I know we need to improve them so we stop shooting ourselves in the foot.”

“What then, exactly, is your definition of ‘work?’” he asks, with a genuinely curious expression on his face. “Well, I can tell you that Steve has stated over and over to me in no uncertain terms that we need to get Phoenix out the door. That qualifies as work in my mind.” He looks up, appearing to have a conversation with himself. “Yes, that’s certainly one type of work. But you’re still missing the three other types of work that it Operations is responsible for. To me, that’s nowhere near the level of understanding of work you need in order to fix your problems around project deliverables, outages, and compliance.”

He gestures broadly with both arms outstretched, “In the 1980s, this plant was the beneficiary of three incredible scientifically-grounded management movements. You’ve probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: wip is the silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it, you can’t control wip.”

He continues, “Because of how Mark was releasing work, inventory kept piling up in front of our bottleneck, and jobs were never finished on time. Every day was an emergency. For years, we were awarded Best Customer of the Year from our air freight shipment company, because we were overnighting thousands of pounds of finished goods to angry customers almost every week.” He pauses and then says emphatically, “Dr. Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion. Astonishing, but true! Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck.”

“Oh, really?” he turns to me, frowning intensely. “Let me guess. You’re going to say that it is pure knowledge work, and so therefore, all your work is like that of an artisan. Therefore, there’s no place for standardization, documented work procedures, and all that high-falutin’ ‘rigor and discipline’ that you claimed to hold so near and dear.”

“Your job as vp of it Operations is to ensure the fast, predictable, and uninterrupted flow of planned work that delivers value to the business while minimizing the impact and disruption of unplanned work, so you can provide stable, predictable, and secure it service.”

You must figure out how to control the release of work into it Operations and, more importantly, ensure that your most constrained resources are doing only the work that serves the goal of the entire system, not just one silo.

“Once you figure this out, young Bill, you will be well on your way toward understanding the Three Ways,” he says. “The First Way helps us understand how to create fast flow of work as it moves from Development into it Operations, because that’s what’s between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.”

The need for rigor and discipline, and constantly practicing and honing our skills are important lessons I’ve kept with me from the Marines. The lives of my men depended upon it there, and my job depends upon it here. Creating that predictability is what I’m most intent on instilling in my it Operations group.

“Remember, there are four types of work. You’ve named business project work as one. When you have the other three, give me a call.”

What did Erik mean that there were four categories of work? I think back to my meeting with Wes and Patty. Wes mentioned we have a separate list for it infrastructure projects and business projects. Are infrastructure projects another type of work?

As I ponder this, an e-mail notification window pops up on my screen, indicating another e-mail expecting a response. Are e-mails another type of work?

I think about it some more. Then I shake my head and quickly e-mail Steve, letting them know that I connected with Erik. I’m certain that a decade from now, I’ll be telling my friends about my brief encounter with the raving madman on the manufacturing plant floor. I need to get moving. Paige will be seriously annoyed if I come home late on a Friday night. When I undock the laptop from the docking station, an incredibly loud alarm pierces the air. “Holy crap!” I shout, realizing that the sound is coming from my laptop. Fumbling, I try to turn down the volume, to power it off, but nothing makes the sound stop. Frantically, I flip the laptop over and try to take the battery out, but the tape keeps it attached. I grab a letter opener, finally managing to slash the tape and get the battery out. The laptop finally goes silent.

“I’ve had my staff establish what our levels of demand and capacity truly are,” I continue. “We’ve started to inventory everything we’re being asked to do, regardless of how big or small. Based on the analysis so far, it’s clear to me that the demand for it work far exceeds our ability to deliver. I’ve asked them to make more visible what the pipeline of work looks like, so we can make more informed decisions about who should be working on what and when.”

I point him to the second page of the handout, saying, “As you can see, my team and I have come up with a request for six additional resources that we’re most short-handed on.” I go for the close, saying, “My goal is to increase our throughput so we won’t be in this position again, and to get as many of these projects done as we can. I’d like your approval to get these openings immediately, so we can start our search. Talent like Brent is not easy to find, and we need to start sooner rather than later.”

I listen to Wes and Patty trade ideas back and forth before I say, “Let’s go back to our goals: get the left and right hands to know what the other is doing, give us some situational awareness during outages, and give audit some evidence that we’re addressing change control. “We need to focus on the riskiest changes,” I continue. “The 80/20 rule likely applies here: Twenty percent of the changes pose eighty percent of the risk.”

Patty adds, “We need to create some standard procedures around these changes—like when we’ll want them implemented—and have key resources not only aware of them but also standing by, just in case things go wrong—even the vendors.”

Excited, Patty says, “Excellent! The first ones are the low-risk changes that itil calls ‘standard changes.’ For changes we’ve done many times before successfully, we just preapprove. They still need to be submitted, but they can be scheduled without us.” When everyone nods, she continues, “That leaves about two hundred changes that are medium-risk changes that we still need to look at.”

I realize that changes are the third category of work.

If we had exactly the amount of resources to take on all our project work, does this mean we might not have enough cycles to implement all these changes? I debate with myself whether I’m on the verge of some large and meaningful insight. Erik asked me what my organization’s equivalent to the job release desk on the plant floor. Does change management have anything to do with it? Suddenly I laugh out loud at the absurd number of questions I’ve just asked myself. I feel like a one-man debate club. Or that Erik tricked me into doing some philosophical navel-gazing. Thinking for a moment, I decide there’s value in knowing that changes represent yet another category of work but don’t know why. I’ve now identified three of the four categories of work. For a brief moment, I wonder what the fourth category of work is.

“Processes are supposed to protect people. We need to figure out how to protect Brent,” I say. I then describe how I already told Brent to send everyone wanting anything to Wes. “What? You want me to micromanage all of his time? I don’t have time to be Brent’s secretary or be some sort of help desk person!” he shouts.

“Based on Wes’ story, we shouldn’t even let Brent touch the keyboard. He’s allowed to tell people what to type and shoulder-surf, but under no condition will we allow him to do something that we can’t document afterward. Is that clear?”

“To prevent another Brent escalation, we should log every keystroke and record the terminal session. Maybe even have someone follow him around with a video camera and turn on audit logging so we know exactly what he changed.” I like it, although it sounds a bit extreme. However, I suspect that it will take extreme measures to get us out of this situation. I venture, “Maybe we take away his production access, so the only way the work can get done is him telling the level 3s what to do.” Wes guffaws. “He might quit if we did that right away.” “So, who do we have that’s available to put into this level 3 resource pool?” I ask. He hesitates. “Well, we have the two hires we made a year ago that were meant to help shore up Brent. One is working on creating server build standards, but we can take her off of that temporarily. There are two other engineers that we identified for cross-training years ago, but we never had the time to pursue it further. So, that’s three people.”

He continues, “It’s crazy what programmers, and even managers like me, have to learn every couple of years. Sometimes it’s a totally new database technology, a new programming or project management method, or a new technology delivery model, like cloud computing. “Just how many times can you throw out everything you know to keep up with the latest new-fangled trend? I look in the mirror every once in awhile, asking myself, ‘Will this be the year that I give up? Will I spend the rest of my career doing cobol maintenance or become just another has-been middle manager?’”

Cloud computing turns out to be a new paradigm.

“Unlike the other categories of work, unplanned work is recovery work, which almost always takes you away from your goals. That’s why it’s so important to know where your unplanned work is coming from.” I smile as he acknowledges my correct answer, and am oddly pleased that he validated my antimatter notion of unplanned work, as well. He says, “What is this change board that you mentioned?” I tell him about my attempts to get some sort of change process going and my attempt to elevate the discussion above how many fields there were on the change form, which then resulted in getting people to put their intended changes on index cards and our need to juggle them on the board. “Very good,” he says. “You’ve put together tools to help with the visual management of work and pulling work through the system. This is a critical part of the First Way, which is creating fast flow of work through Development and it Operations. Index cards on a kanban board is one of the best mechanisms to do this, because everyone can see wip. Now you must continually eradicate your largest sources of unplanned work, per the Second Way.”

Remember, unplanned work kills your ability to do planned work, so you must always do whatever it takes to eradicate it.

Remember, it goes beyond reducing wip. Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever.”

“Remember, outcomes are what matter—not the process, not controls, or, for that matter, what work you complete.”

“A great team doesn’t mean that they had the smartest people. What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists. Steve continues, “One of my favorite books about team dynamics is Five Dysfunctions of a Team, by Patrick Lencioni. He writes that in order to have mutual trust, you need to be vulnerable. So, I’m going to tell you a little about myself and what makes me tick. And then I’m going to ask you to do the same.

Steve continues, “The goal of this exercise is to get to know one another as people. You’ve learned a bit about me and my vulnerabilities. But that’s not enough. We need to know more about one another. And that creates the basis for trust.” He looks around. “Who wants to go first?” Oh, shit. Marines don’t like this kind of touchy-feely stuff. I immediately avert my eyes, not wanting to be called on first.

Then Wes raises a placating hand. “Look, I’m not blaming you, Chris. I’m just stating a well-known fact. When deployments don’t go as planned, whether the plan was written by your group or mine, it affects everybody else. That’s all I’m saying.” I nod, agreeing with Wes’ characterization. And surprisingly, Chris is nodding, as well.

“Erik has helped me understand that there are four types of it Operations work: business projects, it Operations projects, changes, and unplanned work. But, we’re only talking about the first type of work, and the unplanned work that get’s created when we do it wrong. We’re only talking about half the work we do in it Operations.”

As almost everyone goes to extraordinary lengths to avoid eye contact with him, he says even more loudly, “Hey, it smells like people just had sex in here. Did I miss anything good?” Chris, Patty, and Wes break off their conversation, and with exaggerated nonchalance, grab their original seats.

If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.”

Bill mentioned the four types of work: business projects, it Operations projects, changes, and unplanned work. Left unchecked, technical debt will ensure that the only work that gets done is unplanned work!”