Kill It with Fire: Manage Aging Computer Systems (and Future Proof Modern Ones)

by Marianne Bellotti

People without confidence self-sabotage. They create self-fulfilling prophecies and display signs of learned helplessness.

The team members were completely demoralized. They had lost faith in their ability to ship code safely, so they backed off larger, more creative solutions to technical challenges that might have helped them. They became resigned to their situation, as if outages were inevitable.

They didn’t test things. When things went wrong, they didn’t do a thorough investigation and confirm what the failure points had been. They avoided those things not because they didn’t understand that they were important, but because they had lost faith in their own abilities. After so many failures and years of denied resource requests, they felt other people in the organization assumed they were bad engineers and were desperate to avoid confirming that.

Confidence comes before success. Success rarely creates confidence. When teams don’t have confidence in themselves, they will always find something to debunk successful outcomes. They got lucky. The outcome wasn’t as good as it should have been or could have been had another team been in charge. The successful outcome did not outweigh past failures.

They fail because the organization deploys solutions that actually reinforce unsuccessful conditions. If you’re coming into a project in the middle, your most important task as a leader is figuring out where those cycles are and stopping them.

“Planning is problem solving, while design is problem setting.”

Problem-solving versus problem-setting is the difference between being reactive and being responsive. Reactive teams jump around aimlessly. Setbacks whittle away their confidence and their ability to coordinate. Momentum is hard to maintain. Responsive teams, on the other hand, are calmer and more thoughtful.

Think of this as a toolkit. Some of these exercises are loosely adapted from The Surprising Power of Liberating Structures: Simple Rules to Unleash a Culture of Innovation by Henri Lipmanowicz and Keith McCandless, which is a great resource for further learning.

Exercise: Shared Uncertainties5 This exercise also starts by asking team members to identify potential risks and challenges to a project’s success, but this time, you’re looking for differences in how such risks are perceived. Give each team member a four-quadrant map with the following axes: Simple to complex Problems are simple if they are well defined and understood. They are complicated if their causes are unknown or if solving them means giving up something else of value. Orderly to chaotic Problems are orderly when there isn’t much debate about the correct way to solve them, although those solutions might be long and tedious. They are chaotic when their solutions could accidentally make the situation worse.

Regarding simple/chaotic and orderly/complex problems, if you have any of those, they are good issues to focus early conversations around. They are often the most intimidating and anxiety-inducing.

Have each team member brainstorm an ordered list of actions they can take right now to make the situation 15 percent better. The number 15 is arbitrary; don’t quibble over whether the impact of actions would really be only an 8 percent improvement. The point is these actions don’t need to come close to solving the problem; they just need to move things forward.

The best technical conversations are the ones you don’t need to have.

Nothing produces out-of-scope digressions more effectively than having people in meetings who don’t need to be there.

Probabilistic outcome-based decision-making is better known as betting. It’s a great technique for decisions that are hard to undo, have potentially serious impacts, and are vulnerable to confirmation bias.

but I find the conversation usually goes better if the list of outcomes is either positive or negative. Then team members place bets as to whether the outcome will come true. Traditionally, this exercise is run with imaginary money. Depending on the specific decision to be made, I sometimes ask them to bet with hours of their time instead of money.

The wondrous thing about this design is that if you ask people to rate their confidence level between 1 and 10, most of them would struggle to answer. It’s the unit itself, the knowledge of how much a dollar or an hour means to them, and what it means to lose a certain amount of dollars or time that helps research subjects articulate their feelings. It doesn’t matter that they will not lose what they’ve bet, just imagining this much money or that much time is enough to help people place where their feelings are on a spectrum.

You can do this exercise alone when struggling with your own decisions.

In 1968, Melvin Conway published a paper titled “How Do Committees Invent?”7 This paper, originally intended for Harvard Business Review but rejected for being too speculative in nature, outlined the ways the structure and incentives of an organization influenced the software product it produced.

Conway’s insight linking the structure of software to the structure of the committees that invented it seemed significant enough for Brooks to repackage the thesis as “Conway’s law” when he published his guide on effectively managing software teams, titled The Mythical Man-Month, in 1975.

“The greatest single common factor behind many poorly designed systems now in existence has been the availability of a design organization in need of work.”

Organizations and products both change, but they do not always change at the same pace. Figuring out whether to change the organization or change the design of the technology is just another scaling challenge.

Stern lectures about the importance of choosing the right technology for the job did not stop this behavior. It stopped when the organization hired engineering managers who developed a career ladder. By defining what the expectations were for every experience level of engineering and hiring managers who would coach and advocate for their engineers, engineers could earn promotions and opportunities without the need to show off.

Organizations end up with patchwork solutions because the tech community rewards explorers.

Therefore, when an organization provides no pathway to promotion for software engineers, they are incentivized to make technical decisions that emphasize their individual contribution over integrating well into an existing system.

Essentially, engineers are motivated to create named things. If something can be named, it can have a creator. If the named thing turns out to be popular, the engineer’s prestige increases, and her career will advance.

Most of the systems I work on rescuing are not badly built. They are badly maintained. Technical decisions that highlight individuals’ unique contributions are not always comprehensible to the rest of the team.

The folly of engineering culture is that we are often ashamed of signing up our organization for a future rewrite by picking the right architecture for right now, but we have no misgivings about producing systems that are difficult for others to understand and therefore impossible to maintain.

Scaling an organization before it needs to be scaled has similar consequences to scaling technology before it needs to be scaled. It restricts your future technical choices.

The teams that make sense in the beginning do not always make sense at the end.

Legacy modernizations are ultimately transitions and require leaders with high tolerance for ambiguity.

“The only thing the government hates more than change is the way things are.”

Punishment and rewards are two sides of the same coin. Rewards have a punitive effect because they, like outright punishment, are manipulative. “Do this and you’ll get that” is not really very different from “Do this or here’s what will happen to you.” In the case of incentives, the reward itself may be highly desired; but by making that bonus contingent on certain behaviors, managers manipulate their subordinates, and that experience of being controlled is likely to assume a punitive quality over time.

If you want to improve people’s tolerance for certain types of risks, change where the organization lands on those two critical vectors of rewards and acknowledgment. You have four options: increase the odds that good behavior will get noticed (especially by peers), decrease the odds that bad outcomes will get noticed, increase the rewards for good behavior, or decrease the punishment for bad outcomes. All of these will alter an organization’s perception of risk and make breaking changes easier.

To summarize, people’s perception of risk is not static, and it’s often not connected to the probability of failure so much as it is the potential feeling of rejection and condemnation from their peers. Since social pressures and rewards are better incentives than money and promotions, you can improve your odds of success by learning how to manipulate an organization’s perception of risk.

Once you know how to manipulate the organization’s perception of risk, successfully managing the break is all about preparation. While you will not be able to predict everything that could go wrong, you should be able to do enough research to give your team the ability to adapt to the unknown with confidence.

Failure does not necessarily jeopardize user trust. If the situation is quickly resolved and users receive clear and honest communication about the problem, the occasional failure can trigger the service recovery paradox and inspire greater user confidence.

Organizations should not shy away from failure, because failure can never be prevented. It can only be put off for a while or redirected to another part of the system. Eventually, every organization will experience failure.

Embracing failure as a source of learning means that your team gains more experience and ultimately improves their skills mitigating negative impacts on users. Practicing recovering fro...

it is essential when you work on these projects that everyone on the team is able to answer this question: How do we know if it’s getting better?

Bullet journaling is effective for me because each page is a snapshot of everything that is on my mind at the time. I record work projects, personal projects, events and social activities, holidays, and illnesses. Anything that I expect to take up a large part of the day, I write down.

As an industry, we reflect on success but study failure. Sometimes obsessively. I’m suggesting that if you’re modeling your project after another team or another organization’s success, you should devote some time to actually researching how that success came to be in the first place.

One of my favorite metaphors for setting a cadence for early and often updates comes from the podcast Legacy Code Rocks (https://www.legacycode.rocks/). Launching a new feature is like having a house party. The more house parties you have in your house before you clean things up, the worse condition your house will be in. Although there isn’t a hard-and-fast rule here that will work for everyone, automatically scheduling some time to reevaluate usage changes and technical debt after every n feature launches will normalize the process of updating the system in ways that will ensure its long-term health.

When people associate refactoring and necessary migrations with a system somehow being built wrong or breaking, they will put off doing them until things are falling apart. When the process of managing these changes is part of the routine, like getting a haircut or changing the oil in your car, the system can be future-proofed by gradually modernizing it.

“The truth is counterintuitive.”

As a general rule, we get better at dealing with problems the more often we have to deal with them.

Throughout this book and in this chapter especially, the message has been don’t build for scale before you have scale. Build something that you will have to redesign later, even if that means building a monolith to start. Build something wrong and update it often.

The secret to building technology “wrong” but in the correct way is to understand that successful complex systems are made up of stable simple systems.

Disaster comes from trying to build complex systems right away and neglecting the foundation with which all system behavior—planned and unexpected—will be determined.

Two tools popular with system thinkers for these kinds of models are Loopy (https://ncase.me/loopy/). and InsightMaker (https://insightmaker.com/

If you’re a student of Fred Brooks’sThe Mythical Man-Month, you might object to the premise of this model. It suggests that one possible solution is to add more people to the team, and we know that software is not successfully built in man-hours. More people do not make software projects go faster.