Kill It with Fire: Manage Aging Computer Systems (and Future Proof Modern Ones)

by Marianne Bellotti

Managing technical debt is about restoring consistency. A good way to approach the challenge is to run a product discovery exercise as if you were going to build a completely new system, but don’t actually build one! Instead, use this new vision to excavate and refocus the current system.

The organization migrated paper processes to digital processes largely without changing them and maintains the original process boundaries within the technology.

Teams that are good at legacy modernization know how to identify the trade-offs and negotiate the best possible deal.

Is it worth losing some accuracy to make things faster? Is it worth migrating to managed services when that makes testing locally more difficult?

If the system is simple, it is possible to anticipate how failure will happen and prevent it, which leads to the second characteristic of systems that experience normal accidents.

I tell my engineers that the biggest problems we have to solve are not technical problems, but people problems. Modernization projects take months, if not years of work. Keeping a team of engineers focused, inspired, and motivated from beginning to end is difficult.

Many of the “stupid” technical choices from the legacy system seem very different.

Good modernization work needs to suppress that impulse to create elegant comprehensive architectures up front. You can have your neat and orderly system, but you won’t get it from designing it that way in the beginning. Instead, you’ll build it through iteration.

To build momentum behind a modernization effort, it’s essential to communicate how modernizing will improve the status quo. Defining a measurable problem explains to the business side of the organization

If you’re thinking about rearchitecting a system and cannot tie the effort back to some kind of business goal, you probably shouldn’t be doing it at all.

With my engineers, I set the expectation that to have a productive, free-flowing debate, we need to be able to sort comments and issues into in-scope and out-of-scope quickly and easily as a team. I call this technique “true but irrelevant,” because I can typically sort meeting information into three buckets: things that are true, things that are false, and things that are true but irrelevant. Irrelevant is just a punchier way of saying out of scope.

A quick trick when two capable engineers cannot seem to agree on a decision is to ask yourself what each one is optimizing for with their suggested approach.

this sounds like a good way to deal with some of our problems

The more an organization has failed at something, the more proof it needs that modernization will bring value. When there’s a history of failure, that first step has to provide enough value to build the momentum necessary to be successful. The obvious problem with that is it means there’s a natural upper bound. There is a point where cynicism is so high, no single first step will ever provide enough value to prove the project will work.

Occasionally, I went as far as looking for a crisis to draw attention to.

Instead, it was a matter of storytelling—taking something that was unreported and highlighting its potential risks. These

My favorite place to start was with security, followed by system stability. One does not need much technical literacy to understand the impact and consequences of getting those issues wrong.

The saying “Ask for forgiveness, not permission” is popular among the startup crowd, but let’s face it, you’re better off asking for forgiveness if it’s believable that you might have been acting in good faith.

That created a situation where the retry queue would fill up to the point where 20 workers would need hours of processing just to clear the messages.

yikes

The pressure to delay maintenance work on legacy systems in favor of new features and products is constant at most organizations. There’s never a good time for it, although it always seems that if the organization could just get through the latest challenge, things will calm down and the cleanup can begin. To avoid endless procrastination, try to align the new features with the goal state.

Opportunity costs are really about selling the strategy up the chain of command. Doing this is easier if the organization has defined service-level objectives(SLOs) or has service-level agreements (SLAs).

Building a product from the beginning with a service-oriented architecture is usually a mistake. Because you don’t have the proper

That’s why monoliths are so great during the early stages of a product. They are tightly coupled, but their complexity and level of abstraction are low. When an engineer makes a change that breaks another part of the system, she knows it immediately and has access to the code to fix the problem she caused.

A former colleague of mine and an experienced engineer from Google used to like to say, “Anything over four nines is basically a lie.” The more nines you are trying to guarantee, the more risk-averse engineering teams will become, and the more they will avoid necessary improvements. Remember, to get five nines or more, they have only seconds to respond to incidents. That’s a lot of pressure.

Code Yellows end when the issue is out of the critical stage, not when the problem is fully resolved.

To be a productive and beneficial exercise, it is essential that the murder board precedes an extremely stressful event. Murder

boards have two goals. The

first is to prepare candidates for a stressful event by making sure they have an answer for every question, a response to every concern, and mitigation strategy for every foreseeable problem. The second goal of a murder board is to build candidates’ confidence. If they go into the stressful event knowing that they survived the murder board pr...