Kill It with Fire: Manage Aging Computer Systems (and Future Proof Modern Ones)

by Marianne Bellotti

Shipping new code gets attention, while technical debt accrues silently and without fanfare. It’s not the age of a system that causes it to fail, but the pressure of what the organization has forgotten about it slowly building toward an explosion.

One of the reasons the DevOps and SRE movements have had such a beneficial effect on software development is that they seek to re-establish accountability. If product engineering teams play a role in running and maintaining their own infrastructure, they are the ones who feel the impact of their own decisions. When they build something that doesn’t scale, they are the ones who are awakened at 3 am with a page. Making software engineers responsible for the health of their infrastructure instead of a separate operations team unmutes the feedback loop.

Throughout this book, I have emphasized thinking about modernization projects not in terms of technical correctness but in terms of value add because it re-establishes the most important feedback loop: Is the technology serving the needs of its users?

Meetings, reports, and dialogues are the least efficient feedback loops. Feedback loops are most effective when the operator feels the impact, rather than just hearing about it. That’s because people are naturally inclined to misinterpret information to suit what they already want to believe. It is more difficult to do that when the feedback is delivered in the form of inconvenience, disruption, interruptions, and additional work.

As a general rule, the discretion to make decisions should be delegated to the people who must implement those decisions. If you are not contributing code or being woken up in the middle of the night to answer a page, have the good sense to remember that no matter how important your job is, you are not the implementor. You do not operate the system, but you can find the operators and make sure they have the air cover they need to be successful. Empower the operators.