An Elegant Puzzle: Systems of Engineering Management

by Will Larson

Organizational design gets the right people in the right places, empowers them to make decisions, and then holds them accountable for their results.

If you finish the entire book, you won’t walk into your office the next day as a perfect manager (I remain grateful for the days I walk into the office feeling like a marginally competent one), but I hope that it’ll stimulate questions about how you’re approaching management, provide a few new approaches for you to experiment with, and help you take a few steps further down the path of engineering management.

An organization is a collection of people working toward a shared goal.

As I’ve gotten more exposure, I’ve come to believe that the fundamental challenge of organizational design is sizing teams.

Managers should support six to eight engineers

This gives them enough time for active coaching, coordinating, and furthering their team’s mission by writing strategies,2 leading change,3 and so on.

Managers-of-managers should support four to six managers

On-call rotations want eight engineers

I’ve sponsored quite a few teams of one or two people, and each time I’ve regretted it. To repeat: I have regretted it every single time.

An important property of teams is that they abstract the complexities of the individuals that compose them. Teams with fewer than four individuals are a sufficiently leaky abstraction that they function indistinguishably from individuals.

As a manager, your obligation is to identify the correct system solution for a given transition, initiate that solution, and then support the team as best you can to create space for the solutions to work their magic.

Sometimes people instead attempt to capture more resources from the existing company, and I’m pretty negative on that. People are not fungible, and generally folks end up in useful places, so I’m skeptical of reassigning existing individuals to drive optimality.

Tactically, ensure that the work your team is doing is valued: the quickest path out of innovation is to be viewed as a team that builds science projects, which inevitably leads to the team being defunded.

Fundamentally, I believe that sustained productivity comes from high-performing teams, and that disassembling a high-performing team leads to a significant loss of productivity, even if the members are fully retained. In this worldview, high-performing teams are sacred, and I’m quite hesitant to disassemble them.

If your company is designing systems to last one order of magnitude and is doubling every six months, then you’ll have to re-implement every system twice every three years. This creates a great deal of risk—almost every platform team is working on a critical scaling project—and can also create a great deal of resource contention to finish these concurrent rewrites.

However, the real productivity killer is not system rewrites but the migrations that follow those rewrites. Poorly designed migrations expand the consequences of this rewrite loop from the individual teams supporting the systems to the entire surrounding organization.

This

you only get value from projects when they finish: to make progress, above all else, you must ensure that some of your projects finish.

The second most effective time thief that I’ve found is ad hoc interruptions: getting pinged on HipChat or Slack, taps on the shoulder, alerts from your on-call system, high-volume email lists, and so on. The strategy here is to funnel interruptions into an increasingly small area, and then automate that area as much as possible. Ask people to file tickets, create chatbots that automate filing tickets, create a service cookbook, and so on. With that setup in place, create a rotation for people who are available to answer questions, and train your team not to answer other forms of interruptions. This is remarkably uncomfortable because we want to be helpful humans, but it becomes necessary as the number of interruptions climbs higher.

The most valuable skill in this situation is learning to say no in a way that is appropriate to your company’s culture.

As an organizational leader, you’ll always have a portfolio of risk, and you’ll always be doing very badly at some things that are important to you. That’s not only okay, it’s unavoidable.

The first step in succession planning is to figure out what you do. This seems like it should be easy, but I’ve found it surprisingly hard! There are the obvious things you do—one-on-ones, meetings, head count planning—but you’re probably filling in a hundred little holes that you don’t even think about.

If you really want a solid grasp on systems thinking fundamentals, you should read Thinking in Systems: A Primer3 by Donella H. Meadows,

They focus on four measures of developer velocity: Delivery lead time is the time from the creation of code to its use in production. Deployment frequency is how often you deploy code. Change fail rate is how frequently changes fail. Time to restore service is the time spent recovering from defects.

For example, if you don’t have a backlog of ready commits, then speeding up your deploy rate may not be valuable. Likewise, if your defect rate is very low, then reducing your recovery time will have little impact on the system.

If you do want the full experience, there are relatively few tools out there to support you. Stella5 is the gold standard, but the price is quite steep, with a nonacademic license costing more than a new laptop. The best cheap alternative that I’ve found is Insight Maker,6 which has some UI quirks but features a donation-based payment model.

Competitive moats. Moats are a more extreme version of a competitive advantage. Moats represent a sustaining competitive advantage, which makes it possible for you to pursue offerings that others simply cannot. It’s useful to consider moats in three different ways:

Find reference users. Can you find users who are willing to be the first users for the solution? If you can’t, you should be skeptical whether what you’re building is worthwhile.

agreeing on strategy and vision has been the most effective approach that I’ve found to alignment at scale.

A strategy recommends specific actions that address a given challenge’s constraints. A structure that I’ve found extremely effective13 is described in Good Strategy/Bad Strategy by Richard Rumelt,14 and has three sections: diagnosis, policies, and actions.

Continuing the above example, you might choose to allow short-term performance to dip in order to invest into long-term performance, combined with a policy of proactive expectation-setting with your stakeholders. Conversely, you might choose to take a hill-climbing approach to long-term performance, with iterative short-term improvement leading to improved long-term performance. Both are valid guiding policies, and both embrace the reality that you have limited resources to invest. When you read bad guiding policies, you think, “so what?” because its found a way to justify entrenching the status quo. When you read good guiding policies, you think, “Ah, that’s really going to annoy Anna, Bill, and Claire,” because the approach takes a clear stance on competing goals.

Folks are often comfortable with hard decisions in the abstract, but struggle to translate policies into the specific steps to implement them.

If strategies describe the harsh trade-offs necessary to overcome a particular challenge, then visions describe a future in which those trade-offs are no longer mutually exclusive.

Test the document! This is a core leadership tool, and your first version will almost certainly be bad. Write a draft, sit down with a few different folks to get their perspectives, then iterate. Keep doing this until you’ve synthesized feedback.

This can be a very empowering moment because goals decouple the “what” from the “how,” but it can also be a confusing transition for everyone involved: writing clear goals takes a bit of practice.

Bad goals are indistinguishable from numbers. “Our p50 build time will be below two seconds,” or “We’ll finish eight large projects.” You’ll know a goal is just a number when you read it and aren’t sure if it’s ambitious or whether it matters.

Good goals are a composition of four specific kinds of numbers: A target states where you want to reach. A baseline identifies where you are today. A trend describes the current velocity. A time frame sets bounds for the change. Put these all together, and a well-structured goal takes the form of: “In Q3, we will reduce time to render our frontpage from 600ms (p95) to 300ms (p95). In Q2, render time increased from 500ms to 600ms.”

and they your help explaining

Typo - they need your help?

As a result, you switch tools a lot, and your ability to migrate to new software can easily become the defining constraint for your overall velocity. Given their importance, we don’t talk about running migrations very often; let’s remedy that!

Effective de-risking is essential, because each team who endorses a migration is making a bet on you that you’re going to get this damn thing done, and not leave them with a migration to an abandoned system that they have to revert to. If you leave one migration partially finished, people will be exceedingly suspicious of participating in the next.

My final tip for finishing migrations centers around recognition. It’s important to celebrate migrations while they’re ongoing, but the majority of the celebration and recognition should be reserved for their successful completion. In particular, starting but not finishing migrations often incurs significant technical debt, so your incentives and recognition structure should be careful to avoid perverse incentives.

Caveat: this is more of a thinking tool than a recipe! My approach for planning organization change: Validate that organizational change is the right tool. Project head count a year out. Set target ratio of management to individual contributors. Identify logical teams and groups of teams. Plan staffing for the teams and groups. Commit to moving forward. Roll out the change. Now, let’s drill into each of those a bit. Figure 3.6

Metrics26 align on outcomes while leaving flexibility around how the outcomes are achieved. Visions27 ensure that you agree on long-term direction while preserving short-term flexibility. Strategies28 confirm you have a shared understanding of the current constraints and how to address them. Organization design allows you to coordinate the evolution of a wider organization within the context of sub-organizations. Head count and transfers are the ultimate form of prioritization, and a good forum for validating how organizational priorities align across individual teams. Roadmaps align on problem selection and solution validation. Performance reviews coordinate culture and recognition. Etc. There are an infinite number of other possibilities, many of which are specific to your company’s particular meetings and forums. Start with this list, but don’t stick to it! For

With all of this in mind, take an hour and write up as many goals as you can for what you’d like to accomplish in the next one to five years. Then prioritize the list, pick a few that you’d like to focus on for the next three to six months, and share it with your manager at your next one-on-one.

The three rules for speaking with the media: Answer the question you want to be asked. If someone asks a very difficult or challenging question, reframe it into one that you’re comfortable answering. Don’t accept a question’s implicit framing, but instead take the opportunity to frame it yourself. Don’t Think of An Elephant by George Lakoff 32 is a phenomenal, compact guide to framing issues. Stay positive. Negative stories can be very compelling. They are quite risky, too! As an interviewee, find a positive framing and stick to it. This is especially true when it comes to competitors and controversy. Speak in threes. Narrow your message down to three concise points, make them your refrain, and continue to refer back to your three speaking points.

the most useful framework I’ve found for thinking through this involves positive and negative freedoms. A positive freedom is the freedom to do something, for example the freedom to pick a programming language you prefer. A negative freedom is the freedom from things happening to you, for example the freedom not to be obligated to support additional programming languages, even if others would greatly prefer them.

Start with the conclusion. Particularly in written communication, folks skim until they get bored and then stop reading. Accommodate this behavior by starting with what’s important, instead of building toward it gradually.

I’m still pretty busy on a day-to-day basis, but I’ve gotten much, much better at getting things done, not by getting faster but by getting more logical about solving problems. The most impactful changes I’ve made to how I manage time are: Quarterly time retrospective. Every quarter, I spend a few hours categorizing my calendar from the past three months to figure out how I’ve invested my time. This is useful for me to reflect on the major projects I’ve done, and also to get a sense of my general allocation of time. I then use this analysis to shuffle my goal time allocation for the next quarter.

When you roll out a policy, it’s quite helpful to declare a future time when you’ll refresh it, which ensures that you’ll have the time to fully evaluate your new policy before attempting revision. It’s fairly common for folks to modify good, effective policy due to concerns that arise before the policy has had time to show its effect. At a sufficiently high rate of change, policy is indistinguishable from exception.

!

This no is explaining your team’s constraints to folks outside the team, and it’s one of the most important activities you undertake as an engineering leader.

expressing your priorities convincingly can be a difficult, daunting task. I recommend breaking it down into three discrete steps: document all your incoming asks, develop guiding principles for how work is selected, and then share subsets of tasks you’ve selected based on those guiding principles.