An Elegant Puzzle: Systems of Engineering Management

by Will Larson

I was lucky early in my management career to have a coworker describe me as the best leader they had worked with. It took several additional years of practice for another to declare me their worst.

Organizational design gets the right people in the right places, empowers them to make decisions, and then holds them accountable for their results.

When I have a problem that I want to solve quickly and cheaply, I start thinking about process design. A problem I want to solve permanently and we have time to go slow? That’s a good time to evolve your culture. However, if process is too weak a force, and culture too slow, then organizational design lives between those two.

I’ve found that two-tier 24/7 support requires eight engineers.

An important property of teams is that they abstract the complexities of the individuals that compose them. Teams with fewer than four individuals are a sufficiently leaky abstraction that they function indistinguishably from individuals.

Keep innovation and maintenance together. A frequent practice is to spin up a new team to innovate while existing teams are bogged down in maintenance. I’ve historically done this myself, but I’ve moved toward innovating within existing teams.5 This requires very deliberate decision-making and some bravery, but in exchange you’ll get higher morale and a culture of learning, and will avoid creating a two-tiered class system of innovators and maintainers.

Teams should be six to eight during steady state. To create a new team, grow an existing team to eight to ten, and then bud into two teams of four or five. Never create empty teams. Never leave managers supporting more than eight individuals.

A team is innovating when their technical debt is sustainably low, morale is high, and the majority of work is satisfying new user needs.

Adding new individuals to a team disrupts that team’s gelling process, so I’ve found it much easier to have rapid growth periods for any given team, followed by consolidation/gelling periods during which the team gels. The organization will never stop growing, but each team will.

Fundamentally, I believe that sustained productivity comes from high-performing teams, and that disassembling a high-performing team leads to a significant loss of productivity, even if the members are fully retained. In this worldview, high-performing teams are sacred, and I’m quite hesitant to disassemble them.

Another reason that I lean away from moving folks off high-performing teams is that most teams have high fixed costs and relatively small variable costs: moving one person can shift an innovating team back into falling behind, and now neither team is doing particularly well. This is especially true on teams responsible for products and services.

if it’s a choice of moving people rapidly or shifting scope rapidly, I’ve found that the latter is more effective and less disruptive. Shifting scope works better than moving people because it avoids re-gelling costs, and it preserves system behavior.

you only get value from projects when they finish: to make progress, above all else, you must ensure that some of your projects finish.

Finally, the one thing that I’ve found at companies with very few interruptions and have observed almost nowhere else: really great, consistently available documentation. It’s probably even harder to bootstrap documentation into a non-documenting company than it is to bootstrap unit tests into a non-testing company, but the best solution to frequent interruptions I’ve seen is a culture of documentation, documentation reading, and a documentation search that actually works.

What I’ve found most successful is to identify a few areas to improve, ensure you’re making progress on those, and give yourself permission to do the rest poorly. Work with your manager to write this up as an explicit plan and agree on what reasonable progress looks like.

As an organizational leader, you’ll always have a portfolio of risk, and you’ll always be doing very badly at some things that are important to you. That’s not only okay, it’s unavoidable.

If you’re working at a well-established company, you may find that there aren’t too many gaps that couldn’t be readily filled by someone else. However, if you’re at a company going through hypergrowth,18 it’s common to find that everyone is already working in the most complex role of their career, and you’ll uncover gaps, gaping and cavernous.

four measures of developer velocity: Delivery lead time is the time from the creation of code to its use in production. Deployment frequency is how often you deploy code. Change fail rate is how frequently changes fail. Time to restore service is the time spent recovering from defects.

The first phase of a planning cycle is exploring the different problems that you could pick to solve. It’s surprisingly common to skip this phase, but that, unsurprisingly, leads to inertia-driven local optimization. Taking the time to evaluate which problem to solve is one of the best predictors I’ve found of a team’s long-term performance.

Prefer experimentation over analysis. It’s far more reliable to get good at cheap validation than it is to get great at consistently picking the right solution. Even if you’re brilliant, you are almost always missing essential information when you begin designing. Analysis can often uncover missing information, but it depends on knowing where to look, whereas experimentation allows you to find problems that you didn’t anticipate.

Strategies are grounded documents which explain the trade-offs and actions that will be taken to address a specific challenge. Visions are aspirational documents that enable individuals who don’t work closely together to make decisions that fit together cleanly.

Because strategies are specific to a given problem, it’s okay—and even encouraged—to write quite a few of them. Over the past year, I’ve worked with people on strategies for how we partner with other teams, how we manage end-to-end API latency, and how we manage infrastructure costs.

People sometimes describe strategy as artful or sophisticated, but I’ve found that the hardest part of writing a good strategy is pretty mundane. You must be honest about the constraints that are making the challenge difficult, which almost always include people and organizational aspects that are uncomfortable to acknowledge. No extent of artistry can solve a problem that you’re unwilling to admit.

You’ll know a vision is succeeding when people reference the document to make their own decisions, and you’ll know it’s struggling when decisions keep happening that don’t fit into its direction.

Good goals are a composition of four specific kinds of numbers: A target states where you want to reach. A baseline identifies where you are today. A trend describes the current velocity. A time frame sets bounds for the change.

When you’re asked to take responsibility for a company’s overall infrastructure costs, you’re going to start from a goal along the lines of “Maintain infrastructure costs at their current percentage of net revenue of 30 percent.” (That percentage is a fictional number for this example’s purposes, since the percentage will depend on your industry and maturity, but I have found that tying it against net revenue is more useful than pinning it at a specific dollar amount.)

The fact that something stops working at significantly increased scale is a sign that it was designed appropriately to the previous constraints rather than being over-designed.23

It’s important to celebrate migrations while they’re ongoing, but the majority of the celebration and recognition should be reserved for their successful completion.

Validate that organizational change is the right tool. Project head count a year out. Set target ratio of management to individual contributors. Identify logical teams and groups of teams. Plan staffing for the teams and groups. Commit to moving forward. Roll out the change.

For whatever controls you pick, the second step is to agree on the degree of alignment for each one. Some of the levels that I’ve found useful are: I’ll do it. Stuff that I will personally be responsible for doing. When you’re going to do something, it’s better to be explicit and avoid confusion on responsibilities. Best used sparingly. Preview. I’d like to be involved early and often. This is probably an area where we aren’t quite on the same page and this will help us avoid redoing work. and this helps us avoid redoing work. Review. I’d like to weigh in before it gets published or fully rolled out, but we’re pretty aligned on the topic. Notes. Projects I’d like to follow but don’t have much I can add to. Often used for wide-reaching initiatives on which we are well aligned, and I want to be able to represent my colleagues’ work correctly. No surprises. The work that we’re currently aligned on but requires updates to keep my mental model in tact. If I’m asked about a related problem, I want to be able to answer it correctly. This is particularly important for me, as my effectiveness is evaluated based on my ability to stay on top of new problems. Let me know. We’re well aligned on this, since my colleagues have done it before and done it well. I want them to let me know if something comes up that I can help with, but otherwise I’m totally confident it’ll go well, so we don’t need to talk about this much.

A prototypical head of engineering will be skilled at organizational design, process design, business strategy, recruiting, mentoring, coaching, public speaking, and written communication. They’ll also have a broad personal network and a broad foundation from product engineering to infrastructure engineering.

The three rules for speaking with the media: Answer the question you want to be asked. If someone asks a very difficult or challenging question, reframe it into one that you’re comfortable answering. Don’t accept a question’s implicit framing, but instead take the opportunity to frame it yourself. Don’t Think of An Elephant by George Lakoff 32 is a phenomenal, compact guide to framing issues. Stay positive. Negative stories can be very compelling. They are quite risky, too! As an interviewee, find a positive framing and stick to it. This is especially true when it comes to competitors and controversy. Speak in threes. Narrow your message down to three concise points, make them your refrain, and continue to refer back to your three speaking points.

Model, Document, Share assumes: It’s better to adopt a great approach slowly. Some teams don’t have the bandwidth to adopt a new approach. The organization may not have resources to coordinate a rollout. You want to decentralize decision-making on this topic. Teams have agency to adopt the appropriate practices for themselves. It’s okay to approach change gradually.

Domineering groups significantly reduce individuals’ negative and positive freedoms, and become churn factories for members. This is most common when those making decisions are abstracted from the consequences of the decisions, e.g., architecture groups in which the members write little code.

Start with the conclusion. Particularly in written communication, folks skim until they get bored and then stop reading. Accommodate this behavior by starting with what’s important, instead of building toward it gradually.

Frame why the topic matters. Typically, you’ll be presenting on an area that you’re intimately familiar with, and it’s probably very obvious to you why the work matters. This will be much less obvious to folks who don’t think about the area as often. Start by explaining why your work matters to the company.

Derive actions from principles. One of your aims is to provide a mental model of how you view the topic, allowing folks to get familiar with how you make decisions. Showing you are “in the data” is part of this. The other aspect is defining the guiding principles you’re using to approach decisions.

Make a clear ask. If you don’t go into a meeting with leadership with a clear goal, your meeting will either go nowhere or go poorly. Start the meeting by explicitly framing your goal!

My general approach to presenting to senior leaders is: Tie topic to business value. One or two sentences to answer the question “Why should anyone care?” Establish historical narrative. Two to four sentences to help folks understand how things are going, how we got here, and what the next planned step is. Explicit ask. What are you looking for from the audience? One or two sentences. Data-driven diagnosis. Along the lines of a strategy’s diagnosis phase,39 explain the current constraints and context, primarily through data. Try to provide enough raw data to allow people to follow your analysis. If you only provide analysis, then you’re asking folks to take you on trust, which can come across as evasive. This should be a few paragraphs, up to a page. Decision-making principles. Explain the principles that you’re applying against the diagnosis, articulating the mental model you are using to make decisions. What’s next and when it’ll be done. Apply your principles to the diagnosis to generate the next steps. It should be clear to folks reading along how your actions derive from your principles and the data. If it’s not, then either rework your principles or your actions! Return to explicit ask. The final step is to return to your explicit ask and ensure that you get the information or guidance you need.

Prioritize long-term success over short-term quality. As your scope increases, the important work that you’re responsible for may simply not be possible to finish. Worse, the work that you believe is most important, perhaps high-quality one-on-ones, is often competing with work that’s essential to long-term success, like hiring for a critical role. Ultimately, you have to prioritize long-term success, even if it’s personally unrewarding to do so in the short term. It’s not that I like this approach, it’s that nothing else works.

Make every office a first-tier office; there are no second-tier offices. A first-tier office must own multiple critical projects, and its work shouldn’t be constrained by support from other offices. Furthermore, it must have many individuals physically present who work together closely.

Teams are staffed in, at most, one office. (I say “at most” in order to support the premise of teams composed entirely of remote engineers.) Employees in an office must be members of a team in that office. Remote employees may work on any team. Employees within a 60 minute commute of an office must work from that office.

The two reasons that applying policy consistently is particularly challenging are: Accepting reduced opportunity space. Good constraints make trade-offs that deliberately narrow your opportunity space. Some of the opportunities that you’ll encounter within that space will be exceptionally good, and it’s hard to stay true when faced with concrete consequences. Locally suboptimal. Satisfying global constraints inevitably leads to local inefficiency, sometimes forcing some teams to deal with deeply challenging circumstances in order to support a broader goal that they may experience little benefit from. It’s hard to ask folks to accept such circumstances, harder to be someone in one of those local inefficiencies, and hardest yet to stick to the decisions at real personal cost to the folks you’re impacting.

Finishes is particularly important, as opposed to does, because partial work has no value, and your team’s defining constraints are often in the finishing stages.

When I started managing, my leadership philosophy was simple: The Golden Rule6 makes a lot of sense. Give everyone an explicit area of ownership that they are responsible for. Reward and status should derive from finishing high-quality work. Lead from the front, and never ask anyone to do something you wouldn’t.

“With the right people, any process works, and with the wrong people, no process works.”

Lately, I’ve come to have something of a mantra for guiding decisionmaking: do the right thing for the company, the right thing for the team, and the right thing for yourself, in that order. This is pretty obvious on some levels, but I’ve found it to be a useful thinking aid.

experienced, successful managers from well-established companies dive into a startup and exit soon thereafter with a legacy of ineffective initiatives.

Optimize locally. Picking technologies that the company can’t support, or building a product that puts you in competition with another team.

Effective managers tend to become the glue in their team, filling any gaps. Sometimes that means doing things you don’t really want to do, in order to set a good example.