An Elegant Puzzle: Systems of Engineering Management

by Will Larson

Test the document! This is a core leadership tool, and your first version will almost certainly be bad. Write a draft, sit down with a few different folks to get their perspectives, then iterate. Keep doing this until you’ve synthesized feedback. If there is feedback you disagree with, embrace the vision as an opportunity to address conflict by explicitly acknowledging disagreements within the vision text.

Refresh periodically. Take some time every year to refresh the vision, and prefer usefulness over consistency. If your old vision doesn’t resonate, it’s okay to start over: it’s a sign that you’ve learned a lot over the past year.

Write simply. Often, visions are saturated with buzzwords, which turns readers off.

Like other leadership tools, a vision or strategy document is a solution to a specific set of problems, and it’s not always useful. If your team is aligned and doing good work, time spent writing these probably won’t be too valuable.

However, if your team is struggling to align with stakeholders, or if you’re struggling to lead a cohesive organization, these documents are exceptionally useful, fairly quick to write as you gain practice, and low risk (at worst, they get ignored).

This can be a very empowering moment because goals decouple the “what” from the “how,” but it can also be a confusing transition for everyone involved: writing clear goals takes a bit of practice.

Bad goals are indistinguishable from numbers. “Our p50 build time will be below two seconds,” or “We’ll finish eight large projects.” You’ll know a goal is just a number when you read it and aren’t sure if it’s ambitious or whether it matters.

There are two particularly interesting kinds of goals: investments and baselines. Investments describe a future state that you want to reach, and baselines describe aspects of the present that you want to preserve.

Although people often talk about goals and metrics18 when they’re writing new plans or reflecting on past plans, my fondest memories of metrics are when I’ve seen them used to drive large organizational change. In particular, I’ve found metrics to be an extremely effective way to lead change with little or no organizational authority, and I wanted to write up how I’ve seen that work.

Explore: The first step is to get data in an explorable format in your data warehouse, an SQL database, or even an Excel spreadsheet. Once there, spend time looking through it and getting a feel for it. Your goal in this phase is to identify where the levers for change are.

Dive: Once you know the three or four major contributors, go deep on understanding those areas and the levers that drive them. Batch costs might be sensitive to number of jobs, total data stored, or new product development, or it might be entirely driven by a couple of expensive jobs. Diving deep helps you build a mental model, and it also kicks off a relationship between you and the teams who you’ll want to partner with most closely.

Attribute: For most company-level metrics (cost, latency, development velocity, etc.), the first step of diving will uncover one team who are nominally accountable for the metric’s performance, but they are typically a cloak. When you pull that cloak aside, that team’s performance is actually driven by dozens of other teams.

Contextualize: Armed with the attribution data, start to build context around each team’s performance. The most general and self-managing tool for this is benchmarking.

Nudge: Once you’ve built context around the data so that folks can interpret it, the next step is to start nudging them to action!

Baseline: In the best case, you’ll be able to drive the organizational impact you need with contextualized nudges, but in some cases that isn’t quite enough. The next step is to work with the key teams to agree on baseline metrics for their performance.

Review: The final phase, which hopefully you won’t need to reach, is running a monthly or quarterly review that looks at each team’s performance, and reaching out to teams to advocate for prioritization if they aren’t sustaining their agreed-upon baselines.

I’ve seen this approach work and, more importantly, I’ve found it to be very scalable. It enables a company to concurrently maintain many baseline metrics without overloading its teams.

The approach is also effective because it tries to minimize top-down orchestration in favor of providing information to encourage teams themselves to adjust priorities.

The most interesting migration I ever participated in was Uber’s migration from Puppet-managed services to a fully self-service provisioning model in which any engineer at the company could spin up a new service in two clicks. Not only could they, they did, provisioning multiple services each day by the time the service was complete, and every newly hired engineer could spin up a service from scratch on their first day.

Migrations are both essential and frustratingly frequent as your codebase ages and your business grows: most tools and processes only support about one order of magnitude of growth22 before becoming ineffective, so rapid growth makes migrations a way of life. This isn’t because you have bad processes or poor tools—quite the opposite.

The fact that something stops working at significantly increased scale is a sign that it was designed appropriately to the previous constraints rather than being over-designed.

Lore tells us that Googlers have a phrase, “running to stand still,” to describe a team whose entire capacity is consumed in upgrading dependencies and patterns, such that the group can’t make forward progress on the product/system they own.

Migrations are the only mechanism to effectively manage technical debt as your company and code grow. If you don’t get effective at software and system migrations, you’ll end up languishing in technical debt.

The good news is that while migrations are hard, there is a pretty standard playbook that works remarkably well: de-risk, enable, then finish.

The first phase of a migration is de-risking it, and to do so as quickly and cheaply as possible. Write a design document and shop it with the teams that you believe will have the hardest time migrating. Iterate. Shop it with teams who have atypical patterns and edge cases. Iterate. Test it against the next six to twelve months of roadmap. Iterate.

After you’ve evolved the design, the next step is to embed into the most challenging one or two teams, and work side by side with those teams to build, evolve, and migrate to the new system. Don’t start with the easiest migrations, which can lead to a false sense of security.

Documentation and self-service tooling are products, and they thrive under the same regime: sit down with some teams and watch them follow your instructions, then improve them.

The last phase of a migration is deprecating the legacy system that you’ve replaced. This requires getting to 100 percent adoption, and that can be quite challenging.

Start by stopping the bleeding, which is ensuring that all newly written code uses the new approach. That can be installing a ratchet in your linters,25 or updating your documentation and self-service tooling.

My final tip for finishing migrations centers around recognition. It’s important to celebrate migrations while they’re ongoing, but the majority of the celebration and recognition should be reserved for their successful completion.

I believe that, at quickly growing companies, there are two managerial skills that have a disproportionate impact on your organization’s success: making technical migrations cheap, and running clean reorganizations.

There are two best kinds of reorganizations: The one that solves a structural problem. The one that you don’t do. There is only one worst kind of reorg: the one you do because you’re avoiding a people management issue.

Is the problem structural? Organization change offers the opportunity to increase communication, reduce decision friction, and focus attention; if you’re looking for a different change, consider if there’s a more direct approach.

Can you define clear interfaces for each team? Can you list the areas of ownership for each team? Have you created a gap-less map of ownership, such that each responsibility is owned by a team? Try to avoid implicitly creating holes of ownership. If you need to create explicit holes of ownership, that’s a better solution (essentially, defining unstaffed teams).

Explanation of reasoning driving the reorganization. Documentation of how each person and team will be impacted. Availability and empathy to help bleed off frustration from impacted individuals.

Discuss with heavily impacted individuals in private first. Ensure that managers and other key individuals are prepared to explain the reasoning behind the changes. Send an email out documenting the changes.

Be available for discussion. If necessary, hold an organization all-hands, but probably try not to. People don’t process well in large groups, and the best discussions take place in small rooms. Double down on doing skip-level one-on-ones.

Metrics26 align on outcomes while leaving flexibility around how the outcomes are achieved. Visions27 ensure that you agree on long-term direction while preserving short-term flexibility. Strategies28 confirm you have a shared understanding of the current constraints and how to address them. Organization design allows you to coordinate the evolution of a wider organization within the context of sub-organizations. Head count and transfers are the ultimate form of prioritization, and a good forum for validating how organizational priorities align across individual teams. Roadmaps align on problem selection and solution validation. Performance reviews coordinate culture and recognition.

The intersection that I’ve found between the individual’s and their manager’s perspective is the career narrative. I’ve explained these fabled documents a few times before, in “Roles over Rocket Ships”29 and “Partnering with Your Manager,”

A different approach would be to instead work on identifying the gaps that would keep you from being a strong head of engineering, and then start using your current role to help fill those gaps. A prototypical head of engineering will be skilled at organizational design, process design, business strategy, recruiting, mentoring, coaching, public speaking, and written communication.

Managers tend to have a strong sense of the business’s needs, and that gives them the superpower of finding the intersection of your interests and the business’s priorities. That translation is a creative pursuit, so don’t leave this entirely to your manager: participate as well! Brainstorm projects, research how folks at other companies have pursued similar goals, and educate your manager on aspects of your goals that they don’t know much about. (For example, engineers often have more conference speaking experience than engineering managers do.)

Answer the question you want to be asked. If someone asks a very difficult or challenging question, reframe it into one that you’re comfortable answering. Don’t accept a question’s implicit framing, but instead take the opportunity to frame it yourself. Don’t Think of An Elephant by George Lakoff 32 is a phenomenal, compact guide to framing issues.

Stay positive. Negative stories can be very compelling. They are quite risky, too! As an interviewee, find a positive framing and stick to it. This is especially true when it comes to competitors and controversy.

Speak in threes. Narrow your message down to three concise points, make them your refrain, and continue to refer ba...

One of the trickiest, and most common, leadership scenarios is leading without authority, and I’ve written about one of the styles that I’ve found surprisingly effective in those conditions. I call it Model, Document, Share.

Model. Start measuring your team’s health (maybe using short, monthly surveys) and your team’s throughput (do some lightweight form of task sizing, even if you just do it informally with a senior engineer on the team or with yourself), which will allow you to establish the baseline before your change.

Then just start running kanban. Don’t publicize it, don’t make a big deal about it, just start doing it with your team. Frame it as a short experiment with the team, and start trying it. Keep iterating on it until you’re confident it works. Have the courage to keep at it for a while, and also the courage to stop doing it if it doesn’t work after a month or two. Use the team’s health and throughput metrics to ground your decision about whether it’s working.

Document. After you’ve discovered an effective approach, document the problem you set out to solve, the learning process you went through, and the details of how another team would adopt the ...

Share. The final step is to share your documented approach, along with your experience doing the rollout, in a short email. Don’t ask everyone to adopt the practice, don’t lobby for change, just present the approach and your experience following it.

You’ll spend the majority of your time refining approaches that work effectively for your team, a bit of your time documenting how you did it, and almost no time trying to convince folks to change their approach. Strangely, in my experience, this has often led to more adoption than top-down mandates have.