An Elegant Puzzle: Systems of Engineering Management

by Will Larson

Companies are vastly complicated systems with dozens of feedback loops, and managing the kind and quality of freedom is simply another mechanism to be adjusted, albeit with immense care and consideration.

First, Tom DeMarco’s Slack13 has an excellent suggestion for a good starting state between positive and negative freedoms for engineering teams: generally follow the standard operating procedure (i.e., keep doing what you’re already doing, the way you’re doing it), but always change exactly one thing for each new project. Perhaps use a new database, a new web server, a different templating language, a static JavaScript front-end, whatever—but always change exactly one thing.

Second, I’m always terrified of getting on the wrong side of history, so I’ve spent some time considering how this discussion of freedoms relates to Ben Horowitz’s recent post on “Can Do vs. Can’t Do Cultures.”14 I read that article as describing how young companies that are focused on innovati...

A deeply flawed system can’t be saved by band-aids, but it can easily absorb your happiness to slightly extend its viability.

If you step back, you conserve your energy and avoid creating rifts by pushing others away in hero mode, and you will be ready to be a part of a new—hopefully more functional—system after the reset does occur.

Start by building out a map of your past year. Each time there was a change that meaningfully changed how you work, mark that down as a transition. These could be your direct manager changing, your team’s mission being redefined, a major reorganization, whatever—what counts is if how you work changed. What skills did you rely on to navigate the transition? What skills did the transition give you an opportunity to develop?

The good news is that both the stable eras and the transitions are great opportunities for growing yourself.

“If you’re offered a seat on a rocket ship, don’t ask what seat! Just get on.” —Sheryl Sandberg

The same advice applies if you’re within a quickly growing company or startup: don’t treat growth as a foregone conclusion. Growth only comes from change, and that is something you can influence.

No matter how many times you’ve done it, changing companies is stressful because of the requisite job search and interviewing. Having conducted hundreds of interviews across a number of companies, I feel a bit more prepared to interview each time I do it, but being back on the interviewee’s side of the table always leaves me humbled.

A good interview experience starts with being kind to your candidate.

Make sure your candidates know that you’re excited about them. I first encountered this idea reading Rands’s “Wanted” article,4 and he does an excellent job of covering it there.

Whenever you extend an offer to a candidate, have every interviewer send a note to them saying that they enjoyed the interview. (Compliment rules apply: more detailed explanations are much more meaningful.) At that point, as an interviewer, it can be easy to want to get back to your “real job,” but resist the temptation to quit closing just before you close: it’s a very powerful human experience to receive a dozen positive emails when you’re pondering if you should accept a job offer.

Interviewing is not a natural experience for anyone involved.

For direct feedback from candidates, I’ve started to ask every candidate during my “manager interview” sessions how the process has been and what we could do to improve. The feedback is typically surprisingly candid, although many candidates aren’t really prepared to answer the question after five hours of interviews.

The other, more common, mechanism is to have the recruiters do a casual debrief with each candidate at the end of the day.

Similarly, a frequent follow-up question is whether sourcing is a high-leverage task for engineering managers. I think it is: candidates are more excited to chat with someone who’d be managing them than they are to chat with a recruiter whom they’ll mostly work with during the interview process. Likewise, I think it’s a valuable signal showing that managers care about hiring enough to invest their personal energy and attention into it.

closing thought, the single clearest indicator of strong recruiting organizations is a close, respectful partnership between the recruiting and engineering functions.

The most sacred responsibilities of management are selecting your company’s role model, identifying who to promote, and deciding who needs to leave.

Somewhat unexpectedly, performance designations are usually not meant to be the primary mechanism for handling poor performance. Instead, feedback for weak performance should be delivered immediately. Waiting for performance designations to deal with performance issues is typically a sign of managerial avoidance. That said, it does serve as an effective backstop for ensuring that these kinds of issues are being addressed.

If you’re looking for more, Laszlo Bock’s Work Rules!13 is a good read.

Many employees rely entirely on their manager to come up with a step-by-step path to high performance. That only works when designation momentum is taking you in a direction you’re happy with. If it’s not, you need to be the active participant in your success.

Propose a set of clear goals to your manager, and iterate together toward an explicit agreement on the expectations to hit the designation you’re aiming for.

As a manager, you need to coordinate with your peers to ensure that you are opening the gate together in a consistent fashion. It’s easy to miss these moments, but if you do you may inadvertently eject individuals from their natural cohort of peers. You can usually fix this in a subsequent cycle, but you’ll have missed out on momentum. After each cycle, take an hour and try to guess when the gates might open next, and talk with your peers about

There are, surely, hundreds more interesting topics when it comes to how performance systems work in practice as opposed to in design. Although these systems seem quite simple, I keep learning something new each time I go through a performance cycle, and I suspect that is a widely shared experience.

Anyone who has flipped through Cracking the Coding Interview by Gayle Laakmann McDowell22 knows that evaluating candidates for a new role is a coarse science.

One tension in management is staying far enough out of the details to let folks innovate, yet staying near enough to keep the work well-aligned with your company’s value structures.

One of my coworkers, Davin Bogan,1 likes to say that “shipping is a habit,” and a well-run sprint both helps teams establish that habit and serves as a mechanism that creates visibility within a team that hasn’t quite gotten there yet. As a team’s direct manager, you can use this to ground concerns around individuals who might not be ramping up successfully, and, as you move into middle management, sprints are useful for debugging within your organization.

Don’t Think of an Elephant! Know Your Values and Frame the Debate by George Lakoff While written from a political perspective that some might find challenging, this book completely changed how I think about presenting ideas. You may be tempted to instead read Lakoff’s more academic writing, but I’d recommend reading this first as it’s much briefer and more readable.

Peopleware: Productive Projects and Teams by Timothy Lister and Tom DeMarco The book that has given generations of developers permission to speak on the challenges of space planning and open offices. Particularly powerful in grounding the discussion in data.

Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency by Tom DeMarco Documents a compelling case for middle managers as the critical layer where organizational memory

Finite and Infinite Games by James P. Carse Success in most life situations is about letting everyone continue to play, not about zero-sum outcomes. This seems pretty obvious, but for me it helped reset my sense of why I work.

Dynamo: Amazon’s Highly Available Key-Value Store” If you read only the abstract, you’d be forgiven for not being overly excited about the Dynamo paper. This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazon’s core services use to provide an always-on experience. To achieve this level of availability, Dynamo sacrifices consistency under certain failure scenarios. It makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.

Hints for Computer System Design” Butler Lampson3 is winner of the ACM Turing Award (among other awards), and worked at the Xerox PARC. This paper concisely summarizes many of his ideas around systems design, and is a great read. In his words: Studying the design and implementation of a number of computers has led to some general hints for system design. They are described here and illustrated by many examples, ranging from hardware such as the Alto and the Dorado to application programs such as Bravo and Star.

Twelve-Factor App4 wants to serve as a checklist for operable applications).

Processing” Apache Kafka9 has become a core piece of infrastructure for many internet companies. Its versatility lends it to many roles, serving as the ingress point to “data land” for some and as a durable queue for others. And that’s just scratching the surface.

Borg, Omega, and Kubernetes” While the individual papers for each of Google’s orchestration systems (Borg, Omega, and Kubernetes) are worth reading in their own right, this article is an excellent overview of the three:

The Byzantine Generals Problem” Another classic Leslie Lamport paper on consensus, the Byzantine Generals Problem explores how to deal with distributed actors that intentionally or accidentally submit incorrect messages:

“No Silver Bullet—Essence and Accident in Software Engineering” A seminal paper from the author of The Mythical Man-Month, “No Silver Bullet” expands on discussions of accidental versus essential complexity, and argues that there is no longer enough accidental complexity to allow individual reductions in that accidental complexity to significantly increase engineer productivity.