Working Backwards: Insights, Stories, and Secrets from Inside Amazon

by Colin Bryar

An Amazon quarterly business review, for instance, might be broken down like this instead: Introduction Tenets Accomplishments Misses Proposals for Next Period Headcount P&L FAQ

Some groups at Amazon go around the room, ask for high-level feedback, then pore over the document line by line. Other groups ask a single individual to give all their feedback on the entire document, then ask the next person in the audience to do the same. Just pick a method that works for you—there’s no single correct approach.

During the discussion stage, it’s also important that notes be taken on behalf of the entire audience, preferably by someone knowledgeable about the subject who is not the primary presenter. The presenter is generally too involved in answering questions to capture effective notes at the same time. If I don’t see anyone taking notes at the discussion stage, I will politely pause the meeting and ask who is going to do so. It’s vital that we capture and record the salient points of the ensuing discussion, as those comments become part of the output of the narrative process.

You are not just commenting on a document, you’re helping to shape an idea, and thereby becoming a key team member for that business.

Jeff has an uncanny ability to read a narrative and consistently arrive at insights that no one else did, even though we were all reading the same narrative. After one meeting, I asked him how he was able to do that. He responded with a simple and useful tip that I have not forgotten: he assumes each sentence he reads is wrong until he can prove otherwise. He’s challenging the content of the sentence, not the motive of the writer. Jeff, by the way, was usually among the last to finish reading.

“If Amazon is really customer obsessed, why do we penalize the 99 percent of customers who are honest and want to return an item by making them wait until our returns department receives the item to make sure it’s the right item and that it’s not damaged?” This type of thinking—in which you assume there is something wrong with the sentence—led Amazon to create the no-hassle return policy, which specifies that the customer should get a refund even before Amazon receives the returned goods. (The refund is reversed for the small percentage of people who do not send back the item.)

Working Backwards is a systematic way to vet ideas and create new products.

In the end, what turned out to work best was relying on the core Amazon principle of customer obsession and a simple yet flexible way of writing narrative documents. These two elements form the Working Backwards process—starting from the customer experience and working backwards from that by writing a press release that literally announces the product as if it were ready to launch and an FAQ anticipating the tough questions. While this next section describes the evolution of Working Backwards as seen through the experience of the digital team,

As I was to learn, the process for creating the digital media business would be quite different because there was so much more to creating a great digital media customer experience than simply adding the next retail category to the Amazon website.

It was clear that half-baked mock-ups were no better, perhaps worse, than no mock-ups at all. To Jeff, a half-baked mock-up was evidence of half-baked thinking. And he was quick to say so, often using strong language to make his point inescapably clear. Jeff wanted us to know that we couldn’t just charge down the first available and most convenient path to chase after this opportunity. We needed to think through our plan in detail.

A common question asked by executives when reviewing the product features in the PR is “so what?” If the press release doesn’t describe a product that is meaningfully better (faster, easier, cheaper) than what is already out there, or results in some stepwise change in customer experience, then it isn’t worth building.

The PR/FAQ process creates a framework for rapidly iterating and incorporating feedback and reinforces a detailed, data-oriented, and fact-based method of decision-making. We found that it can be used to develop ideas and initiatives—a new compensation policy, for example—as well as products and services. Once your organization learns how to use this valuable tool, it is addicting. People start to use it for everything.

Over time, we refined and normalized the specifications for the PR/FAQ. The press release (PR) portion is a few paragraphs, always less than one page. The frequently asked questions (FAQ) should be five pages or less. There are no awards for extra pages or more words. The goal isn’t to explain all the excellent work you have done but rather to share the distilled thinking that has come from that work.

The more PR/FAQs they read, and the more products they build and launch using the PR/FAQ process, the more capable they become at identifying the omissions and flaws in the author’s thinking.

Consumer Needs and Total Addressable Market (TAM) How many consumers have this need or problem? How big is the need? For how many consumers is this problem big enough that they are willing to spend money to do something about it? If so, how much money would they be willing to spend? How many of these consumers have the characteristics/capabilities/constraints necessary to make use of the product?

These consumer questions will enable you to identify the core customers by filtering out those who don’t meet the product constraints. In the case of Melinda, for example, you would eliminate people who: don’t have enough space on their front porch for this product don’t have a front porch or similar outdoor area with access to the street at all (e.g., most apartment dwellers) don’t have a suitable source of electricity wouldn’t be pleased to have a large storage/mailbox on their front porch don’t receive many deliveries or deliveries that need refrigeration don’t live in areas where package theft is a problem don’t have interest or ability to pay $299 to answer the need

Research into these questions (e.g., how many detached homes are there in a given area?) can help you estimate the total addressable market (TAM), but like any research, there will be a wide error bar. The author and readers of the PR/FAQ will ultimately have to decide on the size of the TAM based on the data gathered and their judgment about its relevance. With Melinda, this process would likely lead to the conclusion that the TAM is in fact pretty small.

Economics and P&L What are the per-unit economics of the device? That is, what is the expected gross profit and contribution profit per unit? What is the rationale for the price point you have chosen for the product? How much will we have to invest up front to build this product in terms of people, technology, inventory, warehouse space, and so on?

Price is a key variable in the authoring of your PR/FAQ. There may be special assumptions or considerations that have informed your calculation of the price point—perhaps making it relatively low or unexpectedly high—that need to be called out and explained. Some of the best new product proposals set a not-to-exceed price point because it forces the team to innovate within that constraint and face the tough trade-offs early on.

Dependencies How will we convince couriers (USPS, UPS, FedEx, Amazon Fulfillment, Instacart, etc.) to actually use this device instead of their current/standard delivery methods? How will we ensure that couriers (who don’t work for you and over whom you have no control) will use the Melinda UI properly and bother to actually put packages in it instead of just leaving the package by the front door like they typically do? Won’t it take more time (which is precious) for them to make a delivery than it does today? What third-party technologies are we dependent on for Melinda to function as promised?

A common mistake among less-seasoned product managers is to not fully consider how third parties who have their own agendas and incentives will interact with their product idea, or what potential regulatory or legal issues might arise.

Feasibility What are the challenging product engineering problems we will need to solve? What are the challenging customer UI problems we will need to solve? What are the third-party dependencies we will need to solve? How will we manage the risk of the up-front investment required?

These questions are intended to help the author clarify to the reader what level of invention is required and what kind of challenges are involved in building this new product. These criteria vary from product to product, and there are different types of challenges ranging from technical to legal to financial to third-party partnerships and customer UI or acceptance.

It is important to note that, during our time with Amazon, most PR/FAQs never made it to a stage where they were launched as actual products. What this means is that a product manager will put in a lot of time exploring product ideas that never get to market.

The fact that most PR/FAQs don’t get approved is a feature, not a bug. Spending time up front to think through all the details of a product, and to determine—without committing precious software development resources—which products not to build, preserves your company’s resources to build products that will yield the highest impact for customers and your business.

Leadership and management are often about deciding what not to do rather than what to do. Bringing clarity to why you aren’t doing something is often as important as having clarity about what you are doing.

But this is not a drawback. It is, in fact, a huge benefit of the process—a considered, thorough, data-driven method for deciding when and how to invest development resources. Generating and evaluating great ideas is the real benefit of the Working Backwards process.

companies in general, have very little ability to directly control output metrics. What’s really important is to focus on the “controllable input metrics,” the activities you directly control, which ultimately affect output metrics such as share price.

Shortly after that holiday season we held a postmortem, out of which was born the Weekly Business Review (WBR). The purpose of the WBR was to provide a more comprehensive lens through which to see the business.

the initial Amazon WBR, they turned to a well-known Six Sigma process improvement method called DMAIC, an acronym for Define-Measure-Analyze-Improve-Control.1 Should you decide to implement a Weekly Business Review for your business, we recommend following the DMAIC steps as well. The order of the steps matters. Progressing through this metrics life cycle in this order can prevent a lot of frustration and rework, allowing you to achieve your goals faster.

Amazon takes this philosophy to heart, focusing most of its effort on leading indicators (we call these “controllable input metrics”) rather than lagging indicators (“output metrics”).

Input metrics track things like selection, price, or convenience—factors that Amazon can control through actions such as adding items to the catalog, lowering cost so prices can be lowered, or positioning inventory to facilitate faster delivery to customers. Output metrics—things like orders, revenue, and profit—are important, but they generally can’t be directly manipulated in a sustainable manner over the long term. Input metrics measure things that, done right, bring about the desired results in your output metrics.

One of the metrics we initially chose for selection was the number of new detail pages created, on the assumption that more pages meant better selection. Once we identified this metric, it had an immediate effect on the actions of the retail teams. They became excessively focused on adding new detail pages—each team added tens, hundreds, even thousands of items to their categories that had not previously been available on Amazon. For some items, the teams had to establish relationships with new manufacturers and would often buy inventory that had to be housed in the fulfillment centers. We soon saw that an increase in the number of detail pages, while seeming to improve selection, did not produce a rise in sales, the output metric.

number of detail pages, which we refined to number of detail page views (you don’t get credit for a new detail page if customers don’t view it), which then became the percentage of detail page views where the products were in stock (you don’t get credit if you add items but can’t keep them in stock), which was ultimately finalized as the percentage of detail page views where the products were in stock and immediately ready for two-day shipping, which ended up being called Fast Track In Stock.

we decided that in each category, we wanted 95 percent of detail page views to display a product that was in stock and ready for immediate shipping.

The deck represents a data-driven, end-to-end view of the business. While departments shown on org charts are simple and separate, business activities usually are not. The deck presents a consistent, end-to-end review of the business each week that is designed to follow the customer experience with Amazon. This flow from topic to topic can reveal the interconnectedness of seemingly independent activities.

Therefore, at Amazon, everyone from the individual contributor to the CEO must have detailed knowledge of input metrics to know whether the organization is maximizing outputs.

It is worth noting here that, at Amazon, even the most senior executives review the full WBR deck of metrics, including all the inputs and outputs. Metrics—as well as anecdotes about the customer experience—are the area where the leadership principle Dive Deep is most clearly demonstrated by senior leaders. They carefully examine the trends and changes in the metrics; audit incidents, failures, and customer anecdotes; and consider whether the input metrics should be updated in some way to improve the outputs.

We focus on variances and don’t waste time on the expected People like talking about their area, especially when they’re delivering as expected, and even more so when they exceed expectations, but WBR time is precious. If things are operating normally, say “Nothing to see here” and move along. The goal of the meeting is to discuss exceptions and what is being done about them. The status quo needs no elaboration.

Every week they review the deck before the WBR and respond by discussing what action they plan to take to address the variances. This is a hard-earned lesson; we’ve seen a metric owner display their metrics in front of a group where it’s obviously the first time that person has seen the data. That’s a big mistake, a waste of everyone else’s time, and will most definitely result in a kerfuffle with the senior leader in the room. By the time the WBR meeting occurs, each metric owner should have thoroughly analyzed the metrics they own.

Sometimes even the well prepared are hit with a question to which the right answer isn’t immediately apparent. In that case, the owner is expected to say something like, “I don’t know. We are still analyzing the data and will get back to you.” This is preferable to guessing, or worse, making something up on the fly.

Strong culture

We try not to browbeat (it’s not the Inquisition) It’s okay to dig into a meaningful variation that needs more attention, and to point out when high standards have not been met. Still, success demands an environment where people don’t feel intimidated when talking about something that went wrong in their area.

If people become afraid of pointing out their own mistakes because they will feel humiliated in front of their peers, it’s human nature for them to do whatever they can to hide those mistakes in future meetings. Variances that get glossed over are lost learning opportunities for everybody. To prevent this, mistakes should be acknowledged as a chance to take ownership, understand the root cause, and learn from the experience. Some tension is unavoidable and appropriate, but we think it’s better to establish a culture where it’s not just okay, it’s actually encouraged to openly discuss mistakes.

Adding YOY growth rates in addition to the underlying metric you are measuring is a great way to spot trends.

At first, an investigation into her complaint revealed that an edge-case bug—another way of saying a rare occurrence—had bumped her card balance over the limit. Many companies would dismiss such cases as outliers, and thus not worthy of attention, on the assumption that they rarely happen and are too expensive to fix. At Amazon, such cases were regularly attended to because they would happen again and because the investigation often revealed adjacent problems that needed to be solved. What at first looked to be just an edge case turned out to be more significant. The bug had caused problems in other areas that we did not initially notice. We quickly fixed the problem for her and for all other impacted customers.

Data and anecdotes make a powerful combination when they’re in sync, and they are a valuable check on one another when they are not.

what we called “the big red button” on their control screen. Once that button was clicked, two things happened: the “Add to Cart” and “1-Click” buttons would disappear from the product page so no customers could buy that product, and the category manager would immediately be notified that purchasing for one of their products had been disabled until they could investigate and fix the issue.

It proved once again that giving employees the right tools to solve problems and relying on their good judgment is a powerful combination. It is used widely across Amazon.

Contradictory as this may sound, variation in data is normal. And unavoidable. It’s therefore critical to differentiate normal variation (noise) from some fundamental change or defect in a process (signal).

and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”