Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value

by Teresa Torres

Chip and Dan Heath, in their book Decisive, outline four villains of decision-making that lead to poor decisions. The first villain is looking too narrowly at a problem. This is exactly why we want to explore multiple ways of framing the opportunity space. The second villain is looking for evidence that confirms our beliefs. This is commonly known as confirmation bias. We’ll be discussing this bias often throughout the book. We’ll be exploring several habits that will help us overcome this bias and ensure that we are considering both confirming and disconfirming evidence. The third villain is letting our short-term emotions affect our decisions. In the product world, this often shows up when we fall in love with our ideas. The fourth villain is overconfidence. This, too, is common in the product world. We are often sure our ideas will be runaway successes.

Poor decision making

Assigning a team a leading indicator is always better than assigning a lagging indicator.

Leading vs lagging indicators

Assigning product outcomes to product trios increases a sense of responsibility and ownership. If a product team is assigned a business outcome, it’s easy for the trio to blame the marketing or customer-support team for not hitting their goal. However, if they are assigned a product outcome, they alone are responsible for driving results. When multiple teams are assigned the same outcome, it’s easy to shift blame for lack of progress.

Defining product outcomes for product

Common goal-setting advice encourages us to set specific, measurable, achievable, relevant, and time-bound (S.M.A.R.T.) goals.

SMART goals

If your product leader is asking you to deliver an outcome with no input from your team, try these tips to shift to a two-way negotiation: If you are being asked to deliver a business outcome, try mapping out which product outcomes might drive that business outcome, and get feedback from your leader. If you are being asked to deliver a product outcome, ask your leader for more of the business context. Try asking, “What business outcomes are we trying to drive with this product outcome?”

Business vs product outcome. Lagging vs leading

What needs, pain points, and desires matter most to this customer?

How to interview customers

Instead of asking, “What criteria do you use when purchasing a pair of jeans?”—a direct question that encourages our participant to speculate about their behavior—we want to ask, “Tell me about the last time you purchased a pair of jeans.” The story will help us uncover what criteria our participant used when purchasing a pair of jeans, but because the answer is situated in a specific instance (an actual time when they bought jeans), it will reflect their actual behavior, not their perceived behavior. Finding the right story question can be challenging.

When was the last time you bought jeans? Tell me about it

Even so, you may still have to dig to uncover the full story. You can use common storytelling tactics to help you do so. A good story has a protagonist who encounters experiences on a timeline. Temporal prompts are one of the most effective ways to guide the participant through their own story. You can ask, “Start at the beginning. What happened first?” You can use the experience map you created in Chapter 4 to help guide your participant. Prompt for the beginning of the story. If your participant isn’t sure where to start, you can further prompt, “Where were you? Set the scene for me.” As the participant tells their story, you can encourage them to keep going by asking them, “What happened next?” Sometimes they might skip a few steps, and you may need to ask, “What happened before that?” Thinking about their story as having a beginning, a middle, and an end can help you guide the participant. Use your customer-experience map to help you track their story. Listen for specific nodes. Ask about nodes that were left out of the story. Stories also take place in specific locations; protagonists encounter challenges, and they receive help from supporting characters. Other characters might present obstacles or interfere with the protagonist’s progress. Keep these elements in mind as they’ll help you tease out what the participant forgets. You might ask, “Who was with you?” “What challenges did you encounter?” “How did you overcome that challenge?” “Did anyone help you?” You’ll no...

How to excavate a story

An interview snapshot is a one-pager designed to help you synthesize what you learned in a single interview. It’s how you are going to turn your copious notes into actionable insights. Your collection of snapshots will act as a reference or index to the customer knowledge bank you are building through continuous interviewing. After you’ve conducted even a handful of interviews, let alone the dozens you will conduct each year, interviews will start to blur together.

Interview snapshot template

I recommend that teams assess opportunities using the following criteria: opportunity sizing, market factors, company factors, and customer factors. Opportunity sizing helps us answer the questions: How many customers are affected and how often? However, we don’t need to size each opportunity precisely. This can quickly turn into a never-ending data-gathering mission. Instead, we want to size a set of siblings against each other. For each set that we are considering, we want to ask, “Which of these opportunities affects the most customers?” and “the most often?” We can and should make rough estimates here. You can use behavioral data (e.g., site analytics, sales-funnel analytics), support tickets, sample surveys, or even your interview snapshots, to quickly evaluate which opportunities are impacting the most customers. It’s important, however, to distinguish how many customers from how often. Every customer might be impacted by an opportunity, but the need or pain point might arise only occasionally. Addressing this opportunity will have a different impact than addressing an opportunity that impacts some customers all the time. Market factors help us evaluate how addressing each opportunity might affect our position in the market. Depending on the competitive landscape, some opportunities might be table stakes, while others might be strategic differentiators. Choosing one over the other will depend on your current position in the market. A missing table stake could torpedo...

How to assess oportunities

Jeff Bezos, founder and CEO of Amazon, made this exact argument in his 2015 letter to shareholders,33 where he introduced the idea of Level 1 and Level 2 decisions. He describes a Level 1 decision as one that is hard to reverse, whereas a Level 2 decision is one that is easy to reverse. Bezos argues that we should be slow and cautious when making Level 1 decisions, but that we should move fast and not wait for perfect data when making Level 2 decisions.

Level 1 and 2

When you get stuck, start with your competition. But then look wider. Ask yourself: What other industries have solved similar problems? They don’t need to be similar or even be in an adjacent industry. You are looking for similarities in the target opportunity. For example, if you work for a job board and you are helping recruiters evaluate job candidates, you can look at other job boards, but you can also look at how online shopping sites help shoppers evaluate products, you can look at how travel aggregators help travelers choose hotels, and you can look at how insurance companies present different policies. These industries are unrelated to each other, but they are each solving analogous problems. Additionally, when you are stuck, you can start to consider what your extreme users might need. What would a power user want? What does the first-time user need? What about people with different disabilities? How about people who live in remote locations or bustling cities? Young people? Senior citizens? Your extreme users will vary by product, but thinking about the needs of different types of users as they relate to your target opportunity can help you generate more ideas that may work for everyone. And finally, don’t be afraid to consider wild ideas. Some people don’t like this suggestion, because wild ideas are rarely pursued. But wild ideas can improve more feasible ideas. This is where the power of mixing and matching different solutions to identify even-better ideas com...

How to get unstuck

The escalation of commitment46 is a bias in which the more we invest in an idea, the more committed we become to that idea. The more the city explored this idea (and this idea alone), the more committed to the idea they became—despite its flaws.

What is escalation of committment

In the opening quote of this chapter, Marty Cagan, author of INSPIRED, highlights that the best product teams complete a dozen or more discovery iterations every week. This pace is possible only when we step away from the concept of testing ideas and instead focus on testing the assumptions that need to be true in order for our ideas to succeed. By explicitly enumerating our assumptions, we can start to look for both confirming and disconfirming evidence to either support or refute each assumption. Additionally, assumption testing is generally quicker than idea testing, and the faster pace helps us to guard against the escalation of commitment. The less time we invest in an idea, the less likely we are to fall in love with it.

Why assumptions are good for product teams

Types of Assumptions Assumptions come in all shapes and sizes. As product trios, we are primarily concerned with assumptions in the following categories:47 Desirability assumptions: Does anyone want it? Will our customers get value from it? As we create solutions, we assume that our customers will want to use our solution, that they will be willing to do the things that we need them to do, and that they’ll trust us to provide those solutions. All of these types of assumptions fall into the desirability category. Viability assumptions: Should we build it? There are many ideas that will work for our customers but won’t work for our business. If we want to continue to serve customers over time, we need to make sure that our solutions are viable—that they create a return for our business. This typically means that the idea will generate more revenue than it will cost to build, service, and maintain. However, some ideas are designed to be loss leaders and instead contribute to another business goal besides revenue. But somewhere down the line, the idea must create enough value for the business to be worth the effort to create and maintain. Feasibility assumptions: Can we build it? We primarily think about feasibility as technical feasibility. Is it technically possible? Feasibility assumptions, however, can also include, “What’s feasible for our business?” For example, will our legal or security team allow for it? Will our culture support it? Does it comply with regulations?48 ...

Types of assumptions

Let’s walk through an example. Suppose we are working at our streaming-entertainment company, and we are exploring three different solutions (setting up a good compare-and-contrast decision) for the target opportunity “I want to watch live sports.” Integrate local networks (e.g., ABC, CBS, NBC) into our service License broadcast rights directly from the different sports leagues and serve the sporting events up ourselves Bundle our streaming service with a partner who streams live sports To story map our first idea—integrating local networks—we need to start by identifying the key players. In this case, it’s easy to make the mistake of thinking our key players are our business-development folks and their equivalent at the respective local channel providers. But remember, with story mapping, we want to assume the solution already exists. We aren’t story mapping what it would take to build the service—we are story mapping what the key players need to do for our consumer to get value from it. So, in this case, our key players might be the sports consumer (i.e., our subscriber), our streaming platform, and any number of local TV channel partners. Our story map might include the following steps: Our subscriber comes to our platform to watch live sports. Our platform has live sports options for our subscriber to choose from. Our subscriber browses the options and chooses a sport that is currently on a local TV channel. Our local channel provider’s content is available to stream. ...

Story mapping is a tool for idea testing - here we havexq clear exAmple

Let’s return to our example to see how this works. From our first step of the story map, “Our subscriber comes to our platform to watch sports,” we can generate the following assumptions: Desirability: Our subscriber wants to watch sports. Desirability: Our subscriber wants to watch sports on our platform. Usability: Our subscriber knows they can watch sports on our platform. Usability: Our subscriber thinks of our platform when it’s time to watch sports. Feasibility: Our platform is available when our subscriber wants to watch sports. From our second step, “Our platform has live sports options for our subscriber to choose from,” we can generate the following assumptions: Desirability: Our platform has the sports our subscriber wants to watch. Usability: Our subscriber can find where to go on our platform to find sports. Feasibility: We know what sports are available right now. Feasibility: We can display what sports are available right now. From our third step, “Our subscriber chooses a sport that is currently on a local TV channel,” we can generate the following assumptions: Desirability: Our subscriber wants to watch a sport that is on a local TV channel. Usability: Our subscriber can find the sport they want to watch. Usability: Our subscriber knows how to select and play the sport they want to watch. From our fourth step, “Our local channel provider’s content is available,” we can generate the following assumptions: Feasibility: The local provider’s source content wil...

Testing assumptions using story mapping

Pre-mortems are a great way to generate assumptions. They leverage prospective hindsight—a technique where you imagine what might happen in the future. A pre-mortem encourages you to ask, “Imagine it’s six months in the future; your product or initiative launched, and it was a complete failure. What went wrong?” As you generate reasons for why your product or service might fail, you are exposing assumptions that your idea depends upon that may not be true. The success of pre-mortems, however, hinges on one key factor—phrasing the question as if the outcome is certain. In our case, that means we have to consider that the product or service did fail, not that it might fail.

How to do an effective pre mortem

Be specific. Why will your solution address the target opportunity? Your answer will contain one or more assumptions that you’ll want to capture. For example, “Adding local channels will allow our subscribers to watch live sports” because “The sports that our subscribers want to watch are on local channels” because “Most of the major sports are on local channels” and “Our subscribers are more likely to want to watch major sports.” Each phrase in quotes is an assumption we can test. Why does addressing the opportunity “I want to watch live sports” drive the outcome “Increase weekly viewer minutes”? “People will watch sports in addition to what they already watch.” “Even if people cut out other shows, sporting events are long, and their individual viewing sessions will be longer.” “If individual viewing sessions are longer, weekly viewing minutes will go up.”

Generating assumptions using opportunity trees

With assumption mapping, you’ll be quickly evaluating each assumption on two dimensions. I recommend you start with assessing “How much do we know about this assumption?” In other words, what evidence do we already have that tells us this assumption is true or false? If we have a lot of evidence that it’s true, then we would place the assumption on the left side of the x-axis. If we have very little evidence, we would place it on the right side of the x-axis. Next, we want to assess how important this assumption is to the success of your idea. Now, all assumptions are important to the success of your idea as currently designed. However, some assumptions can be easily worked around if they turn out not to be true. Others may be much harder to work around. These would make these assumptions more important. The more important an assumption is, the higher on the y-axis it gets placed. The less important it is, the lower on the y-axis it gets placed. Each

Assumption testing

Avoid These Common Anti-Patterns As you work to identify the hidden assumptions behind your ideas, be sure to avoid these common anti-patterns: Not generating enough assumptions. Generating assumptions, like ideating, is intended to be a divergent exercise. The goal is to identify as many “gotchas” as you can to increase the chance that you generate the riskiest ones. However, many teams dramatically underestimate how many assumptions underlie their ideas. When I do these exercises, I often generate 20–30 assumptions for even a simple idea. If that sounds overwhelming, remember, you won’t need to test all of these assumptions. Most of them will be harmless. You’ll use the assumption-mapping exercise to quickly find the riskiest ones. However, if you don’t generate the riskiest assumptions, the mapping exercise won’t help you sort something that you haven’t uncovered. Use the five assumption categories and the exercises in this chapter to help you generate as many assumptions as you can. Phrasing your assumptions such that you need them to be false. Generating assumptions can be a bit of a devil’s-advocate exercise. You are looking for what might go wrong with your ideas. As a result, you might be tempted to phrase your assumptions negatively. For example, if you need your users to log in to your service, you might phrase your assumption as, “Customers won’t remember their password.” However, this is backwards. You need customers to remember their password for your idea to ...

Antipatterns to look out for

To avoid this situation, we want to get specific with our evaluation criteria. Instead of saying, “Some people choose sports,” we want to say, “At least 3 out of 10 people choose sports.” We want to define both how many people we’ll test with and how many people need to exhibit the behavior that we expect to see. By defining these criteria upfront, you are doing two things. First, you are aligning as a team around what success looks like so that you all know how to interpret the results. This will help to ensure that your assumption tests are actionable. And second, you are helping to guard against confirmation bias. Remember, confirmation bias makes us more likely to see the evidence that our idea will succeed than the evidence that it might not succeed. If we don’t define our success criteria upfront, when we try to interpret the results, our brains will actively look for evidence that supports the assumption, and we’ll likely miss the evidence that could refute it. To avoid this, we want to define what success looks like upfront (before we see the results). So how do we choose the numbers? This is a subjective decision. Your goal is to find the right balance between speed of testing and what aligns your team around an actionable outcome. You want to test your assumption with as few people as possible (as it will be faster) but with the number of people that still gives your team the information they need to act on the data.

Defining criteria to measure assumption success

Our goal as a product team is not to seek truth but to mitigate risk.

What is the voal fr a product team

There are two tools that should be in every product team’s toolbox—unmoderated user testing and one-question surveys. Unmoderated user-testing services allow you to post a stimulus (e.g., a prototype) and define tasks to complete and questions to answer. Participants then complete the tasks and answer the questions on their own time. You get a video of their work. These types of tools are game changers. Instead of having to recruit 10 participants and run the sessions yourself, you can post your task, go home for the night, and come back the next day to a set of videos ready for you to watch.

Tools to test ideas

Desirability isn’t enough. Viability is the key to long-term success.

Viability above all

One of the hardest challenges with opportunity selection is identifying the right opportunity for right now. However, a round of assumption tests should help you assess fit quickly. These stories are a good reminder of why we want to run quick tests rather than overinvest in the best tests.

Choose the right opportunity for now and not tomorrow

When meeting with stakeholders, start at the top of your tree. Remind your stakeholders what your desired outcome is. Ask them if anything has changed since you last agreed to this outcome. This sets the scope for the conversation. Share how you mapped out the opportunity space. Highlight the top-level opportunities. Drill into the detail only when and where they ask for it. Ask them if you missed anything. Consider that they may have knowledge of opportunities that you might have missed. Capture their suggestions. You can always vet them in your future customer interviews. Share how you assessed and prioritized the opportunity space. Use the tree structure to walk through each decision you made. Choose the appropriate level of detail based on the stakeholder you are talking to. Ask them if they would have made a different decision at each decision point. Consider their feedback. Share more context about your target opportunity. Help them fully understand the customer need or pain point you intend to address. Use your interview snapshots to help your stakeholders empathize with your customers. Answer their questions. This step is critical. Your stakeholder needs to fully understand the opportunity you are pursuing before you share solutions with them. This is what sets the context for how to evaluate solutions and moves the conversation away from opinions and preferences. Share the solutions you generated. Ask them if they have any of their own ideas. Make sure you capture...

How to communicate results to stakeholders

“Our goal is to reduce the number of lost sales from not having a journey builder (outcome). We interviewed customers and learned that existing journey builders are too complex; marketers don’t know how to get started (opportunity one), and, when they do, their journeys are hard to maintain (opportunity two), and they often create redundant journeys (opportunity three). We decided to focus on reducing the complexity by helping marketers to zoom out from the messages they are sending to focus on the goals they are trying to achieve. We explored a few different ways to do this, but the most promising one is our lifecycle-maps idea. In testing, we found that marketers had no problem getting started and that they loved the high-level view on their work.”

How to present our work to stakeholders