The Lean Product Playbook: How to Innovate with Minimum Viable Products and Rapid Customer Feedback

by Dan Olsen

This is reminiscent of the “five whys” technique. Asking a customer “why” too many times can make them feel defensive, so it's a good idea to mix it up with other phrases such as “Could you please tell me more about that?,”

It's common for users to ask the moderator questions during the test. For example, a user might ask a moderator, “So, should I click here to log in?” Rather than replying yes or no, a good moderator might ask, “What would you expect to happen if you clicked there?,” or might say, “Do whatever you would do if you were by yourself.” Good moderators often use the judo move of answering a question with a question.

Asking open versus closed questions is less a matter of moderator bias and has more to do with the moderator's skill level.

A helpful technique is to get in the habit of saying your next question in your mind before you verbalize it. That way, if it is a closed question, you can change it to an open-ended question before you pose it to the customer.

You can let users know that you will address their questions at the end of the test.

If the user couldn't understand your product, it's clearly your company's fault—not the user's.

ask, “On a scale of 0 to 10, with 10 being best, how valuable did you find the product?,” or “Based on what you saw today, how likely would you be to use the product?” You could also ask, “How easy to use was the product?” You can ask verbally or you can give the customer a short form to fill out,

The wrap-up section is also the time to answer any questions that came up during the test or that the customer has at the end.

Feedback on usability has to do with how easy it is for customers to understand and use your product, whereas feedback on product-market fit has to do with how valuable they find your product.

“Build” simply means having something that you can test with customers, which could be a live product or design artifacts, such as wireframes or mockups. “Design something to test” is a broader, more accurate description, so I prefer the label “design” for this step. The goal is to identify and create what will let you test your hypotheses while consuming the least resources. “Measure” implies numerical data—but keep in mind that “measure” doesn't have to be as quantitative as it sounds.

The “learn” step is interesting. There are actually two things going on in this step. First, you are learning new things from the results of each test. Second, you use what you learn to modify the hypotheses that led to the test you just ran.

As you go through this loop, you transition from problem space to solution space and back again.

To summarize: you test and improve your problem space thinking by showing customers a product or design artifact in the solution space and soliciting their feedback on it.

A pivot is larger in magnitude than the change you normally see as you iterate along the path you have chosen; it means a significant change in direction.

You shouldn't change direction every time you hit a rough patch, nor should you drop what you're doing to chase each cool new idea you come up with, also known as shiny object syndrome.

You should consider pivoting if you just don't seem to be achieving gains in product-market fit after several rounds of trying to iterate.

way, if you haven't yet identified a customer archetype that is very excited about your MVP, then you should consider pivoting.

When developers are asked to estimate the effort for a task, they take into account the “known knowns.” Skilled estimators will also account for the known unknowns in their estimates.

Even in Agile, you should design before you code; you're just doing so in much smaller increments.

When the risk of failure or the cost of making changes is too high, it's better to spend more time gaining a higher level of confidence before starting implementation.

Agile encourages early and continuous delivery of working software with a mindset focused on creating value for customers.

Stories that are too big to complete in one iteration are called epics, which must be broken down before they can be accepted into a sprint.

I.E one large feature that needs to be split down.

For many teams, “done” means a product that could be shipped, called a “shippable product” or a “potentially releasable product.”

Regardless of your deployment process, the goal is to ensure the product is in a shippable state at the end of the sprint.

In kanban, the quantity of active work is managed by constraining the amount of “work in progress” or WIP. The team decides on the maximum number of cards each column can contain, which is called a WIP limit. Team members pull work items forward sequentially through each state of work. However, they can only move a work item to the next column if that column has spare capacity.

Most of the time, your backlog is like ice; the rank order is frozen and fixed. But when new requirements come in or priorities change, you briefly melt the ice into liquid water so you can rearrange things. Once you're done reordering your backlog, you freeze it again.

A major bug that you don't detect until after you launch your product is much more costly than one found during development. First, it negatively impacts customers. Second, it is usually more time consuming for the team to figure out the root cause of production bugs and fix them because they are no longer actively working on that code. Third, because the defect is live, the customer pain persists until the bug is fixed and you deploy the new code to customers.

The team should test two different aspects of the product when they build new functionality or make improvements to existing functionality. The first, called validation testing, checks to see if the new or improved functionality works as expected—that it is consistent with the associated user stories and design artifacts.

The second aspect of product testing is to ensure that none of the other existing functionality was inadvertently broken during the process of building the new or improved functionality.

test-driven development, a technique where developers write automated tests before they write code.

Continuous Integration

Continuous Deployment

In order to work well, continuous deployment requires a robust analytics system.

Ellis advocates, as I do, that you should not invest in trying to grow your business until after you have achieved product-market fit.

“How would you feel if you could no longer use [product X]?” The four possible responses are: Very disappointed Somewhat disappointed Not disappointed (it isn't really that useful) N/A—I no longer use [product X]

Peter Drucker, you can't manage what you don't measure.

product. Dave called his framework “Startup Metrics for Pirates” because if you make an acronym for his five metrics—acquisition, activation, retention, revenue, and referral—it spells “AARRR!”

The Equation of Your Business for a Subscription Revenue Model

Achieving Profitability

Your rate of improvement depends on how quickly you can identify and implement good ideas.

If your coefficient is greater than one, then your product is officially “viral,” which means that each current user generates more than one new user, resulting in exponential growth—

Instead of instantly switching from the old version of their product by launching the new version, they keep the old version running for almost all users and “launch” the new version to a small percentage of users. Then, they compare key metrics across the new and old versions. Before ramping up the percentage of users who see the new version, the product team wants to make sure the metrics targeted for improvement are performing better and that other key metrics aren't materially worse. This process, called throttling, is a great way to apply Lean principles to reduce risk after you've launched your product.

The Lean Product Process guides you through the formulation and testing of your hypotheses with these six steps: Determine your target customers Identify underserved customer needs Define your value proposition Specify your minimum viable product (MVP) feature set Create your MVP prototype Test your MVP with customers

process starts in the problem space and progresses to the solution space.

1. Have a point of view but stay open-minded.

2. Articulate your hypotheses.

3. Prioritize ruthlessly.

4. Keep your scope small but focused.

5. Talk to customers.

6. Test before you build.