The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses

by Eric Ries

Instead of looking at cumulative totals or gross numbers such as total revenue and total number of customers, one looks at the performance of each group of customers that comes into contact with the...

Each conversion rate shows the percentage of customers who registered in that month who subsequently went on to take the indicated action.

Thus, among all the customers who joined IMVU in February 2005, about 60 percent of them logged in to our product at least one time.

this funnel analysis as the traditional sales funnel that is used to manage prospects on thei...

If you look closely, you’ll see that the graph shows some clear trends. Some product improvements are helping—a little. The percentage of new customers who go on to use the product at least five times has grown from less than 5 percent to almost 20 percent. Yet despite this fourfold increase, the percentage of new customers who pay money for IMVU is stuck at around 1 percent.

After months and months of work, thousands of individual improvements, focus groups, design sessions, and usability tests, the percentage of new customers who subsequently pay money is exactly the same as it was at the onset even though many more customers are getting a chance to try the product.

once I had data in hand, my interactions with customers changed.

For example, we kept making it easier and easier for customers to use IMVU with their existing friends. Unfortunately, customers didn’t want to engage in that behavior. Making it easier to use was totally beside the point.

this eventually led to a critically important pivot: away from an IM add-on used with existing friends and toward a stand-alone network one can use to make new friends.

OPTIMIZATION VERSUS LEARNING

If you are building the wrong thing, optimizing the product or its marketing will not yield significant results. A startup has to measure progress against a high bar: evidence that a sustainable business can be built around its products or services.

However, the fault was not in the engineers; it was in the process the whole company was using to make decisions. They had customers but did not know them very well.

unasked and unanswered were other lurking questions: Did the company have a working engine of growth? Was this early success related to the daily work of the product development team? In most cases, the answer was no; success was driven by decisions the team had made in the past. None of its current initiatives were having any impact. But this was obscured because the company’s gross metrics were all “up and to the right.” As we’ll see in a moment, this is a common danger. Companies of any size that have a working engine of growth can come to rely on the wrong kind of metrics to guide their actions.

traditional numbers used to judge startups “vanity metrics,” and innovation accounting requires us to avoid the temptation to use them.

VANITY METRICS: A WORD OF CAUTION

vanity metrics: they give the rosiest possible picture. You’ll see a traditional hockey stick graph

But think back to the same data presented in a cohort style. IMVU is adding new customers, but it is not improving the yield on each new group.

From the traditional graph alone, you cannot tell whether IMVU is on pace to build a sustainable business;

ACTIONABLE METRICS VERSUS VANITY METRICS

Following standard agile practice, Grockit’s work proceeded in a series of sprints, or one-month iteration cycles. For each sprint, Farb would prioritize the work to be done that month by writing a series of user stories, a technique taken from agile development. Instead of writing a specification for a new feature that described it in technical terms, Farb would write a story that described the feature from the point of view of the customer. That story helped keep the engineers focused on the customer’s perspective throughout the development process.

This system is called agile development for a good reason: teams that employ it are able to change direction quickly, stay light on their feet, and be highly responsive to changes in the business requirements

There was always a certain amount of data that showed improvement: perhaps the total number of customers was increasing, the total number of questions answered by students was going up, or the number of returning customers was increasing. However, I sensed that Farb and his team were left with lingering doubts

Was the increase in their numbers actually caused by their development efforts? Or could it be due to other factors, such as mentions of Grockit in the press?

the Grockit team was struggling with the age-old startup problems: How do we know which features to prioritize? How can we get more customers to sign up and pay?

Unlike many visionaries, who cling to their original vision no matter what, Farb was willing to put his vision to the test.

What Farb and his team didn’t realize was that Grockit’s progress was being measured by vanity metrics: the total number of customers and the total number of questions answered. That was what was causing his team to spin its wheels; those metrics gave the team the sensation of forward motion even though the company was making little progress.

because Grockit was using the wrong kinds of metrics, the startup was not genuinely improving.

Those metrics would go up and down seemingly on their own. He couldn’t draw clear cause-and-effect inferences.

Compared to a lot of startups, the Grockit team had a huge advantage: they were tremendously disciplined. A disciplined team may apply the wrong methodology but can shift gears quickly once it discovers its error.

Cohorts and Split-tests

Grockit switched to cohort-based metrics, and instead of looking for cause-and-effect relationships after the fact, Grockit would launch each new feature as a true split-test experiment. A split-test experiment is one in which different versions of a product are offered to customers at the same time. By observing the changes in behavior between the two groups, one can make inferences about the impact of the different variations.

(This technique is sometimes called A/B testing after the practice of assigning letter names to each variation.)

Split testing often uncovers surprising things. For example, many features that make the product better in the eyes of engineers and designers have no impact on customer behavior.

Hypothesis Testing at Grockit

lazy registration is considered one of the design best practices for online services. In this system, customers do not have to register for the service up front. Instead, they immediately begin using the service and are asked to register only after they have had a chance to experience the service’s benefit.

As a result of this hypothesis, Grockit’s design required that it manage three classes of users: unregistered guests, registered (trial) guests, and customers who had paid for the premium version of the product. This design required significant extra work to build and maintain:

I encouraged the team to try a simple split-test. They took one cohort of customers and required that they register immediately, based on nothing more than Grockit’s marketing materials. To their surprise, this cohort’s behavior was exactly the same as that of the lazy registration group: they had the same rate of registration, activation, and subsequent retention. In other words, the extra effort of lazy registration was a complete waste even though it was considered an industry best practice.

Think about the cohort of customers who were required to register for the product before entering a study session with other students. They had very little information about the product,

By contrast, the lazy registration group had a tremendous amount of information about the product because they had used it. Yet despite this information disparity, customer behavior was exactly the same. This suggested that improving Grockit’s positioning and marketing might have a more significant impact on attracting new customers than would adding new features.

THE VALUE OF THE THREE A’S

Actionable

For a report to be considered actionable, it must demonstrate clear cause and effect.

By contrast, vanity metrics fail this criterion. Take the number of hits to a company website. Let’s say we have 40,000 hits this month—a new record. What do we need to do to get more hits? Well, that depends. Where are the new hits coming from? Is it from 40,000 new customers or from one guy with an extremely active web browser? Are the hits the result of a new marketing campaign or PR push? What is a hit, anyway? Does each page in the browser count as one hit, or do all the embedded images and multimedia content count as well?