Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))

by Alistair Croll

The Lean Startup movement is galvanizing a generation of entrepreneurs. It helps you identify the riskiest parts of your business plan, then finds ways to reduce those risks in a quick, iterative cycle of learning. Most of its insights boil down to one sentence: Don’t sell what you can make; make what you can sell. And that means figuring out what people want to buy.

Customer development is focused on collecting continuous feedback that will have a material impact on the direction of a product and business, every step of the way.

What Is a Concierge MVP? The Minimum Viable Product is the smallest thing you can build that will create the value you’ve promised to your market. But nowhere in that definition does it say how much of that offering has to be real. If you’re considering building a ride-sharing service, for example, you can try to connect drivers and passengers the old-fashioned way: by hand. This is a concierge approach. It recognizes that sometimes, building a product — even a minimal one — isn’t worth the investment. The risk you’re investigating is, “Will people accept rides from others?” It’s emphatically not, “Can I build software to match drivers and passengers?” A Concierge MVP won’t scale, but it’s fast and easy in the short term. Now that it’s cheap, even free, to launch a startup, the really scarce resource is attention. A concierge approach in which you run things behind the scenes for the first few customers lets you check whether the need is real; it also helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee.

A good metric is comparative.

A good metric is understandable. If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture. A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental health of a company.[5] You need some, too.

“Accounting” metrics like daily sales revenue, when entered into your spreadsheet, need to make your predictions more accurate. These metrics form the basis of Lean Startup’s innovation accounting, showing you how close you are to an ideal model and whether your actual results are converging on your business plan. “Experimental” metrics, like the results of a test, help you to optimize the product, pricing, or market. Changes in these metrics will significantly change your behavior. Agree on what that change will be before you collect the data: if the pink website generates more revenue than the alternative, you’re going pink; if more than half your respondents say they won’t pay for a feature, don’t build it; if your curated MVP doesn’t increase order size by 30%, try something else.

One other thing you’ll notice about metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something). Together, they tell you a lot about your cash flow. Similarly, viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate.

If you want to choose the right metrics, you need to keep five things in mind: Qualitative versus quantitative metrics Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight. Vanity versus actionable metrics Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action. Exploratory versus reporting metrics Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations. Leading versus lagging metrics Leading metrics give you a predictive understanding of the future; lagging metrics

explain the past. Leading metrics are better because you still have time to act on them — the horse hasn’t left the barn yet. Correlated versus causal metrics If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.

Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much. And if you don’t know which metrics would change your organization’s behavior, you aren’t being data-driven. You’re floundering in data quicksand.

The real metric of interest — the actionable one — is “percent of users who are active.” This is a critical metric because it tells us about the level of engagement your users have with your product. When you change something about the product, this metric should change, and if you change it in a good way, it should go up. That means you can experiment, learn, and iterate with it.

Another interesting metric to look at is “number of users acquired over a specific time period.” Often, this will help you compare different marketing approaches — for example, a Facebook campaign in the first week, a reddit campaign in the second, a Google AdWords campaign in the third, and a LinkedIn campaign in the fourth. Segmenting experiments by time in this way isn’t precise, but it’s relatively easy.[7] And it’s actionable: if Facebook works better than LinkedIn, you know where to spend your money.

Eight Vanity Metrics to Watch Out For It’s easy to fall in love with numbers that go up and to the right. Here’s a list of eight notorious vanity metrics you should avoid.

The “unknown unknowns” are most relevant to startups: exploring to discover something new that will help you disrupt a market. As we’ll see in the next case study, it’s how Circle of Friends found out that moms were its best users. These “unknown unknowns” are where the magic lives. They lead down plenty of wrong paths, and hopefully toward some kind of “eureka!” moment when the idea falls into place. This fits what Steve Blank says a startup should spend its time doing: searching for a scalable, repeatable business model.

A leading metric (sometimes called a leading indicator) tries to predict the future. For example, the current number of prospects in your sales funnel gives you a sense of how many new customers you’ll acquire in the future. If the current number of prospects is very small, you’re not likely to add many new customers. You can increase the number of prospects and expect an increase in new customers. On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem — but by the time you’re able to collect the data and identify the problem, it’s too late. The customers who churned out aren’t coming back. That doesn’t mean you can’t act on a lagging metric (i.e., work to improve churn and then measure it again), but it’s akin to closing the barn door after the horses have left. New horses won’t leave, but you’ve already lost a few.

There’s a correlation between ice cream consumption and drowning. Does that mean we should ban ice cream to avert drowning deaths? Or measure ice cream consumption to predict the fortunes of funeral home stock prices? No: ice cream and drowning rates both happen because of summer weather.

Analytics Lessons Learned First, know your customer. There’s no substitute for engaging with customers and users directly. All the numbers in the world can’t explain why something is happening. Pick up the phone right now and call a customer, even one who’s disengaged. Second, make early assumptions and set targets for what you think success looks like, but don’t experiment yourself into oblivion.

Another way to understand cohorts is to line up the data by the users’ experience — in the case of Table 2-3, we’ve done this by the number of months they’ve used the system. This shows another critical metric: how quickly revenue declines after the first month.

Cohort experiments that compare groups like the one in Table 2-2 are called longitudinal studies, since the data is collected along the natural lifespan of a customer group. By contrast, studies in which different groups of test subjects are given different experiences at the same time are called cross-sectional studies. Showing half of the visitors a blue link and half of them a green link in order to see which group is more likely to click that link is a cross-sectional study. When we’re comparing one attribute of a subject’s experience, such as link color, and assuming everything else is equal, we’re doing A/B testing.

Rather than running a series of separate tests one after the other — which will delay your learning cycle — you can analyze them all at once using a technique called multivariate analysis. This relies on statistical analysis of the results to see which of many factors correlates strongly with an improvement in a key metric.

In mathematics, a local maximum is the largest value of a function within a given neighborhood.[15] That doesn’t mean it’s the largest possible value, just the largest one in a particular range. As an analogy, consider a lake on a mountainside. The water isn’t at its lowest possible level — that would be sea level — but it’s at the lowest possible level in the area surrounding the lake. Optimization is all about finding the lowest or highest values of a particular function. A machine can find the optimal settings for something, but only within the constraints and problem space of which it’s aware, in much the same way that the water in a mountainside lake can’t find the lowest possible value, just the lowest value within the constraints provided.

Figure 5-1. Even pirates need metrics, says Dave McClure These five elements don’t necessarily follow a strict order — users may refer others before they spend money, for example, or may return several times before signing up — but the list is a good framework for thinking about how a business needs to grow

The fundamental KPI for stickiness is customer retention. Churn rates and usage frequency are other important metrics to track. Long-term stickiness often comes from the value users create for themselves as they use the service. It’s hard for people to leave Gmail or Evernote, because, well, that’s where they store all their stuff. Similarly, if a player deletes his account from a massively multiplayer online game (MMO), he loses all his status and in-game items, which he’s worked hard to earn.

The key metric for this engine is the viral coefficient — the number of new users that each user brings on. Because this is compounding (the users they bring, in turn, bring their own users), the metric measures how many users are brought in with each viral cycle. Growth comes from a viral coefficient of greater than one, but you also have to factor in churn and loss. The bigger the coefficient, the faster you grow. Measuring viral coefficient isn’t enough. You also need to measure the actions that make up the cycle. For example, when you join most social networks, you’re asked to connect to your email account to find contacts, then you’re given the option to invite them. They receive emails, which they might act upon. Those distinct stages all contribute to virality, so measuring actions is how you tweak the viral engine — by changing the message, simplifying the signup process, and so on.

The two knobs on this machine are customer lifetime value (CLV) and customer acquisition cost (CAC). Making more money from customers than you spend acquiring them is good, but the equation for success isn’t that simple. You still need to worry about cash flow and growth rate, which are driven by how long it takes a customer to pay off. One way to measure this is time to customer breakeven — that is, how much time it will take to recoup the acquisition cost of a customer.

if 40% of people (or more) say they’d be very disappointed to lose the service, you’ve found a fit, and now it’s time to scale.

As noted in Chapter 5, Eric Ries talks about three engines that drive company growth: the sticky engine, the viral engine, and the paid engine. But he cautions that while all successful companies will ultimately use all three engines, it’s better to focus on one engine at a time. For example, you might make your product sticky for its core users, then use that to grow virally, and then use the user base to grow revenue. That’s focus. In the world of analytics and data, this means picking a single metric that’s incredibly important for the step you’re currently working through in your startup. We call this the One Metric That Matters (OMTM).

Knowing which metric to focus on isn’t enough. You need to draw a line in the sand as well. Let’s say that you’ve decided “New Customers Per Week” is the right metric to focus on because you’re testing out new ways of acquiring customers. That’s fair, but it doesn’t answer the real question: How many new customers per week do you need? Or more specifically: How many new customers per week (per acquisition channel) do you think defines a level of success that enables you to double down on user acquisition and move to the next step in the process? You need to pick a number, set it as the target, and have enough confidence that if you hit it, you consider it success. And if you don’t hit the target, you need to go back to the drawing board and try again.

Sergio Zyman, Coca-Cola’s CMO, said marketing is about selling more stuff to more people more often for more money more efficiently.[22

Later in the book we’re going to outline six sample businesses. But before we do that, we want to talk about how we came up with them. Think of one of the flipbooks you had as a kid — the kind where you could combine different body parts on each page to make different characters. You can build business models this way, but instead of heads, torsos, and feet, you have several aspects of a business: the acquisition channel, selling tactic, revenue source, product type, and delivery model.

As if that weren’t confusing enough, you can employ several at once: Amazon is a transactional, physical-delivery, SEM (search engine marketing), simple-purchase retailer, but it’s also running sub-businesses such as user-generated content in the form of product reviews. So unlike those relatively simple children’s books, your business can quite easily be a many-headed monster.

And you probably can’t measure stickiness for just a few people if the service requires a critical mass of users to be useful. This means you have to know where the risk is, but focus, in the right order, on just enough optimization to get the business to a place where that risk can be quantified and understood.

Finding these engagement patterns means analyzing data in two ways: To find ways you might improve things, segment users who do what you want from those who don’t, and identify ways in which they’re different. Do the engaged users all live in the same city? Do all users who eventually become loyal contributors learn about you from one social network? Are the users who successfully invite friends all under 30 years old? If you find a concentration of desirable behavior in one segment, you can then target it. To decide whether a change worked, test the change on a subset of your users and compare that subset’s results to others. If you put in a new reporting feature, reveal it to half of your users, and see if more of them stick around for several months. If you can’t test features in this way without fallout — the customers who didn’t get the new feature might get angry — then at the very least, compare the cohort of users who joined after the feature was added to those who came before.

As Shopify data scientist Steven H. Noble[32] explains in a detailed blog post,[33] the simple formula for churn is:

Here’s the churn calculation for February: If 2.5% of customers churn every month, it means that the average customer stays around for 40 months (100/2.5). This is how you can start to calculate the lifetime value of a customer (40 months × average monthly revenue per user).

This spreads out the total number of customers across the period, which is better, but it still presents a problem if things are growing quickly. If you have 100 customers at the start of the month, and 10,000 at the end, this formula assumes you have 5,050 customers in the middle of the month — which you don’t, if you’re on a hockey stick. Most of your new customers come in the later part of the month, so an average won’t work. What’s more, most of your churns will, too.

Ultimately, the math gets really complex. There are two ways to simplify it. The first is to measure churn by cohort, so you’re comparing new to churned users based on when they first became users. The second way is really, really simple, which is why we like it: measure churn each day. The shorter the time period you measure, the less that changes during that specific period will distort things.

In a SaaS model, most of the complexity comes from two things: the promotional approach you choose, and pricing tiers.

If you’re trying to estimate the size of a market, it’s a good idea to do both a top-down and a bottom-up analysis, and compare the results. This helps to check your math. A top-down analysis starts with a big number and breaks it into smaller parts. A bottom-up one does the reverse. Consider, for example, a restaurant in New York City. A top-down model would look at the total money people spend dining out in the US, then the percentage of that in New York, then the number of restaurants in the city, and finally calculate the revenues for a single restaurant. A bottom-up model would look at the number of tables in a restaurant, the percent that are occupied, and the average price per party. Then it would multiply this by days of the year (adjusting for seasonality). This is an oversimplification — there are plenty of other factors to consider such as location, type of restaurant, and so on. But the end result should provide two estimates of annual revenue. If they’re wildly different, something is wrong with your business model.

Don’t just ask questions. Know how the answers to the questions will change your behavior. In other words, draw a line in the sand before you run the survey. Your earlier problem interviews showed you an opportunity; now, you’re checking to see whether that opportunity exists in the market as a whole. For each quantifiable question, decide what would be a “good” score. Write it down somewhere so you’ll remember.

Virality is simply users sharing your product or service with others. There are three kinds of virality: Inherent virality is built into the product, and happens as a function of use. Artificial virality is forced, and often built into a reward system. Word-of-mouth virality is the conversations generated by satisfied users, independent of your product or service. All three matter, but should be treated as distinct forms of growth and analyzed in terms of the kind of traffic they bring in.

Growth hacking is an increasingly popular term for data-driven guerilla marketing. It relies on a deep understanding of how parts of the business are related, and how tweaks to one aspect of a customer’s experience impact others. It involves: Finding a metric you can measure early in a user’s lifecycle (e.g., number of friends a user invites) through experimentation, or, if you have the data, an analysis of what good users have in common Understanding how that metric is correlated to a critical business goal (e.g., long-term engagement) Building predictions of that goal (e.g., how many engaged users you’ll have in 90 days) based on where the early metric is today Modifying the user experience today in order to improve the business goal tomorrow (e.g., suggesting people a user might know), assuming today’s metric is causing a change

What Makes a Good Leading Indicator? Good leading indicators have a few common characteristics: Leading indicators tend to relate to social engagement (links to friends), content creation (posts, shares, likes), or return frequency (days since last visit, time on site, pages per visit). The leading indicator should be clearly tied to a part of the business model (such as users, daily traffic, viral spread, or revenue). After all, it’s the business model that you’re trying to improve. You’re not just trying to increase number of friends per user — you’re trying to increase the number of loyal users. The indicator should come early in the user’s lifecycle or conversion funnel.

As you might expect from an analytics firm, the Parse.ly team collects and analyzes a lot of data. In addition to using Dash themselves, they rely on Woopra for engagement and to arm their sales team, Graphite for tracking time-series data, and Pingdom for uptime and availability.

For example, a freemium model requires a huge base of prospective customers. Lincoln Murphy does a great job of laying out the math on addressable market size in a presentation entitled The Reality of Freemium in SaaS.[72] One of his big conclusions: without a huge potential market and a number of other factors, freemium just doesn’t work. Understanding the mechanics of various markets and business models helps you triangulate the combinations that work best.

When looking at a market at this stage, you need to narrow it down and go niche. Using “size of company” as your metric for market definition isn’t good enough. We see this all the time, but SMBs (small and medium businesses) are not a market; the category’s just too broad. Look for important similarities between companies inside of a broadly defined market. Industry is a good place to start. But also consider geography, how they purchase products, what they’ve recently purchased, budgets, industry growth, seasonality, legislative constraints, and decision makers. All of these factors help define a true market you can go after quickly.

Hibernation Breakeven A particularly conservative breakeven metric is hibernation. If you reduced the company to its minimum — keeping the lights on, servicing existing customers, but doing little else — could you survive? This is often referred to as “ramen profitability.” There’s no new marketing spend. Your only growth would come from word of mouth or virality, and customers wouldn’t get new features. But it’s a breakeven point at which you’re “master of your own destiny” because you can survive indefinitely.

Harvard professor Michael Porter describes a variety of generic strategies by which companies compete.[73] Firms can focus on a niche market (a segmentation strategy), they can focus on being efficient (a cost strategy), or they can try to be unique (a differentiation strategy).

In the Scale stage, you want to compare higher-order metrics like Backupify’s OMTM — customer acquisition payback — across channels, regions, and marketing campaigns. For example: is a customer you acquire through channels less valuable than one you acquire yourself? Does it take longer to pay back direct sales or telemarketing? Are international revenues hampered by taxes?

A Summary of the Scale Stage When you’re scaling, you know your product and your market. Your metrics are now focused on the health of your ecosystem, and your ability to enter new markets. You’ll look at compensation, API traffic, channel relationships, and competitors at this stage — whereas before, these were distractions. You need to understand if you’re focused on efficiency or differentiation. Trying to do both as a way of scaling is difficult. If you’re efficiency-focused, you’re trying to reduce costs; if you’re differentiation-focused, you’re increasing margins. As you grow, you’ll need to have more than one metric at a time. Set up a hierarchy of metrics that keeps the strategy, the tactics, and the implementation aligned with a consistent set of goals. We call this the three threes.