Lean Analytics: Use Data to Build a Better Startup Faster

by Alistair Croll

“Avoiding false metrics.”[

customer satisfaction or pipeline flow is vital to a successful business. But if you want to change behavior, your metric must be tied to the behavioral change you want.

metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something).

viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate.

Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight.

Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action.

Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations.

Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them — the horse hasn’t left the barn yet.

If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.

Initially, you’re looking for qualitative data. You’re not measuring results numerically. Instead, you’re speaking to people — specifically, to people you think are potential customers in the right target market. You’re exploring. You’re getting out of the building.

Collecting good qualitative data takes preparation.

If you have a piece of data on which you cannot act, it’s a vanity metric.

Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much.

“Total active users” is a bit better — assuming that you’ve done a decent job of defining an active user — but it’s still a vanity metric. It will gradually increase over time, too, unless you do something horribly wrong.

The real metric of interest — the actionable one — is “percent of users who are active.” This is a critical metric because it tells us about the level of engagement your users have with your product. When you change something about the product, this metric should change, and if you change it in a good way, it should go up. That means you can experiment, learn, and iterate with it.

Another interesting metric to look at is “number of users acquired over a...

Eight Vanity Metrics to Watch Out For

On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem — but by the time you’re able to collect the data and identify the problem, it’s too late. The customers who churned out aren’t coming back.

For leading indicators to work, you need to be able to do cohort analysis and compare groups of customers over periods of time.

Finding a correlation between two metrics is a good thing. Correlations can help you predict what will happen. But finding the cause of something means you can change it.

correlation is good. Causality is great. Sometimes, you may have to settle for the former — but you should always be trying to discover the latter.

First, know your customer. There’s no substitute for engaging with customers and users directly.

A segment is simply a group that shares some common characteristic. It might be users who run Firefox, or restaurant patrons who make reservations rather than walking in, or passengers who buy first-class tickets, or parents who drive minivans.

Cohort Analysis A second kind of analysis, which compares similar groups over time, is cohort analysis.

Each group of users is a cohort — participants in an experiment across their lifecycle.

Rather than running a series of separate tests one after the other — which will delay your learning cycle — you can analyze them all at once using a technique called multivariate analysis. This relies on statistical analysis of the results to see which of many factors correlates strongly with an improvement in a key metric.

The Lean Analytics Cycle

Evaluating the Metrics You Track

Alex Osterwalder’s Business Model Canvas.[12] As

Never start a company on a level playing field — that’s where everyone else is standing.

Create a Lean Canvas

10 common pitfalls that entrepreneurs should avoid as they dig into the data their startups capture.

Pirate Metrics

Value comes not only from a transaction (revenue) but also from their role as marketers (referral) and content creators (retention).

Stickiness isn’t only about retention, it’s also about frequency, which is why you also need to track metrics like time since last visit. If you have methods of driving return visits such as email notifications or updates, then email open rates and click-through rates matter, too.

The most important question in the survey is “How would you feel if you could no longer use this product or service?” In Sean’s experience, if 40% of people (or more) say they’d be very disappointed to lose the service, you’ve found a fit, and now it’s time to scale.

The OMTM is the one number you’re completely focused on above everything else for your current stage.

While it’s great to track many metrics, it’s also a sure way to lose focus. Picking a minimal set of KPIs on which your business assumptions rely is the best way to get the entire organization moving in the same direction.

You need to pick a number, set it as the target, and have enough confidence that if you hit it, you consider it success. And if you don’t hit the target, you need to go back to the drawing board and try again.

Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts, which often happens at an inflection point for your business:

To decide which metrics you should track, you need to be able to describe your business model in no more complex a manner than a lemonade stand’s. You need to step back, ignore all the details, and just think about the really big components.

selling more stuff to more people more often for more money more efficiently.[22]

The key here is analytics. You need to segment real, valuable users from drive-by, curious, or detrimental ones. Then you need to make changes that maximize the real users and weed out the bad ones. That may be as blunt as demanding a credit card up front — a sure way to reject curious users who don’t have any intention of committing or paying. Or it may be a subtler approach, such as not trying to reactivate disengaged users once they’ve been gone for a while.

If you’re a SaaS provider with low incremental costs for additional users, freemium may work, as long as you clearly separate engaged from casual users.

Not all customers are good. Don’t fall victim to customer counting. Instead, optimize for good customers and segment your activities based on the kinds of customer those activities attract.

This is usually the right place to focus for SaaS companies, because they seldom get a second chance to make a first impression, and need users to keep coming back. In other words, they care about stickiness.

You know it’s time to scale when your paid engine is humming along nicely, which happens when the CAC is a small fraction of the CLV — a sure sign you’re getting a good return on your investment.

The ultimate metric for engagement is daily use. How many of your customers use your product on a daily basis?

Finding these engagement patterns means analyzing data in two ways: To find ways you might improve things, segment users who do what you want from those who don’t, and identify ways in which they’re different. Do the engaged users all live in the same city? Do all users who eventually become loyal contributors learn about you from one social network? Are the users who successfully invite friends all under 30 years old? If you find a concentration of desirable behavior in one segment, you can then target it. To decide whether a change worked, test the change on a subset of your users and compare that subset’s results to others. If you put in a new reporting feature, reveal it to half of your users, and see if more of them stick around for several months. If you can’t test features in this way without fallout — the customers who didn’t get the new feature might get angry — then at the very least, compare the cohort of users who joined after the feature was added to those who came before.

Churn is the percentage of people who abandon your service over time. This can be measured weekly, monthly, quarterly, etc., but you should pick a timespan for all your metrics and stick to it in order to make comparing them easier.