Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))

by Alistair Croll

“Everyone has data, the key is figuring out what pieces will improve your learning and decision making.

Everyone knows they need metrics, but finding ones that are specific, measurable, actionable, relevant, and timely is a huge challenge.

To be honest, in far too many of my companies, the accounting was incredibly simple anyway: revenue, margins, free cash flows — they were all zero.

Most of its insights boil down to one sentence: Don’t sell what you can make; make what you can sell. And that means figuring out what people want to buy.

Unfortunately, it’s hard to know what people really want. Many times, they don’t know themselves. When they tell you, it’s often what they think you want to hear.[

Analytics can help. Measuring something makes you accountable. You’re forced to confront inconvenient truths. And you don’t spend your life and your money building something nobody wants.

Lean Startup helps you structure your progress and identify the riskiest parts of your business, then learn about them quickly so you can adapt. Lean Analytics is used to measure that progress, helping you to ask the most important questions and get clear answers quickly.

A startup is an organization formed to search for a scalable and repeatable business model.

Lean Analytics focuses on the measure stage.

Management guru and author Peter Drucker famously observed, “If you can’t measure it, you can’t manage

This is a concierge approach. It recognizes that sometimes, building a product — even a minimal one — isn’t worth the investment. The risk you’re investigating is, “Will people accept rides from others?” It’s emphatically not, “Can I build software to match drivers and passengers?” A Concierge MVP won’t scale, but it’s fast and easy in the short term. Now that it’s cheap, even free, to launch a startup, the really scarce resource is attention. A concierge approach in which you run things behind the scenes for the first few customers lets you check whether the need is real; it also helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee. Initial

The key metric Airbnb tracked was shoots per month, because it had already proven with its Concierge MVP that more professional photographs meant more bookings.

When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment. Define what success looks like beforehand, and know what you’re going to do if your hunch is right.

Usually, those metrics matter because they relate to your business model — where money comes from, how much things cost, how many customers you have, and the effectiveness of your customer acquisition strategies.

In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.

A good metric is comparative. Being able to compare a metric to other time periods, groups of users, or competitors helps you understand which way things are moving. “Increased conversion from last week” is more meaningful than “2% conversion.”

good metric is understandable. If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture.

A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental h...

Ratios are easier to act on.

Ratios are inherently comparative.

Ratios are also good for comparing factors that are somehow opposed, or for which there’s an inherent tension.

A good metric changes the way you behave. This is by far the most important criterion for a metric:

One other thing you’ll notice about metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something).

Similarly, viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate.

Qualitative versus quantitative metrics

Vanity versus actionable metrics

Exploratory versus reporting metrics

Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations.

Leading versus laggin...

Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act ...

Correlated versus caus...

you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.

Initially, you’re looking for qualitative data. You’re not measuring results numerically. Instead, you’re speaking to people — specifically, to people you think are potential customers in the right target market. You’re exploring. You’re getting out of the building.

Unprepared interviews yield misleading or meaningless results.

If you have a piece of data on which you cannot act, it’s a vanity metric.

The real metric of interest — the actionable one — is “percent of users who are active.”

Number of hits. This is a metric from the early, foolish days of the Web. If you have a site with many objects on it, this will be a big number. Count people instead.

Number of downloads. While it sometimes affects your ranking in app stores, downloads alone don’t lead to real value. Measure activations, account creations, or something else.

Analytics has a role to play in all four of Rumsfeld’s quadrants: It can check our facts and assumptions — such as open rates or conversion rates — to be sure we’re not kidding ourselves, and check that our business plans are accurate. It can test our intuitions, turning hypotheses into evidence. It can provide the data for our spreadsheets, waterfall charts, and board meetings. It can help us find the nugget of opportunity on which to build a business.

In the early stages of your startup, the unknown unknowns matter most, because they can become your secret weapons.

Circle of Friends was a social graph application in the right place at the right time — with the wrong market.

There’s a “critical mass” of engagement necessary for any community to take off. Mild success may not give you escape velocity. As a result, it’s better to have fervent engagement with a smaller, more easily addressable target market. Virality requires focus.

On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem

In the early days of your startup, you won’t have enough data to know how a current metric relates to one down the road, so measure lagging metrics at first. Lagging metrics are still useful and can provide a solid baseline of performance. For leading indicators to work, you need to be able to do cohort analysis and compare groups of customers over periods of time.

Churn is important (and we discuss it at length throughout the book), but looking at it myopically won’t let you iterate and adapt at the speed you need.

Ultimately, you need to decide whether the thing you’re tracking helps you make better decisions sooner. As we’ve said, a real metric has to be actionable. Lagging and leading metrics can both be actionable, but leading indicators show you what will happen, reducing your cycle time and making you leaner.

Cohort experiments that compare groups like the one in Table 2-2 are called longitudinal studies, since the data is collected along the natural lifespan of a customer group.

Much of Lean Analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business,

Bud Caddell has three clear criteria for deciding what to spend your time on: something that you’re good at, that you want to do, and that you can make money doing.