Lean Analytics: Use Data to Build a Better Startup Faster

by Alistair Croll

Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof.

A good metric is comparative. Being able to compare a metric to other time periods, groups of users, or competitors helps you understand which way things are moving. “Increased conversion from last week” is more meaningful than “2% conversion.”

A good metric is understandable. If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture.

Ratios are easier to act on. Think about driving a car. Distance travelled is informational. But speed — distance per hour — is something you can act on, because it tells you about your current state, and whether you need to go faster or slower to get to your destination on time.

Ratios are inherently comparative. If you compare a daily metric to the same metric over a month, you’ll see whether you’re looking at a sudden spike or a long-term trend. In a car, speed is one metric, but speed right now over average speed this hour shows you a lot about whether you’re accelerating or slowing down.

If you have a piece of data on which you cannot act, it’s a vanity metric.

Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much.

Consider, for example, “total signups.” This is a vanity metric. The number can only increase over time (a classic “up and to the right” graph). It tells us nothing about what those users are doing or whether they’re valuable to us. They may have signed up for the application and vanished forever. “Total active users” is a bit better — assuming that you’ve done a decent job of defining an active user — but it’s still a vanity metric. It will gradually increase over time, too, unless you do something horribly wrong. The real metric of interest — the actionable one — is “percent of users who are active.” This is a critical metric because it tells us about the level of engagement your users have with your product.