Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))

by Alistair Croll

If you want to choose the right metrics, you need to keep five things in mind: Qualitative versus quantitative metrics Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight. Vanity versus actionable metrics Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action. Exploratory versus reporting metrics Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations. Leading versus lagging metrics Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them — the horse hasn’t left the barn yet. Correlated versus causal metrics If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.

Consider, for example, “total signups.” This is a vanity metric. The number can only increase over time (a classic “up and to the right” graph). It tells us nothing about what those users are doing or whether they’re valuable to us. They may have signed up for the application and vanished forever. “Total active users” is a bit better — assuming that you’ve done a decent job of defining an active user — but it’s still a vanity metric. It will gradually increase over time, too, unless you do something horribly wrong. The real metric of interest — the actionable one — is “percent of users who are active.” This is a critical metric because it tells us about the level of engagement your users have with your product. When you change something about the product, this metric should change, and if you change it in a good way, it should go up. That means you can experiment, learn, and iterate with it.

Jason Billingsley recommends testing an individualized send schedule equal to the signup time of the unique user. So, if a user signs up at 9 a.m., schedule to send her updates at 9 a.m. “Most email tools aren’t set up for such a tactic, but it’s a highly valuable test that could yield significant results,” he says.