Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))

by Alistair Croll

Unfortunately, it’s hard to know what people really want. Many times, they don’t know themselves. When they tell you, it’s often what they think you want to hear.[1] What’s worse, as a founder and entrepreneur, you have strong, almost overwhelming preconceptions about how other people think, and these color your decisions in subtle and insidious ways.

You’re forced to confront inconvenient truths.

And you don’t spend your life and your money building something nobody wants.

It depends on what the meaning of the word “is” is.

— William Jefferson Clinton

You need to lie to yourself, but not to the point where you’re jeopardizing your business.

Instincts are experiments. Data is proof.

When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment.

In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.

A good metric is comparative. Being able to compare a metric to other time periods, groups of users, or competitors helps you understand which way things are moving. “Increased conversion from last week” is more meaningful than “2% conversion.”

A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental health of a company.[5] You need some, too. There are several reasons ratios tend to be the best metrics: Ratios are easier to act on. Think about driving a car. Distance travelled is informational. But speed — distance per hour — is something you can act on, because it tells you about your current state, and whether you need to go faster or slower to get to your destination on time. Ratios are inherently comparative. If you compare a daily metric to the same metric over a month, you’ll see whether you’re looking at a sudden spike or a long-term trend. In a car, speed is one metric, but speed right now over average speed this hour shows you a lot about whether you’re accelerating or slowing down. Ratios are also good for comparing factors that are somehow opposed, or for which there’s an inherent tension. In a car, this might be distance covered divided by traffic tickets. The faster you drive, the more distance you cover — but the more tickets you get. This ratio might suggest whether or not you should be breaking the speed limit.

A good metric changes the way you behave. This is by far the most important criterion for a metric: what will you do differently based on changes in the metric?

One other thing you’ll notice about metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something).

Together, they tell you a lot about your cash flow. Similarly, viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate.

Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight.

Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them — the horse hasn’t left the barn yet.

Quantitative data abhors emotion; qualitative data marinates in it.

If you have a piece of data on which you cannot act, it’s a vanity metric.

Another interesting metric to look at is “number of users acquired over a specific time period.” Often, this will help you compare different marketing approaches — for example, a Facebook campaign in the first week, a reddit campaign in the second, a Google AdWords campaign in the third, and a LinkedIn campaign in the fourth. Segmenting experiments by time in this way isn’t precise, but it’s relatively easy.[7] And it’s actionable: if Facebook works better than LinkedIn, you know where to spend your money.

Number of downloads. While it sometimes affects your ranking in app stores, downloads alone don’t lead to real value. Measure activations, account creations, or something else.

They had friends who, once invited, were 50% more likely to become engaged users themselves.

On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period)

good understanding of conversion rate and sales-cycle length.

Finding a correlation between two metrics is a good thing.

Sometimes there’s a huge gulf between what you assume and what users actually do.

Sometimes, however, the differences are subtler. You might assume your product has to be used daily to succeed, only to find out that’s not so.

First, know your customer. There’s no substitute for engaging with customers and users directly.

Second, make early assumptions and set targets for what you think success looks like, but don’t experiment yourself into oblivion.

Testing usually involves comparing two things against each other through segmentation, cohort analysis, or A/B testing.

A segment is simply a group that shares some common characteristic.

Users who join you in the first week will have a different experience from those who join later on.

Each group of users is a cohort

branding,

Data-driven optimization can perform this kind of iterative improvement. What it can’t do, however, is say, “You know what? Four wheels would be way better!” Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.

Yet having a big vision is important: starting a company without one makes you susceptible to outside influences, be they from customers, investors, competition, press, or anything else. Without a big vision, you’ll lack purpose, and over time you’ll find yourself wandering aimlessly.

We sometimes remind early-stage founders that, in many ways, they aren’t building a product. They’re building a tool to learn what product to build.

Dave McClure’s Pirate Metrics

acquisition, activation, retention, revenue, and referral

Churn rates and usage frequency are other important metrics to track. Long-term stickiness often comes from the value users create for themselves as they use the service. It’s hard for people to leave Gmail or Evernote, because, well, that’s where they store all their stuff. Similarly, if a player deletes his account from a massively multiplayer online game (MMO), he loses all his status and in-game items, which he’s worked hard to earn. Stickiness isn’t only about retention, it’s also about frequency, which is why you also need to track metrics like time since last visit. If you have methods of driving return visits such as email notifications or updates, then email open rates and click-through rates matter, too.

When you’re focused on retention, you may be looking at churn, and experimenting with pricing, features, improving customer support, and so on.

Many startups can build a product and solve technical issues, some can attract the right (and occasionally large) audiences, but few make money. Even giants like Twitter and Facebook have struggled with extracting money from their throngs of users.

More stuff means adding products or services, preferably those you know your customers want so you don’t waste time building things they won’t use or buy. For intrapreneurs, this means applying Lean methods to new product development, rather than starting an entirely new company.

More people means adding users, ideally through virality or word of mouth, but also through paid advertising. The best way to add users is when it’s an integral part of product use — such as Dropbox, Skype, or a project management tool that invites outside users outsiders — since this happens automatically and implies an endorsement from the inviting user.

More often means stickiness (so people come back), reduced churn (so they don’t leave), and repeated use (so they use it more frequently). Early on, stickiness tends to be a key knob on which to focus, because until your core early adopters find your ...

More money means upselling and maximizing the price users will pay, or the revenue from ad clicks, or the amount of content they create, or th...