Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success

by Sean Ellis

Let’s say experiment A is testing a small change, such as the color of the sign-up button. As results start coming in, it becomes clear that the increase in the number of new visitors signing up is very small—garnering just 5 percent more sign-ups than the original button color. Besides the obvious assumption that changing the color of the sign-up button may not be the key factor holding back new users from signing up, it’s also an indication that you’ll have to let the experiment run quite a long time in order to have enough data to make a solid conclusion. As you can see from the chart above, to reach statistically significant results for this test, you’d need a whopping 72,300 visitors per variant—or, in other words, you’d have to wait 72 days to get conclusive results. As Johns put it in an interview with First Round Review, “That’s a lifetime when you’re a start-up!” In a case like this what a start-up really ought to do is abandon the experiment quickly and move on to a next, potentially higher-impact, one.

I do not bet on small changes or something obviously will fail.