It took an additional test, with a truly outside-the-box execution, to understand what we’d already intuited to be true.
Also important here is the design of the experiment and the result measured. Looking at lagging indicators (retention) versus leading indicators (sign ups) and missing the unexpected finding that followed from the experiment. What mattered most was the first experiment, not because it was right or perfect not because it provided new information that led to the next experiment with the most valuable and unexpected finding.
