Trustworthy Online Controlled Experiments Quotes

“For Bing, over 50% of US traffic is from bots, and that number is higher than 90% in China and Russia.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“While “statistical significance” measures how likely the result you observe or more extreme could have happened by chance assuming the null, not all statistically significant results are practically meaningful.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“most who have run controlled experiments in customer-facing websites and applications have experienced this humbling reality: we are poor at assessing the value of ideas.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Stable Unit Treatment Value Assumption (SUTVA)”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“duality between p-values and confidence intervals. For the Null hypothesis of no-difference commonly used in controlled experiments, a 95% confidence interval of the Treatment effect that does not cross zero implies that the p-value is < 0.05.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“The p-value is the probability of obtaining a result equal to or more extreme than what was observed, assuming that the Null hypothesis is true. The conditioning on the Null hypothesis is critical.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“statistical and practical significance”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Day-of-week effect:”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Statistical power is the probability of detecting a meaningful difference between the variants when there really is one (statistically, reject the null when there is a difference).”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“95% confidence interval [Δ − 1.96 σ, Δ + 1.96 σ] to assess statistical significance. If zero lies outside of the confidence interval, we declare significance”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“whether the confidence interval overlaps with zero.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“use a p-value less than 0.05, meaning that if there is truly no effect, we can correctly infer there is no effect 95 out of 100 times.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“p-value for the difference,”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Null hypothesis”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“fake door or painted door approach”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“multi-armed bandit”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Additional Reading”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“EVI: Expected Value of Information”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“However beautiful the strategy, you should occasionally look at the results.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Defining guardrail metrics for experiments is important for identifying what the organization is not willing to change,”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair – Douglas Adams”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“The signature feature for Amazon’s recommendation was “People who bought item X bought item Y,” but this was generalized to “People who viewed item X bought item Y” and “People who viewed item X viewed item Y.” A proposal was made to use the same algorithm for “People who searched for X bought item Y.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

“Dan McKinley at Etsy (McKinley 2013) wrote “nearly everything fails” and for features, he wrote “it’s been humbling to realize how rare it is for them to succeed on the first attempt. I strongly suspect that this experience is universal, but it is not universally recognized or acknowledged.” Finally, Colin McFarland wrote in the book Experiment! (McFarland 2012, 20) “No matter how much you think it’s a no-brainer, how much research you’ve done, or how many competitors are doing it, sometimes, more often than you might think, experiment ideas simply fail.”
― Ron Kohavi, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing