Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

by Ron Kohavi

Coming up with a single weighted combination may be hard initially, but you can start with classifying your decisions into four groups: 1. If all key metrics are flat (not statistically significant) or positive (statistically significant), with at least one metrics positive, then ship the change. 2. If all key metrics are flat or negative, with at least one metric negative, then don’t ship the change. 3. If all key metrics are flat, then don’t ship the change and consider either increasing the experiment power, failing fast, or pivoting. 4. If some key metrics are positive and some key metrics are negative, then decide based on the tradeoffs. When you have accumulated enough of these decisions, you may be able to assign weights.