Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

by Ron Kohavi

The p-value is the probability of obtaining a result equal to or more extreme than what was observed, assuming that the Null hypothesis is true. The conditioning on the Null hypothesis is critical.