Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

by Ron Kohavi

The platform needs some interface and/or tools to easily manage many experiments and their multiple iterations. Functionalities should include: Writing, editing, and saving draft experiment specifications. Comparing the draft iteration of an experiment with the current (running) iteration. Viewing the history or timeline of an experiment (even if it is no longer running). Automatically assigning generated experiment IDs, variants, and iterations and adding them to the experiment specification. These IDs are needed in the experiment instrumentation (discussed later in this chapter). Validating that there are no obvious errors in the specifications, such as configuration conflicts, invalid targeting audience, and so on. Checking the status of an experiment as well as starting/stopping an experiment. To guard against human error, usually only experiment owners or individuals with special permission can start an experiment. However, due to the asymmetry of harming users, anyone can stop an experiment, although alerts are generated to ensure that experiment owners are informed. Also, since the experiment is impacting real users, additional tools or workflows are needed to check the experiment variants before they go live. Options range from test code that must be run before deployment or a permission control system where experiments must get approval from trusted experts.