A group of users could be selected, half with their check box default checked (group A) and the other half with the new mechanism (group B). Whichever design has the better results would be used for all users after the test. Flag flips can be used both to control the test and to enable the winning design when the test is finished.
Not sure a/b testing and feature flag toggling should be through same mechanism. Seems too much risk of fouling up your a/b tests. But might be great for the /code/ not to care: it just consults a single call to determine which treatment a give request or customer should receive.
