You need a hypothesis, and that hypothesis should be part of a bigger product vision. So you can A/B test where the “Buy” button should go on a Web page, whether it should be blue or orange, but you shouldn’t be testing whether or not a customer should buy online. If you’re testing the core of your product, if the basic functionality can flex and change depending on the whims of an A/B test, then there is no core. There’s a hole where your product vision should be and you’re just shoveling data into the void.
