Statistical Inference as Severe Testing Quotes

“The influence of the biased selection is not on the believability of H but rather on the capability of the test to have unearthed errors. The error probing capability of the testing procedure is being diminished. If you engage in cherry picking, you are not “sincerely trying,” as Popper puts it, to find flaws with claims, but instead you are finding evidence in favor of a well-fitting hypothesis that you deliberately construct – barred only if your intuitions say it’s unbelievable. The job that was supposed to be accomplished by an account of statistics now has to be performed by you. Yet you are the one most likely to follow your preconceived opinions, biases, and pet theories.”
― Deborah G Mayo, Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars

“If you report results selectively, it becomes easy to prejudge hypotheses: yes, the data may accord amazingly well with a hypothesis H, but such a method is practically guaranteed to issue so good a fit even if H is false and not warranted by the evidence. If it is predetermined that a way will be found to either obtain or interpret data as evidence for H, then data are not being taken seriously in appraising H. H is essentially immune to having its flaws uncovered by the data. H might be said to have “passed” the test, but it is a test that lacks stringency or severity. Everyone understands that this is bad evidence, or no test at all. I call this the severity requirement.”
― Deborah G Mayo, Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars