Skeptics Proclaim Themselves the Odds-On Favorites

I've covered Daryl Bem's precognition study here in the past. But I wanted to include a quickie here, from ScienceDaily, on a study by Jeffrey Rouder, at the University of Missouri, and Richard Morey from the University of Groningen in the Netherlands, who performed a new statistical analysis of Bem's series of precognition experiments and determined his research offers only "modest" support for ESP.

Their application of a relatively new statistical method that quantifies how beliefs should change in light of data suggests skeptics should move their belief meter by a factor of 40. In other words, a skeptic giving odds against ESP being true at 1 million to 1 would now need to revise that down to 25,000 to 1.

The full paper is available here. But a couple of points from me, first.

Rouders and Morey are skeptical of psi but in one important instance they do all right by Bem, noting that the more scathing analysis of Eric-Jan Wagenmakers failed to capture the real success of his experiments.

"On balance, according to Wagenmakers et al., there is little systematic evidence for ESP. We have added a third column as a validity check, and it provides the direction of the effect. In several of Bem's experiments, one could be reasonably sure that if ESP held, the effect should be in one direction and not the other. For example, in Bem's Experiment 1, discussed previously, participants were instructed to indicate the curtain behind which there was an erotic picture, and, if ESP held, their performance should be greater rather than worse than chance. If there were no ESP, we would expect the observed performance to be slightly below chance for some experiments and slightly above chance for others. Table 1 shows that the direction of all 10 were in the direction hypothesized by Bem. This concordance serves as evidence for ESP that is not captured by Wagenmakers et al.'s analysis. In fact, the Bayes factor of getting all 10 contrasts to be in the same direction is about 100:1 in favor of ESP."

What's novel about their means of analysis is that it factors in the degree to which Bem's psi-positive findings should influence what they call an "appropriately skeptical" reader.

In other words, the beliefs we build up, based on the evidence at hand, can be ascribed an odds value, and contrary evidence can then be rated in terms of how much this might reinforce our beliefs or force us to revise them. The amount of subjectivity involved in this process—anyone care to define the "appropriately skeptical reader"?—is itself staggering.

The full paper is worth reading, though, if only to see how quickly and casually Rouder and Morey unravel themselves. They dismiss the piles of evidence in favor of telepathy with a quick wave of the hand—"We worry about the frequency of unreported studies"—primarily, I think, because they have absolutely no idea what they're talking about. As I write in Fringe-ology, research actually shows that parapsychologists are far more likely than any other field of science to report negative results.

Rouder and Morey also don't take the obvious and necessary step of calculating how many people are actually involved in conducting telepathy experiments around the world—maybe a dozen or 20?—and how many negative experiments they'd have to do in order to undo the positive findings that have been published.

Why don't they look into these factors? My guess is simple confirmation bias.

They don't believe psi is legitimately possible, focusing on the "lack of a plausible mechanism", and so they aren't interested in thinking as critically about their own analysis as they are psi itself.