Voodoo Research
I wrote the following back in 2000, when I worked at STATS, but was unable to find an outlet for it. Now that Jonah has mentioned something strongly related on The Corner, I thought it worth posting here.
Voodoo Research
June 01 2000
Iain Murray
Applying skepticism to the Skeptical Inquirer
Skepticism in scientific inquiry is a good thing. It protects the research from the twin evils of prejudice and equivocation and forms a bright light that can shine through all sorts of obfuscatory cloudings of the issue. Skeptical Inquirer, the journal of CSICOP (the Committee for the Scientific Investigation of Claims Of the Paranormal), is therefore also a good thing. It calmly and rationally debunks all sorts of idiotic claims about aliens, ghosts, graphology and the Loch Ness Monster. But for all the motes in other people’s eyes, there is almost always a beam in one’s own. In the Skeptical Inquirer’s case, the problem is with religion, and skepticism seems to give way to prejudice.
The Inquirer’s January/ February issue confirms this. In a piece entitled “Two Paranormalisms or Two and a Half? An Empirical Exploration,” State University of New York sociology professor Erich Goode examines the proposition that those who believe in traditional religion are unlikely to give credence to paranormal claims and other “new age” beliefs. He suggests that there is a gray area where a “traditional Christian background lays the foundation for many paranormalisms.” In an effort to examine this suggestion, Professor Goode conducted an experiment, which he claims backs up his thesis.
Unfortunately, it is here that Professor Goode’s skepticism breaks down. He conducted his experiment by means of a survey of 484 undergraduate college students, spread over three courses. The first objection to the results being meaningful should be immediately apparent. Far from being a representative sample of the population at large, the sample, encompassing but three courses and not including graduates, fails even to be representative of students at the University. No opinion pollster would expect payment if he presented the results of a poll based on such an unrepresentative section of the American people. Such “quasi-experimental” surveys do have a value in helping to refine questions that need to be asked - they are like large focus groups in that respect - but to claim that they provide concrete evidence about the proposition is as unscientific as a belief in Bigfoot.
A second problem lies in the characteristics of the sample. The students at Stony Brook seem exceptionally credulous. A full 72% of the sample agree with, and a further 17% are “not sure” about, the statement “Some people have ESP.” Astonishingly, 60% of the sample believe that “Astrology is scientific” (perhaps there are grounds for the University of Oxford reinstating its medieval Chair in Astrology), and 43% of the sample agrees that “Some numbers are lucky.” The figures drop off considerably for the statements “Many UFOs are alien vehicles” and “King Tut’s curse killed people,” but they are bolstered by a hefty degree of “not sures” in both cases. By contrast, only 54% agreed with the statement “Heaven exists,” 43% with “Angels exist” and a mere 15% with “The Devil exists.”
These figures are illuminating, or, perhaps, they have made darkness visible. In this sample, at least, “new age” beliefs are prevalent. Perhaps that is not surprising; students are, by their very nature, open-minded people. Given that, and the proportion of those who believe in what used to be called “unconventional” beliefs, it should come as no surprise that there is a significant number that accepts both Christian and “new age” tenets. Any mathematician will tell you that there is a good possibility of significant overlap between two large subsets of a sample. Goode found a statistical likelihood that someone who believed Christian teachings was more likely to believe “new age” precepts as well. That should not have come as a surprise given the characteristics of the sample.
But even given this, Goode fails to distinguish between two categories of “new age” belief. The first category consists of beliefs which may be seen as somehow compatible with Christianity - lucky numbers (prominent in the Bible and Christian tradition), ESP (almost necessary for a belief in certain forms of miracles) and astrology (a prominent part of Christian belief until the Reformation, and now perhaps regarded as being part of the occult evil which opposes Christianity) - while the second consists of propositions to which the Christian church is either neutral or hostile - alien activity (perhaps contrary to a creationist view of humanity) and King Tut’s curse (predicated on neither God nor the Devil). There was a significant degree of overlap between Christian belief and the first category (78% of believers in heaven also believed in ESP), but a much smaller overlap between Christianity and the second (only 20% of believers in heaven believe many UFOs are alien vehicles).
Professor Goode and the editors of Skeptical Inquirer failed to apply to this research the skepticism they normally apply to other claims. They did not give sufficient caveats about the unrepresentiveness of the sample, they have failed to note the statistical likelihood of there being overlaps given the characteristics of the sample and they have failed to flag up an interesting dichotomy in the overlap. It is unfortunate, but from now on I have grounds to be skeptical about the Skeptical Inquirer.