Talk about cognitive dissonance. Professor Lawrence Sanna, formerly of the University of Michigan, published the results of a series of studies which had different averages (means) but uncannily similar standard deviations—and those deviations were highly significant.
With apologies to Shakespeare’s Queen Gertrude: “The researcher doth deviate too much, methinks.” Of course, the subject matter under consideration IS experimental psychology. So, in one sense, pushing the envelope goes with the territory. Still…there are boundaries…as Professor Sanna discovered.
Each week, OTW publishes a recent scientific journal retraction arising from shoddy, lazy or downright fraudulent research. These are examples of researchers who omitted or falsified data, used data out of context or employed such awful logic that they were forced to retract their study.
The examples are collected by Retraction Watch (RW) and we are honored to be able to present them with permission from RW to our readers. Retraction Watch was started in 2010 by Adam Marcus and Ivan Oransky, M.D.
Sanna’s “Odd Statistical Patterns”
Dr. Sanna was trying to determine if riding in an elevator affected a person’s ability to make moral judgments. Specifically, he asked the question—does increasing people’s vertical height, such as by riding an ascending escalator, affects their moral virtues?
He then designed some tests to answer that particular, pressing question.
Like we said, pushing the envelope. Here are the studies he designed.
Rising up to higher virtues: Experiencing elevated physical height uplifts prosocial actions, cited twice, according to Thomson Scientific’s Web of Knowledge
Think and act globally, think and act locally: Cooperation depends on matching construal to action levels in social dilemmas, cited three times
When thoughts don’t feel like they used to: Changing feelings of subjective ease in judgments of the past, cited three times
So this other researcher, one Uri Simonsohn, was reading Sanna’s studies when he noticed an “odd statistical pattern”.
Uri Simonsohn, by the way, isn’t just your run of the mill lab rat researcher. Dr. Simonsohn is, apparently, the proof reader’s proof reader. Ed Yong, writing in the July issue of Nature, credits Simonson with also spotting questionable data in studies by social psychologist Dirk Smeesters.
So, here is Dr. Simonsohn whiling away a July afternoon, perusing the latest literature on embodied cognition and he comes across these studies with such titles as “when thoughts don’t feel like they used to…” and “cooperation depends on matching construal to action levels in social dilemmas” when he spotted something strange.
Yes, everything he was reading was normal—until he spotted the strange numbers. Only in academia.
So he’s looking at the numbers in Sanna’s studies and something just didn’t add up. Recall’s Dr. Simonsohn for the journal Nature: “The evidence was very strong compared to the other papers and it puzzled me. Every result was super-significant, and there were very large effects.”
Simonsohn also commented in that same Nature story that although the results of tests had different means, they had uncannily similar standard deviations. “I ran simulations and the similarity was extremely unlikely for proper random samples, ” he recalls. Simonsohn found three other papers by Sanna and several from other researchers that used one of the methods found in the elevation paper.
That particular study method, by the way, was a cooperation game that involved fishing. We’re not making this up. The other question is who would fund these studies? Can we get the name? Phone number?
In September, Simonsohn sent an eight-page report detailing his concerns to Sanna and two of his senior co-authors. He received back raw data, which revealed almost identical ranges between the maximum and minimum data points, across different conditions. “That’s extremely rare, ” Simonsohn says.
According to RW, Sanna then contacted the publisher of the Journal of Experimental Social Psychology and asked them to retract three papers published from 2009 to 2001.
The publisher complied.
Resignation
Sanna, according to Yong, told Simonsohn that he resigned his professorship at Michigan at the end of May. The reasons for Sanna’s resignation are not known.
In both Smeesters’ and Sanna’s work, Yong says that odd statistical patterns in the data raised concerns with Simonsohn. “But the similarity between the cases ends there. Smeesters’ resignation was announced on 25 June by his institution, Erasmus University Rotterdam in the Netherlands, which undertook a review and concluded that two of his papers should be retracted. Sanna’s resignation, by contrast, remains mysterious: UNC [University of North Carolina where Sanna was a professor of psychology] did not release the results of its review, and the University of Michigan will not explain why Sanna resigned.
Co-Authors Not Involved
Yong writes that Simonsohn exchanged e-mails with Sanna and his co-authors throughout October, offering to discuss his concerns. Eventually, the replies stopped. When Simonsohn contacted three graduate students (who each appeared as co-authors on at least one of the four papers), all said that they were not involved in collecting or analyzing data. Simonsohn adds that he has no evidence or suggestion of any data manipulation by the co-authors.
Professor Lawrence J. Sanna
Microsoft Academic ResearchAccording to Microsoft Academic Research, Sanna is credited with 76 publications and 862 citations.
These retractions bring up the dangers of drawing wishful conclusions and associations from data that may or may not be accurate, properly peer-reviewed or is biased. Also concerning is when co-authors are later found not to have collected or analyzed data, particularly when they are subservient to the lead author.
On the positive side, Sanna apparently knew how to obtain funding for some pretty ethereal studies. Surely there’s a place for someone with such talents.
These studies bring to mind the words of Steve Jobs in historian Walter Isaacson’s biography of the same name: “You can’t understand me if you haven’t dropped acid.”
