In 1981, Al Gore was the chair of the House Committee on Science and Technology's Subcommittee on Oversight and Investigations, and in spring of that year he chaired two days of hearings on data fabrication and plagiarism in biomedical sciences. Testifying at those hearings, the president of the National Academy of Sciences, Philip Handler, said that scientific fraud is rare and, 'it occurs in a system that operates in an effective, democratic and self-correcting mode.' Handler was correct that on average and in the long term, science is self-correcting. Solid discoveries are followed up with more work, extending and verifying them. Mistakes and fraud cannot be reproduced and eventually fade from the research conscience. However, 'long term' in here can be decades if not more. In the more immediate reality, an army of factors conspire against the self-correcting ideal. See The Myth of Self-Correcting Science by Sarah Estes and Unlike a Rolling Stone: is science really better than journalism at self-correction? by Adam Marcus and Ivan Oransky. There are few incentives for publishing negative results and many reasons why most scientists simply tend to move on when they cannot replicate another group's findings. But what happens to those few who do take the time to write a paper challenging a previously-published work? They ignore the possible retribution from the scientists behind the original work (scientists who may be reviewing their grants, fellowships, tenure applications, and future papers). They take the time to establish that the negative result is not just a mistake on their part when replicating someone's work. They commit their time and energy to writing the paper and then — and then, they discover that publishing such a paper is often pure torture. Compare the self-reported rejection rates for general papers versus those that question a previous result. The frequency of getting 3 or more rejections jumps from 14% to 37%. [Note: these are online Twitter polls, biased by my followers; the absolute numbers are not necessarily representative of different fields, but the comparison between the two polls is meaningful.] Below is an e-mail sent to a colleague of mine who was then a graduate student (we'll call the colleague, Terry). Terry had shared with a professor a draft of a paper, asking for feedback prior to submission. Here is the feedback: Terry - it' been a tough 3 days for me and I regret being tough on you - but when you say that Mike and I are wrong in saying that [science details redacted], you are not only wrong - but disrespectful. I like your work showing that [redacted] - good for you. What pisses me off is that you use this observation to discredit Mike and me. I'm OK with this, people have done this to me all of my career. That you have done this has made you my enemy - such is life. If you can make more fundamental discoveries than Mike on the [redacted], good luck. But just because you have been at Stanford and Caltech, don't think you can beat us with politics. Mike is not stupid, nor am I - we could care less about politics, systems biology and all of the stuff that you think is your advantage. You are into a gunfight with a knife - we have real guns - Harvard, Stanford, nowhere will help you fighting me - L. Frequently, the original authors whose work is in question are asked to review or provide a response to the new paper. The professor above could very well be that reviewer. Moreover, journals and editors tend to resist publishing such corrections. That is why PLOS ONE, from the moment it was founded in 2006, is such an important outlier in the biomedical publishing landscape. From their negative results collection: The publication of negative, null and inconclusive results is important to provide scientists with balanced information and avoid the duplication of efforts testing similar hypotheses, which waste valuable time and research resources in the process. PLOS ONE considers all work that makes a contribution to the field, independent of impact. This includes negative findings which are valuable to the community in cases where the result is illuminating in the context of previous work. I recently put out a call for stories from those whose papers in PLOS ONE challenge prior results. Here are some of them. (I will be adding more to this post as I get them, and if you have an example, feel free to share it in the comments below or to email me at lenny at protocols dot io.) ------------------------------------- Citation: Fan X, Struhl K (2009) Where Does Mediator Bind In Vivo? PLoS ONE 4(4): e5029. doi:10.1371/journal.pone.0005029 Xiaochun Fan and Kevin Struhl questioned two papers in Molecular Cell that claimed the Mediator protein complex binds to promoters of many genes. From Kevin Struhl: The paper was originally sent to Molecular Cell in March, 2008. It was rejected several months later; decision letter below. It was sent to JBC in June, 2008. It was reviewed, mostly positive, and revisions were asked for. The revision was sent back in November, 2008, but the Editor changed her mind and decided that we didn’t learn anything new about Mediator. This wasn’t the issue in the original reviews, and was a ridiculous decision. So, we sent it to PLoS ONE in January 2009 From the rejection: It is important to us that the papers we publish are correct and the scientific issues you raise should certainly be made aware by the field. However, the consensus is that it will be more appropriate to communicate your study as a standard science debate rather than in the specific Matters Arising format. In this case, the reviewers are simply not sufficiently enthusiastic about the paper for us to consider it further for Molecular Cell. We would therefore like to suggest that your paper may best be published in a more specialized journal. ------------------------------------- Citation: Miller CN, Yang J-Y, England E, Yin A, Baile CA, Rayalam S (2015) Isoproterenol Increases Uncoupling, Glycolysis, and Markers of Beiging in Mature 3T3-L1 Adipocytes. PLoS ONE 10(9): e0138344. doi:10.1371/journal.pone.0138344 From Colette Nicole Miller: Briefly, we submitted to 5 journals total before we submitted to PLOS One. While one journal sent us out for review, four desk rejected us and all had a similar reasons 'level of priority' and 'lack of novelty' cited. I wouldn't say that our work was anywhere high impact, but it was important for us to nail down the correct protocol for our cell model. Because our addition to the previously published model showed that their doses and treatment length wasn't enough to produce a physiological response in the cell, we thought it was worthy of publication. We were very happy that we decided to finally submit to PLOS One and found it to be well received when it was published. ------------------------------------- Citation: Lifjeld JT, Laskemoen T, Kleven O, Pedersen ATM, Lampe HM, Rudolfsen G, et al. (2012) No Evidence for Pre-Copulatory Sexual Selection on Sperm Length in a Passerine Bird. PLoS ONE 7(2): e32611. doi:10.1371/journal.pone.0032611 This paper argued against the conclusions of a study published in Biology Letters. From Jan Terje Lifjeld: Yes, we originally submitted our paper to Biology Letters. It was reviewed, revised and resubmitted to that journal, but finally rejected. The decision letter was as follows: [rejection letter] So, the rejection was based on conflicting reviews (we got four reviews on the resubmitted version) and no opening for further revisions. My reading of the letter is that we didn't pass the 'novelty bar' in the journal, so the conflicting reviews were a good excuse for the editor to reject, despite the fact that the criticism was largely on the presentation/interpretations rather than the results (which were quite robust).We thereafter sent the manuscript to PLoS One which published it after one round of revision. ------------------------------------- Citation: Nagelkerke SQ, aan de Kerk DJ, Jansen MH, van den Berg TK, Kuijpers TW (2014) Failure To Detect Functional Neutrophil B Helper Cells in the Human Spleen. PLoS ONE 9(2): e88377. doi:10.1371/journal.pone.0088377 This paper challenged the results published in Nature Immunology. From Taco Kuijpers: YES, of course we had submitted a very nice 2 page short letter to Nature Immunology with a lot of data as supplementals to suggest publication to the editor(s) as a note of caution with the paper in their journal related to the story on Nbh1 and Nbh2. We had worked it out as finally got publsihed in PLoS One. Funny enough we got back a reply of 11 pages (11!) from the authors who had been invited by Nat Immunol to respond. This was probably enough for the journal to refuse our letter and leave it with that. The reply was appalling - to be honest, being from a lab specialized in neutrophil work we were set aside as mere amateurs. What can you do... ------------------------------------- Citation: Sanchez C, Sundermeier B, Gray K, Calin-Jageman RJ (2017) Direct replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious belief. PLoS ONE 12(2): e0172636. doi:10.1371/journal.pone.0172636 This work disagreed with a paper in Science. From Robert Calin-Jageman: Thanks for the interest in the backstory. Your guess is right—it was a long and painful process to publication. PLOS One was our 4th attempt. I’ve summarized the whole process below. Perhaps more detail than you’d ever want, but it was helpful from my end to put this all together. In my career, I’ve never had such a painful and long process from completion to publication. One might have hoped that the tide had turned for publishing replications, but I don’t see things getting any easier—it’s still tons of work for essentially no reward. As you’ll see below, part of the problem is that editors had criteria for publication that were a) not the same as for the original paper, and b) impossible to meet given the flaws in the original research. [Note: At the request of Professor Calin-Jageman, I am not sharing the rest of this story until a full profile of it is published (due soon). I will add the rest of the reply once that story is published.] [Update 3/15/17] The story profiling this work is now published Religious Belief and Analytical Thinking Don’t Necessarily Cancel Each Other Out, so I am sharing Robert Calin-Jageman's full e-mail to me below: Thanks for the interest in the backstory. Your guess is right—it was a long and painful process to publication. PLOS One was our 4th attempt. I’ve summarized the whole process below. Perhaps more detail than you’d ever want, but it was helpful from my end to put this all together. In my career, I’ve never had such a painful and long process from completion to publication. One might have hoped that the tide had turned for publishing replications, but I don’t see things getting any easier—it’s still tons of work for essentially no reward. As you’ll see below, part of the problem is that editors had criteria for publication that were a) not the same as for the original paper, and b) impossible to meet given the flaws in the original research. One thing that *might* be changing is interest from the media. Maybe because the original author blogged favorably about the replication (see below) I was contacted by a science journalist (DalmeetSingh) who is prepping an article about the original study and the replication. Not that we are looking for media attention, but given that the original was covered in the press so much it is nice to see some small effort to correct the public record on the finding. On that note, feel free to use any of this email in any way you see fit. But if you are going to blog or post publically about it, could you please wait until the article by Singh goes live? My understanding is this should be a week or two. Cheers, Bob 1) We submitted the paper to Science on 3/21/2016 as a 300-word comment. It was returned 3/29/2016 without review Thank you for submitting your manuscript "Analytic thinking does not promote religious disbelief: Direct replications of Gervais & Norenzayan (2012)" to Science. Because your manuscript was not given a high priority rating during the initial screening process, we have decided not to proceed to in-depth review. The overall view is that the scope and focus of your paper make it more appropriate for a more specialized journal. We are therefore notifying you so that you can seek publication elsewhere. To which I replied: Ok - thanks for letting me know. It's a shame Science doesn’t find it a high priority to correct the record on unreliable papers it has published. I guess science as whole is self-correcting, but Science the journal is not. No response… 2) After that, we sent the manuscript to Psych Science, which now has an article format for very brief reports of replications of work published in Psych Science (though, as far as I know, no one has every managed to publish this way). It was a long shot, given that the original was not in Psych Science. We submitted on 4/12/2016. The result was rejection without review on 4/17/2106. The editor was sympathetic in some ways: Thank you for the opportunity to consider "Analytic thinking does not promote religious disbelief: Direct replications of Gervais & Norenzayan (2012)" for Psychological Science. An Associate Editor and I have read the manuscript. There is a lot to like about your manuscript. I appreciate that the studies were pre-registered, that they had large N, and that the procedure was conducted with four different samples. Also, the paper is very well written; you tell the story in a clear and effective manner and with great economy. But in my judgment there was very little reason to believe that the manipulation would have any affect on analytic thinking. So the finding that, sure enough, it didn't, is not sufficiently interesting to warrant publication in Psych Science. You imply that your findings indicate that promoting analytic thinking does not affect religiosity, but your findings cannot speak to that issue because the manipulation did not affect analytic thinking. Consequently, I have decided to decline the manuscript without having it sent out for extended review. 3) Third, we submitted to Social Psychology, expanding our very brief comment into a full-scale manuscript. I think we submitted on 7/21/2016. Social Psychology is a solid journal, but it has a low impact factor and I expected the manuscript to breeze through. To my surprise, it was rejected without review on 7/29/2016. The editor wrote: I read your manuscript with great interest. I can completely understand that it is worthwhile to replicate a study that has been published that prominently. I also admire the amount of work you’ve invested in the project. Yet, I have decided not to send out your paper for review. G&N aim to show that analytical thinking fosters religious disbelief. They do that by showing that their manipulation (i.e. the sculpture task) affects analytical thinking (in a pretest), and that the same manipulation affects self-reported religious belief/disbelief in the main study. This original setup is not ideal, and even in 2012 having a logical chain confirmed by such a setup was most likely not viewed as the optimal way to establish empirical evidence. All of this is of course not your fault. However, it makes the task to replicate that research a bit trickier. A direct replication of that research again leads to a suboptimal design. … Given the suboptimal design of the original study my remarks might lead to the impression that I put an unnecessary burden on any replication effort of the G&N studies. However, I think it would be essential to first look at whether evidence can be found for the manipulation to really affect analytical thinking. What is interesting about the rejections at Psych Science and SP is that both are based on the fact that the manipulation (looking at The Thinker or Discobolus) does not actually alter analytic thinking. This is true—the methods used in the original study now seem to completely lack construct validity (despite data shown in the supplementals of the original paper whic . But holding this up as an objection to the replication means that the original paper is forever protected from assessment. The authors then never have to explain how they got amazing results with a method that is basically inert. That doesn’t seem right! Plus, it could have been possible that the manipulation produces reliable effects, just not through analytic thinking as proposed (poor construct validity is not the same as poor reliability). Anyways, we finally went to PLOS One on 9/13/2016. Normally this is a fast journal, but it took a long while to get through review. PLOS One has a policy that adversarial papers must be sent out for an invited but non-binding review to the original authors. It took the journal about 2-3 weeks just to figure out that our manuscript qualifies for that policy, then it went through regular review, then we amended, then it was re-reviewed, then it went to the original authors, and then we amended again, and then it was finally published on 2/25/2017. The original authors were really gracious. They were very fair during review and did not try to hold up publication. The lead author even blogged about the replication, accepting it as definitive:http://willgervais.com/blog/2017/3/2/post-publication-peer-review . So the crazy thing is that the difficulties in getting this to press came from editors, not from the original researchers, who were quite supportive. -------------------------------------