The reviews of a STAP paper submitted to and rejected by the journal Science in 2012 were posted at Retraction Watch yesterday. They filled in some gaps in the puzzle of the series of events that led to such flawed science being published in Nature in January 2014, but the reviews also raised more questions.
Today, more STAP paper reviews have surfaced.
ScienceInsider posted a piece with additional STAP paper reviews with these coming from Nature reviewers commenting on what would later become accepted and published by Nature only months later in seemingly only moderately revised form.
The Nature reviews (you can read them here on the Science website) are very critical of the STAP papers and raise a host of important, largely still unanswered questions about STAP.
My overall sense is that the three reviewers did a thorough and fair job of reviewing these STAP papers. It sure seems that none of the three reviewers were even remotely close to being comfortable with these papers being published in Nature. In each case it would seem that a major revision would have been necessary prior to even having a remote chance at publication. One of the reviewers summed up a STAP cell article as essentially reporting an unproven, “magical” approach (see screenshot above).
The ball is now firmly in Nature‘s court to facilitate a thorough understanding of the STAP situation. It seems reasonable to expect more from Nature than its one editorial that shrugged off any significant responsibility including this key portion:
“We have concluded that we and the referees could not have detected the problems that fatally undermined the papers. The referees’ rigorous reports quite rightly took on trust what was presented in the papers.”
Nature‘s own reviewers’ comments would seem to directly challenge this statement.
I’m not going to go through all of these criticisms and questions raised in these reviews of the originally submitted Nature STAP papers point-by-point, but the overall consensus was that these papers were seriously flawed. This fits well with the gestalt of the reviewer comments on the rejected STAP/SAC paper at Science.
If you look at the published STAP cell Nature papers and think about the details mentioned in these acidic reviews of the original forms of the same papers, there is a sense that not much fundamentally was improved in the papers during that intervening period of months.
The big question remains then: how did these STAP papers go from being rebuffed based on scathing reviews at Nature on April 4, 2013 to acceptance by the same journal on December 20, 2013 and publication about a month later?
I’m replying to “anon on September 18, 2014 at 7:09 am” but doing it as a new comment because the replies are so nested that a direct “reply” would be only about a few cm in width.
What we do know is (1) the reviews of the initial manuscript in April 2013 were strongly negative; (2) the final published paper (which went on-line in Jan 2014 and was retracted in July 2014) failed to respond adequately to the referee’s comments regarding a large number of important points.
It thus seems reasonable to conclude that it clearly was an error to accept and publish the paper. The only thing we don’t know is who was responsible for the inappropriate decision to publish, and through what review process this decision was reached. Was it the editors alone who decided? Or did one or more of the reviewers recommend publication? Or were there new reviewers who approved it?
I want to use this space to highlight Nature’s inconsistency. Their editorial on the retraction in July 2014 said there was no way the reviewers and editors could have known: “We have concluded that we and the referees could not have detected the problems that fatally undermined the papers.” http://www.nature.com/news/stap-retracted-1.15488
The key point is that Nature’s editorial made claims based on the reviews, which they could see but they refused to reveal. So we had no alternative other than believing them, or not, but without access to the data–the reviews– that would let us independently evaluate their claims. But now, thanks to Science’s publication of the initial reviews, we know that Nature’s editors were well aware of many problems that fatally undermined the initial paper, and that the final paper didn’t respond to these issues satisfactorily.
The only way Nature can regain their credibility is to reveal the rest of the review process in full (after redacting names, etc.), own up to exactly what went wrong, and explain what steps they’re going to take to prevent the recurrence of such problems.
Do editors really trust reviewers? Clearly three reviewers all express major concerns, but they also admit it could be of huge impact. As a Nature editor, you would have seen enough back-stabbing, nastiness, pettiness, jealousy, childish bahavior of many of so-called big-name reviewers, they often just aim to shoot down papers from foreign, small school, little guys. Their comments often sound like “Scientific, logical, miticulous”, always asking for more proof that will take years if not decades or finish. More likely they are washed up that have not made major discoveries for some time themselves. I guess at some point, the editor make up his/her mind, screw these assholes, I will publish this. Based on the extremely low retraction rate, most of times successful. Looking at the big picture, the casino still wins.
If editors do not trust reviewers then why should they ask for reviews? I can understand some reviewers may be biased and thus not trusted. But when all reviews came in largely negative then how can editors push out publication of questionable papers in high-profile? What is the qualification of “general science” magazine editors to over-ride specialized expert reviewers in terms of high-differentiated research fields?
Nature should show its evidence why reviews of STAP cells papers were not trust-worthy and thus could be ignored entirely.
We still don’t have the critical piece of information. It would be conventional practice in the case of a “major revision required” decision for the editors to send the revised manuscript back to the same reviewers to see if they were satisfied with the revision. I’d like to see the review of that manuscript, and find out whether the same reviewers were used. If so, the revision would have to be extremely convincing to sway reviewers who commented so negatively on the earlier submission. What happened between this review and the acceptance of the manuscript in December? That’s what I’d like to know. Were there new reviewers, for example?
The editor(s) handling the STAP cell paper in Nature should be exposed.
I disagree. The editor(s) played a role, but I’m not sure what “exposing” them would achieve except negative things.
What negative things can be achieved with exposing editor(s) who ignored the criticisms of expert peer reviewers and went ahead with a high-profile publication of flawed papers which were rejected by three journals including their own journal before? Which editor(s) wrote “We have concluded that we and the referees could not have detected the problems that fatally undermined the papers. The referees’ rigorous reports quite rightly took on trust what was presented in the papers.”? Does such a conduct of misrepresenting reality a kind of misconduct by editor(s)? Why such unscientific and unethical editor(s) cannot be exposed and punished?
truth4science and I don’t see eye-to-eye on a lot of issues, but here I’m inclined to agree with him.
Researchers’ chances of getting jobs are affected by where you publish, with the implicit (more than likely flawed) assumption that impact factor somehow correlates with quality of research. The fact that this paper was published in Nature almost certainly put Obokata higher up on the short list for her professorship than the competing candidates.
For an editor at Nature to overlook all this, especially given the form of the final published paper, suggests that there is something else going on here. The best case would be “publish the paper solely for hype reasons”; the worst being “corruption at the editorial level”. Both are fairly bad and both are probably expected. This doesn’t make it right.
As scientists, we are judged on our reputations, i.e. whose results do you trust, who does good work, who does *careful* work. For editors at what are at a journal supposedly in the highest echelon to not be held to the same standard is simply ridiculous to me, especially given the CYA statement Nature put out saying there was no way to detect the flaws in the paper.
goresy and GLG can’t make money if they can’t control the process and boss around Nature’s editors.
It’s “The Bloomberg Way”, guys.
These comments assume that no changes were made to the paper and that the referees did not see the revised manuscript and judge it to be ready for publication. That is not known.
Dr. Knoepfler, do you have any reservations to provide the name of the editor at Nature in charge of accepting the STAP papers? This could be informative for scientist who wants their “hot” stories published in Nature where the editor completely disregards concerns raised by reviewers. 😉
Yes, I do have reservations about providing the name.
Paul,
You have so far done science a service both with the questions you have raised ( I share the confusion of others as to why these papers were not rejected) and it’s important that we continue with this until we have a clear picture and an identified lesson to be learnt. However, many careers have been damaged so far and more will be. Is this justified ? For some the answer seems yes but we need now to be careful how far this goes. The principle of the punishment fitting the crime is important to keep in mind just as identifying the lesson to be learned is. In saying this I emphasise I have NO criticism of Paul but rather great praise for him. I just don’t want to see the damage go beyond those who deserve it rather than those merely peripherally caught up in this.
MH
Thanks for the comment, Michael.
Unfortunately the damage from STAP could spread to those not involved. I’ve never in any way including on this blog ever favored anyone being punished in any way for STAP. I stick to the science, policy, and what we can learn.
Paul
If scientific misconduct, when exposed, doesn’t become the subject of disciplinary measures there’s no incentive not to commit misconduct.
Also, if scientific misconduct in Japan (or any other country) isn’t dealt with appropriately, then the reputation of all research in Japan (or in whichever country the misconduct was committed in) will suffer.
So let’s not forget that there is also grave collateral damage to innocent people created by failing to deal appropriately with scientific misconduct.
Michael,
I understand your concern that some innocent scientists might be hurt by the negative image presented by the STAP cell scandal. It is for this very concern that I have emphasized the major punishment should be applied to those journals which have kept selling some pseudo-science to public and meanwhile got rising impact factors. Just thinking of the fair punishment system in any commercial system, can any store which has been caught of frequently selling defected products as good products escape any punishment? But Nature seemed to be just one of such a punishment-immune store that can continue see pseudo-science to public without suffering any valuable damage. Instead its impact factor has kept rising to such a level that even out compete its equivalent journal Science. Want to learn the secret of Nature’s success in the impact factor game? Do a study on how publishing high-profile but retraction-warranted “discoveries” change the value of impact factor.
I think if Nature is not to be punished with its crime to science then many more scientists would be hurt by its corrupted publishing practice. Remember what Nature one published: Publish or Perish. And change it to reflect its reality as Publish and Perish — the STAP way.
One more question about Nature’s handling of the ms. Given the strongly critical nature of the reviews rejection would be the expected decision, but instead the actual decision was major revision (click on this link to submit a revised version…). Why were Nature’s editors so lenient?
Given the comments, “reject” would have seemed more likely than “major revision”.
Why did they go for the latter? Who knows. What do you think?
I think if Nature wants to retain their credibility they need to explain this, as well as their decision to accept the final ms.
Sure, they were (very) critical, but I can totally read these as “major revisions” reviews: After all, the reviewers admit that this would be quite a breakthrough if it WERE true.
In fact, if the reviewers thought it was a load of humbug from the get go I would have expected these reviews to be a lot shorter and deadlier.
The really puzzling thing is how the authors managed to get in in the end without addressing most of these comments.
I don’t know about you, but whenever I want to get a paper rejected I write a long laundry list of problems in the paper (just like the Nature STAP reviews) which I know are problems the authors can’t respond to (which is why I think the paper should be rejected in the first place).
If I were to write a “short and deadly” review it could be dismissed by the editors and authors as a biased and emotional rant.
In any case though, as you correctly point out, even if one accepts “major revision” as an appropriate decision on the ms submitted to Nature in March 2013, the final accepted ms failed to fix most of the problems pointed out by the referees, so it’s baffling that it was accepted.
There are a lot of assumptions being made here that the manuscript was not either revised or seen again by the referees. That is not known. By all means, take lessons from the reports we have, but don’t assume the mystery is solved without more information.
The publication of the STAP papers has been a troubling mystery for those of us who read them and tried the method. The Nature reviews are exactly what we needed to see to settle our concerns at last. Thanks so much, Paul, for pointing these out to us. I never thought we’d be able to see them.
Jeanne
The situation itself is, shall we say, unfortunate, but I appreciate the dedicated commentary here. It takes time to put an article together.
Dr. Knoepler,
I used to enjoy your blog. But really this whole STAP thing. You make it seem like a papers never been retracted before. Goodbye blog…
Hi Daniel,
My view is that this is an extraordinary situation warranting extensive coverage and discussion. That’s it.
Paul
Daniel,
I think Paul has done a very good service to scientists working in this field as well as public interest in this aspect of science. Yes, papers have been retracted and Nature is retracting much more recently. But this STAP cell scandal should not be ended with just retractions of papers. If that is the case, no real lesson will be learnt from this.
The real lesson in this case is: Nature has violated some basic scientific principles and working ethics to publish something that it believed in but not supported by any science. In other words, (unscientific) editor(s) has/have ruled and thus ruined science.
Stay tuned, this STAP cell saga will turn out to be more interesting and thus more amazing than Hwang’s stem cell scandal.