The ultimate fate of the two Nature STAP cell papers remains in troubled limbo.
One of the senior STAP cell authors, Dr. Sasai, held a news conference yesterday in Japan that included a call for the papers to be retracted.
He was variously quoted as believing in STAP cells or alternatively as just thinking it was an unproven hypothesis.
Whether STAP cells or STAP stem cells are real or not, overall the STAP cell situation has been disastrous. Can biomedical science learn anything from this STAP fiasco? Maybe…some thoughts below.
- Organize, keep records, and annotate your images. It seems that the STAP papers were plagued by confusion over images and the way image data was handled or changed. More broadly in science, sometimes research projects generate tons of data even if they are not “big data” genomics projects. In fact, it is not unusual for just one line of cell or developmental biology research to generate hundreds of image files. Each one might have a different exposure time or other varying attributes and researchers might legitimately adjust some images that are too faint, etc. It is wise to use a system where lab members catalog images and a naming paradigm that includes the date. Any changes to images must also be documented in writing and original unmodified forms kept as trackable backup files.
- Don’t always believe your eyes. It seems that some of the STAP authors believe in STAP, but I wonder if “STAP” to them means simply cells glowing green? The reality is that for cells to be STAP they must have functions and pass a whole host of tests, not simply glow green (even if that green is real and not autofluorescence–see next point below) from a Oct4-GFP reporter. Just because you “see the green light” doesn’t mean it is STAP. Human beings including scientists are very visual creatures. Who doesn’t find certain microscopy images captivating? Seriously, a microscopy image can be like a piece of fine art. But sometimes data in the form of a visual image can be deceiving. The more general expression “I see the light!” is about discovery, but usually more about a discovery of beliefs rather than facts.
- Danger, autofluorescence ahead. And speaking of “seeing the green light”, it was only days into the STAP craziness when a stem cell biologist told me in confidence that s/he believed STAP could be largely a mistake due to misinterpretation of autofluorescence as real signal. I still haven’t seen compelling evidence against the notion that the greenness of STAP is just autofluorescence in certain images and FACS data. Perhaps this STAP mess will make the entire biomedical research community more cognizant of the dangers of misreading cellular autofluorescence and the need to check for it.
- Cells are not always what you think they are. It seems quite possible that some of the cells involved in the STAP cell research were not what some of the researchers thought they were. Cell line contamination is a common problem at least in part because different kinds of cells are stored and sometimes simultaneously grown in labs. Cells also grow at different rates so contamination of one cell type with even a few different cells can burgeon into a big problem over a surprisingly short period of time.
- To be a good reviewer, data should always trump big names in importance. One of the problems exemplified by the STAP papers is that big name authors can sometimes sway reviewers inappropriately to be lenient on papers. In the end, as a good reviewer, you have to keep focused on the data, not the reputation of the authors.
- Weigh the risks, benefits and responsibilities of being an author yourself. If you are possibly going to be an author on any given collaborative paper, use caution. Read the paper carefully, ask to see data if you have concerns, think about what it means for you to be an author of this paper, and if in doubt, at least consider saying ‘no’. In this time when most everyone wants more publications, sometimes paradoxically it is best not to be an author. Of course, sometimes potential co-authors or even corresponding authors don’t know about problems in papers and such problems can be hard to find so the decision as to whether to be an author can be tricky. Finally, take the specifics of those “author contributions” sections seriously as to what you did or did not do for any given paper. I wonder at this time how many of the STAP authors, if they could go back in time, would choose not to be author on those papers?
- To editors, be extra-cautious about those “sexy” papers. A paper like either of the STAP ones is certainly exciting on first read and could have big impact. You might call them “editor-bait”. Heck, despite the controversy the STAP papers have already been cited many times in just a couple months by other papers. However, these kinds of high-profile papers are high risk for journals and editors too. As with the reviewer caution above, editors should not be swayed by big name authors if the story seems too good to be true and if anything, the more excited an editor is about a paper the more cautious they should be in how they handle it. Paradoxical? Perhaps, but I think it’s true.
- Update: To journals, give all manuscripts a thorough checkup. EMBO now reportedly has a screening process for manuscripts for image issues (manipulation, duplication, etc) and EMBO editors have indicated that the STAP papers would not have passed. Did Nature not have such screening in play when the STAP papers were reviewed? Does it now hopefully have such a system? Which journals automatically test for plagiarism of text or images? Clearly this kind of automated manuscript checkup should be standard procedure for all journals. Please see important info from EMBO editor Bernd Pulverer in the comments section
- To scientists, don’t fall in love with your hypothesis. STAP almost feels like a fairy tale love story gone bad. I’m not talking about the love of two folks for each other like Cinderella and the Prince in a Disney movie, but rather the way scientists can sometimes fall in love with an idea. Avoiding this trap is naturally easier said than done because ideas can be super exciting.
- Check the hype. There is nothing wrong with being excited about a paper or its potential impact, but be cautious about crossing the line to outright hype. Not everything is a “breakthrough” and that’s OK. Good, strong science doesn’t have to be a stunning breakthrough to have a positive impact. Scientists, journals, and institutions need to walk a fine line between advocating for our work publicly (which is needed) and overstating its importance, especially to the public or reporters. Many media folks are prone to hyping science as well. I believe that STAP was hugely hyped by many of the parties involved.
Any other things possibly to be learned in a positive way from all the STAP calamity?
Thanks for the comment about journal image checks, Paul.
I would like to clarify that while we thought years ago to introduce automated image checking processes, this is not at all trivial. In our experience the human eye is very good at detecting simple image aberrations and duplications (as long as they are within the same manuscript of course).
The process we have in place at EMBO Press is entirely human based: we have a trained image integrity analyst who screens all figures and supplementary figures in a paper using a variety of photoshop tools (similar to ORI standards) – this takes less than 20 minutes per manuscript. In addition, the EMBO scientific editors look at all figures in detail as part of the manuscript assessment process. Any aberration are carefully evaluated and weighed to decide what action to take. Our process is not designed to catch carefully executed fraud, but rather cases of blunt image processing aberrations, beautification and occasionally image manipulation. We do not claim to be, nor do we see it as our responsibility to act as the data police; we aim to prevent mistakes and unacceptable image processing from entering the literature.
Regarding the Nature paper, we asked our image analyst to undertake her routine screen when the first comments circulated and without her knowledge. As I indicated on a comment at Nature, she identified the issues readily (and in fact an additional splice mark).
I’d like to add my tuppence worth, to the last lesson that you had mentioned ‘Check the hype’. The abstract of Obokata et al.’s main paper in the Nature of January 30, 2014 (vol.505, p.641), is a good example for this hype. As a zoologist, my main gripe is that, they do mention the word ‘mammalian’ twice in the abstract, without mentioning ‘mouse’ or ‘mice’ even once. There is a BIG difference between the word ‘mammal’ (which includes mouse, rat, rabbit, dog, elephant, chimpanzee and human) and the specific animal ‘mouse’. If the authors had the temerity to hype and mention ‘mammal’ in the abstract, then they should have provided data for their experiments in other mammalian models too. If the 8 authors of this paper had intention to hype their data, I wonder why Nature’s reviewers goofed on this.
Nice post, just one caveat from me:
“To journals, give all manuscripts a thorough automated checkup.”
This is a good idea, but there is a danger of putting too much faith in these tools. These are good for screening submissions and flagging up possible problems for a closer look. But they should not be taken as authoritative in their own right. No editor should think “it passed a software check, therefore, it must be fine.”
The same goes for plagiarism detectors. In my view there’s no substitute for human scrutiny in the final analysis.
Thanks for the great comment.
You are absolutely right on the caveat. I thought about mentioning that editors should actually read papers and look at the figures very meticulously, but then I thought “don’t they do this already?” Maybe only to varying degrees? My view would be that journals should have an automated screen for figures issues and plagiarism followed up by editorial review by good old fashioned human eyes/brains.
I agree with you in general, but I don’t agree with your point #2 and #3 in this specific case. The authors’ claim on Oct4 expression is not entirely based on a fluorescence tag. Oct4 expression was also confirmed by immuno and qPCR. The authors were aware of a pitfall of autofluorescence. Plus, at the end, what really matters is not whether or not the treated cells express a marker, but is rather if they gain pluripotency (, for which the authors provided several pieces of evidence). I understand that overall integrity of this research came into question due to the revelation after publication that data (and, possibly, samples) were handled very badly and immaturely, but if you just judge their study based on the two papers, their story was complete and each claim of the authors was backed by several methodologies. They are a Nature Article and a Letter after all.
I think green glow is still their #1 focus. And now one author claims to have made STAP 200 times and there are indications that multiple authors may refer to cells meeting only the “glowing green” criterion as “STAP” without the other important follow up…not to mention all the other chaos associated with these papers. Yeah, being a Nature Article and Letter still means something more generally, but in this case, not so much.
See this: http://www.japantimes.co.jp/news/2014/04/14/national/stap-replication-not-conclusive/#.U1BTjSj5SG8
“Obokata issued a statement through her lawyer saying she “produced STAP cells almost every day and often a few times in one day.”
That sure sounds like no more than green glow mojo”
Yes. I know she made a big deal out of green-glowing cells, but that’s not the main finding of the two papers. If it were, it wouldn’t have landed in Nature. So you think a certain treatment drives an endogenous gene in both mRNA (qPCR) and the protein levels (immunocyto), but not an exogenous marker with the same promoter (Oct4-GFP)? I don’t say that’s entirely impossible, but that sounds rather a strange and inconsistent interpretation to me… I prefer assays on endogenous molecules. A marker is a marker and at the end of the day, you need to back it with other methods. Green glow is rather a superficial evidence of pluripotency.
What kind of cells do you think contaminating? ES cells? Let me rephrase what Sasai said: Acid-treated cells are different from ES cells in various aspects, such as light-microscopic morphology and ultrastructure, ability to form aggregates, ability to contribute to the placenta in chimera. Contamination would make a perfect sense if the cells in question exhibited behaviors similar to presumably contaminating cells, but the authors findings are quite the opposite.
It’s a little bit sad that all of the lessons listed here are essentially ‘science 101’ lessons– or possibly even ‘science 1a’ lessons.
We all (that is university trained scientists) learned to keep proper notebooks and label our stuff, to critically examine results, and too look at science rather than the scientists. We shouldn’t need a couple of shoddy Nature papers and a promising career on the brink of collapse to remind us of these lessons.