One of the most important elements of science is critical reading of papers. Most of us come into science as undergrads feeling somewhat naive about what we read in papers. Our default tendency is to believe most or all that we read as “true”.
As we get more experienced, we realize that in fact if anything it is the opposite. We have to view papers very skeptically.
And as we get wiser we realize the weaknesses present in even the best papers and how to spot papers that are weak overall relatively quickly. We realize the weaknesses in our own papers before we submit them too. We start reviewing papers for journals, which is very educational as we get to see what the other reviewers wrote as well. In short, we become critical readers of science literature; a key element of being a scientist who contributes to the community.
I believe it is more challenging for those outside of science to critically read papers without this kind of experience and exposure to so many papers. Over my 22 years in science, I’ve read hundreds of papers at least, I reviewed more than 100 I would guess, and I’ve published about 55 of my own papers. I hope I’ve learned from this process and from my mentors and colleagues.
A debate in the stem cell field is how ready adult stem cell therapies are for use in patients. For Multiple Sclerosis (MS), there is some promising pre-clinical animal data suggesting that some day stem cells may be proven safe and effective as treatments for some patients. I would argue that that day is not today and is unlikely to be a day next year or even the next. It’s going to take time.
An MS patient, SammyJo, who received treatment at Celltex and who was interviewed on NPR about her experience, took issue (voiced in a comment on my recent blog post) with my statement that such treatments are not proven safe and effective. She listed 6 papers to argue that in fact such treatments are safe and effective.
Below I’ve listed the 6 papers included with her comments and my own opinions of each paper. My general take is that these are very poor, weak papers. They all have serious shortcomings if one is to take them as supporting the use of such stem cells in people.
SammyJo’s comments are indented & italicized.
My specific responses are in bold. I also present an overall take on this at the very bottom.
“The reality is that the treatments discussed are not scientifically validated as safe or effected.” – Comment from Paul Knoepfler on NPR story.
I welcome input from a stem cell researcher on this. Here are 6 papers I reviewed that made me feel comfortable pursuing this course of treatment, for my untreatable MS. The review papers each cover further studies. Please point our why this science is lacking, for someone in my condition to rely upon:
1. “Mesenchymal stem cells as treatment for MS – progress to date” An exhaustive amount of preclinical data has shown that the intravenous administration of mesenchymal stem cells (MSC) effectively ameliorates experimental autoimmune encephalomyelitis (EAE), a model of multiple sclerosis (MS), through the release of anti-inflammatory and neuro protective molecules. Based on these results, several small pilot clinical trials in subjects with advanced MS have demonstrated that MSC administration is safe and provided an early signal of clinical effectiveness. - Multiple Sclerosis Journal http://msj.sagepub.com/content/early/2012/11/01/1352458512464686?elq=d5772cd1683c44a69ca3361a5408027e
Paul: This paper talks about mouse work mainly using an interesting, but imperfect mouse model of MS. The notably “small” human clinical trials presented provided what the authors claimed was an “early signal” of effectiveness, which is a weak statement if one were to use this paper to support human transplants. I would also quote the abstract (emphasis mine):
The current aim of clinicians and scientists interested in the development of MSC-based strategies for the treatment of MS is to have the ultimate demonstration in large clinical trials that MSC can inhibit CNS inflammation and foster tissue repair as realized clinically, with functional recovery, or visualized by magnetic resonance imaging (MRI).
The goal overall, they admit, is to have large clinical trials that prove effectiveness. Celltex began treating patients without FDA approval and without an IND or clinical trial context. Note that clinical trials also should not charge patients to participate.
2. “Safety of Cell Therapy with Mesenchymal Stromal Cells (SafeCell): A Systematic Review and Meta-Analysis of Clinical Trials” Based on the current clinical trials, MSC therapy appears safe. However, further larger scale controlled clinical trials with rigorous reporting of adverse events are required to further define the safety profile of MSCs. (Manoj et al 2012) http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3485008/ PMCID: MC3485008
Paul: Again, the authors admit that further larger scale controlled clinical trials are required before anyone can be sure on safety. Also the paper does report a whole range of adverse events in patients who get the transplants. The authors wisely noted major limitations of their study (emphasis mine):
“Our systematic review has several limitations. First, despite our comprehensive search strategy, there are a number of completed but unpublished industry sponsored studies and studies published in abstract form only that may alter the safety profile of MSCs. Second, we pooled adverse events across heterogeneous disease states. Given the limited number of clinical MSC studies, and the small sample sizes of each, it was important to pool data across trials to determine if any potential signals of harm existed. Previously, we have advocated this approach when individual trials are not adequately powered to detect potential harm. [57] However, we acknowledge that the occurrence, type, and severity of adverse events may vary significantly between different populations and according to different MSC characteristics (e.g. dose, type). The limited number of included RCTs precluded the conduct of these sensitivity analyses. Third, the majority of RCTs included in our analysis would be considered a high risk of bias. Although double blinding an MSC trial may be considered ethically unacceptable, it is difficult to justify the lack of concealment of the allocation of patients in many studies.”
3. Cleveland Clinic is several years into a human trial for MS using expanded bone marrow derived stem cells, and they have isolated a growth factor that is part of the anti inflammatory effect that offers such immediate symptom relief:
“Hepatocyte growth factor mediates mesenchymal stem cell–induced recovery in multiple sclerosis models.” http://www.ncbi.nlm.nih.gov/pubmed/22610068 PMID:22610068
“Recovery From Multiple Sclerosis By Growth Factor In Stem Cells” http://www.medicalnewstoday.com/articles/245816.php
Paul. All mouse experiments in these studies. They don’t address human safety at all. I don’t see how this study supports transplants into human patients, but as a stem cell scientist I find the data interesting.
The last 3 papers address the specific Celltex method:
4. Safety of Intravenous Infusion of Human Adipose Tissue-Derived Mesenchymal Stem Cells in Animals and Humans Dr JC Ra http://www.ncbi.nlm.nih.gov/pubmed?term=21303266
Paul: Mostly mouse studies. The few human studies are very weak, only following patients 3 months. Overall, a very unconvincing paper to support the idea that such treatments are safe to be used in human patients.
5. “Taking Stem Cells Beyond Discovery: A Milestone in the Reporting of Regulatory Requirements for Cell Therapy” Stem Cells and Development. August 2011, 20(8): 1295-1296. This article is by leaders in stem cell research at Duke, Tulane and LSU, refers to Dr JC Ra’s study ‘Safety of Intravenous Infusion of Human Adipose MSCs’, as a model for the demonstration that MSCs are safe, with praise for offering a model way to build regulatory frameworks. http://www.ncbi.nlm.nih.gov/pubmed?term=21510815
Paul: This paper is not really an independent paper. It is a comment on #4 paper above. No new data.
6. “Stem cell treatment for patients with autoimmune disease by systemic infusion of culture-expanded autologous adipose tissue derived mesenchymal stem cells.” Journal of Translational Medicine 2011, 9:181 Dr JC Ra, includes 5 case studies at the end. http://www.translational-medicine.com/content/9/1/181
Paul: Very small study, little data presented. Strangely most data discussed is not presented in the actual paper and no figures are shown. Just two small tables. No clear methods presented. Follow up only a few months.
Paul’s overall take on this: Taken together these six papers are, in my opinion, interesting, but from a clinical perspective they are so limited and weak that they do not give me any confidence in the safety or efficacy of adult stem cell treatments for MS patients today.
I realize and respect the fact that the drug therapies for MS today are highly imperfect and not only don’t work for some patients, but also in some cases make them sicker. I also recognize that there is risk from doing nothing as well and I understand the profound effects on quality of life due to MS. However, let’s not kid ourselves that the rationale supporting adult stem cell treatment for MS is strongly scientifically supported at this time. It just isn’t.
Some patients may chose to get the treatments anyway and I won’t judge them, but society generally, the FDA, and for-profit companies as well as the doctors and scientists at such companies have an ethical and importantly a legal responsibility to educate and protect patients.
Dr M.Chandrashekhar MD,
I did a blog post on understanding research even when you are not an expert in that field here http://www.ithinkwell.org/the-public-can-understand-research-trials-and-share-decision-making/ . There is an excellent free access article about placebo, ethics and trials in the post. Perhaps it will help. Additionally we did another on with videos on understanding placebo http://www.ithinkwell.org/placebo-nocebo-and-the-power-inside/.
I am looking forward to Paul’s expertise on the papers too. I think it will help all of us understand stem cells better. It is wonderful he has chosen to take the time do this and has a heart for patients.
I agree that Mouse model responses to Inflammation / Immunomodulation may be different from that of human response.
All the more reason, extensive Clinical Trials need to be carried out.
As regards Double Blinded , Randomised Trials isnt it ethically wrong to give Placebo therapy to a patient of MS ?
Jim, this is an important point. Not paying attention to species dependent responses has led to roadblocks and years of wasted research in many neuroscience fields because theory that was not applicable was carried over. I would be most interested in research that looks at inflammatory and immune response in depth and over time. I am hopeful that cells could bridge this gap and hope that those dedicated and expert in these fields will choose this as a research question
One of the critical things that has to be mentioned with the mouse studies is that they are very bad predictors for immune/inflammatory responses in man. A recent study in PNAS points out the differences at the molecular level. http://www.pnas.org/content/early/2013/02/07/1222878110.full.pdf+html?sid=f60b87ad-5592-455b-82e4-6168e858f4a3.
The N Y Times article on this is http://www.nytimes.com/2013/02/12/science/testing-of-some-deadly-diseases-on-mice-mislead-report-says.html?pagewanted=1&_r=1&
MS is a disease that has a strong immune component this being the rationale for the use of MSCs.
Paul, thanks for addressing the stem cell studies I posted that show safety. First let me reiterate that I made a decision to pursue stem cell therapy as a patient who has exhausted all approved MS therapies, I was progressing to full paraplegia and nursing home care, at age 48. There are no University stem cell trials I qualify for. Patients who are less progressed may choose to wait for that “ultimate demonstration in large clinical trials” that our own stem cells are safe and effective therapy for MS. But I ran out of time, and so far the treatment I had at Celltex has only improved my condition.
You criticized the study I posted, the SafeCell meta-analysis showing safety which had n=1,000. This is 100 times the number of patients in the two papers you point to as proof of danger, where 10 patients were treated and several had complications (Jonsson, et al, n=9 andAlderazi, et al, n=1). That’s not a scientific and reasoned comparison. The real issue is that sick patients need to be able to make their own informed decisions. For example, a sick patient with one year left to live needs to get as much data as possible to make a choice. I am pleased you are willing to discuss this data with us, and encourage you to help us by weighing the pros and cons of autologous cell therapies, not just bashing all stem cell clinics-the good and the bad.
Your perspective is valued and appreciated. Innovative treatments become, in my mind, more important as the severity of illness increases.
I truly hope that Celltex publishes the data from all you brave, wonderful patients to inform all of us about the safety and efficacy data. It will be extremely disappointing if they do not publish these findings.
The study you cite–I don’t see the N=1000?
Thanks, SammyJo!
Below is the n=1012 and safety conclusion from the SafeCell paper. I know that science follows a standard methodology from research, first the pre-clinical, animal, then human studies. Where are we at for autologous mecsenchymal studies, what # of studies and patients need to be tabulated before this therapy can go into use? Can an argument be made for expanded access, based on lack of any therapy for conditions like ALS and progressive MS?
Safety of Cell Therapy with Mesenchymal Stromal Cells (SafeCell): A Systematic Review and Meta-Analysis of Clinical Trials
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0047559
” A total of 1012 participants with clinical conditions of ischemic stroke, Crohn’s disease, cardiomyopathy, myocardial infarction, graft versus host disease, and healthy volunteers were included.”
Conclusions
“Our study provides a systematic examination for adverse events related to the use of MSCs. We did not identify any significant safety signals other than transient fever. Results from our systematic review should provide some assurance to investigators and health regulators that, with the present evidence, this innovative therapy appears safe.”
Thanks, SammyJo. I’m going to find time to carefully read this and I’ll respond. I already have some thoughts, but will put it all into one post next week.
Paul, we’ve been doing our homework. Published evidence so far in 2,154 patients treated with their own expanded mesenchymal stem cells indicates this therapy is safe. Please address the evidence.
http://www.patientsforstemcells.org/how-safe-are-stem-cells/
Where did you get this 2,154 treated patients number? All treated with autologous laboratory expanded MSCs?
Paul, on the infographic http://www.patientsforstemcells.org/education/how-safe-are-stem-cells/ that lists 2,154 patients across 66 trials, it states the source as pubmed.gov and lists the search terms used.
I think a lot of this confusion could be obliterated if everyone chose to register and report all clinical trials. Alexey did an absolute must read blog on this and i blogged on this too http://www.ithinkwell.org/stem-cell-science-and-gsk-urge-trials-reporting/ To register a clinical trial the paper has to be readable by people other than PhDs with multiple years of experience. If you read the successful papers they are complex science that is stated simply and powerfully. Many registries have a mandatory consumer language section. You are right Paul that it is difficult to read some of the papers but you have taken very complex papers and explained them beautifully in the past. Maybe it would be good to share what would make these papers meaningful so that when the public reads them they will know what to search for in a paper. I think this could be very empowering and even change the practice of the stem cell industry.
Right now people are left with the dismay that they have chosen a paper carefully and feel the betrayal of it not measuring up. This can’t feel good. Is there a way we could build some bridges to understanding….it is really wonderful that you have taken the time to explain these papers but what can they do to help themselves read the next papers more clearly?
Correction: I meant “to provide the patient with information regarding what we do know and we do not”, risks and alternatives if there are.
Thanks for the comment, Nathan. Many people now view the “old days” of assisted reproduction as like the Wild Wild West. Scientists basically did whatever they pleased with human gametes and embryos. I personally do not believe that the fertilized egg or early embryo is an actual human being, but I still believe that it is a unique entity and should have be respected. I think many of the IVF researchers were out of control personally and did (and perhaps still do) unethical stuff. Thus, I don’t see the history of IVF as particularly helpful to use as a comparison to the current ethical and practical debates about innovative stem cell treatments.
In terms of who gets to decide, I believe that patients should make their own medical decisions, but not alone without help from others like doctors, scientists, etc. That doesn’t mean that anything goes too. It’s not that simple. For example some doctors, especially those collecting say $30,000 from the patient or who are being pressured by investors, can lead patients down the wrong paths with misinformation. Patients can also be told by clinics to look at certain papers and as Jenny indicated, it can be near impossible to tell whether a paper is good or a bunch of crap or somewhere inbetween. Sometimes it’s not even that easy for us scientists and can take a long time to really figure it out.
In the end in my opinion it is all about balancing risk and benefit along with factoring in ethics. Not so black and white with lots of gray in there to make us all frustrated.
The author is indeed experienced scientist and his approach and review of SummyJo’s references are completely aligned with scientific standards. THe problem as I see it is in the conclusion. My core experience is in reproductive field and I was there in late 80-th early 90-th, when we started to propose infertility treatments as experimental treatments to general population. I remember the voices calling to stop it until safety will be proven and double blind controlled study will be published. We dared to fertilize the human oocyte by single sperm cells injection, bypassing the natural pathway of fertilization process, we froze embryos and gametes, interfere in embryo development in vitro, learned and constantly improved the safety and efficacy. All these procedures are part of standard care practice, yet we still follow up the children born as result of these procedure. Some of procedures (embryo byopsy, preimplantation genetic testings) till now are performed as experimental under patient consent. So, if we would had to follow the similar principle of 100% proven safety and efficacy, hundred thousands of couple would have remained childless till today. The actual regulations of reproductive field came up more than 30 years after first in vitro baby and still elaborating. So, back to the question “who get to decide”, the answer is clear: patient. The responsibility of scientists and physicians is to provide patient honestly and openly with information about what we do now and what we don’t. Patient should have a right to take decision, particularly in modern time of internet and unlimited access to information. Autologous medicine has many different aspects, and would be wrong to try and fit it into standard definitions, developed for different purposes.
To say it’s challenging for non-scientists to read and review such papers is an understatement. Can you give a run-through of the kind of questions you might ask in such a case. I’ve been looking at Shinya Yamanaka and Rongxiang Xu, a pair of foreign stem cell researchers, and outside of Xu’s patents [http://bit.ly/12qxESa] it’s not easy to guage one’s work against the next.
Good read and thanks, Paul. Educating us non-science folk one at a time.
Hi Alex,
Thanks for the feedback.
Great question too. When evaluating research, it is important to look at the journal that it is published in. For example, Yamanaka is published in some of the top journals such as Cell, Nature, Cell Stem Cell. Of course, it’s not necessarily an easy matter for patients to know which journals are top notch and which are very weak. There’s also a middle ground of great journals that are not necessarily in the top 10, but still publish important research.
When looking at a specific article itself, you should try to figure out how many patients were involved, how long were they followed (months is usually too short to make strong conclusions), is it from a company. You can always ask a scientist too, although I admit not so many are quite as responsive.
I am in full agreement with Alex. This is confusing. Most patients looking into cell therapy would not have access to someone who would interpret a medical study for them. Even if someone could understand all the medical language, a lay person such as myself, would have no idea about the strength/weakness of ANY study I was reading. We just assume if it’s a medical study it must be reliable. Why is it all so confusing? Oh, and if you ever have any question on cooking, canning, crochet, organic gardening or raising chickens, don’t hesitate to call on me! 😉 Thank you Paul for being one of the scientists that is easily accessible for answering patient question!
Even more reason to peruse a model like this for development and creation of new data standards with all parties present at the table http://c-path.org
What are your thoughts on C-Path? No option patients patients need pulled out of the “Valley of Death”
Hi Jenny, I’m going to look into it. It’s not something I am familiar with.
Once again, thank you for your fairness and compassion towards the patients who truly have no options. I respect your work and view as a scientist and appreciate your acknowledgment that MS patients face very limited options. You have definitely brought more awareness about the limited MS treatment options and we appreciate that!
As you stated you would not judge a patient for their choice as we cannot judge you for your view. It is truly about bridging the gap and hearing all sides. We may not like the answers we get, but we are all entitled to our own opinion. We are also accountable for our decisions when facing a health battle. It is not an easy position to be in. We can only hope one day in the future this will all be a mute point as better treatments are made available.
Thanks, Tracy. I really do care deeply about patients and ultimately my goal in the stem cell arena (I also work on cancer) is to get as many people helped by stem cells as possible.