Was Einstein Wrong About Insanity?

Was Einstein wrong on his definition of insanity?

“Insanity”, said Einstein, “is doing the same thing over and over again and expecting different results.”

But in science, how sane is it to expect experiments to give precisely the same results?

How big a problem is irreproducibility in science?

Experiments on that question have given variable results. Haha.

But seriously, why is it that so much of what is published in science is so difficult to reproduce? While Einstein (pictured above in an image from Wikipedia) may have had less trouble with irreproducibility in Physics, the biomedical sciences–particularly in pre-clinical and clinical studies–is highly variable and often data are not reproducible.

Most biomedical scientists agree that this is a vexing issue and in the last week even NIH was reported by Nature to be considering taking action on the issue, possibly via new requirements for grantees with awards that are clinical or pre-clinical in nature.

Some biological scientists might flip Einstein’s quote around:

Insanity in Science is Doing the Same Thing Over & Over Again and Expecting the Same Results

Most scientists have had the experience of experimental irreproducibility.

On a micro level it occurs with our own experiments at times. On Monday we do qPCR. On Wednesday we repeat the exact same qPCR to the best of our knowledge doing it precisely the same, but we get different results.

Why? I’m not sure we can know, but often it seems to be the result of how complicated biology can be and variables that we just don’t perceive.

On a macro scale, we may try to repeat a published experiment and we get very different results.

Why?

Sometimes the cause cannot be pinned down. Other times I think it is due to scientists publishing their data too fast, egged on by the quest to be first and get into high impact journals so they simply do not repeat their experiments enough times before publishing.

Reproducibility is central to all science because our goal is to test hypotheses and try to come to solid conclusions based on data. Solid conclusions should be ones that other scientists can confirm.

How big is this irreproducibility problem? It seems to be huge as we look at another section of the Nature article:

In a 2011 internal survey, pharmaceutical firm Bayer HealthCare of Leverkusen, Germany, was unable to validate the relevant preclinical research for almost two-thirds of 67 in-house projects. Then, in 2012, scientists at Amgen, a drug company based in Thousand Oaks, California, reported their failure to replicate 89% of the findings from 53 landmark cancer papers. And in a study published in May, more than half of the respondents to a survey at the MD Anderson Cancer Center in Houston, Texas, reported failing at least once in attempts at reproducing published data (see ‘Make believe‘).

Those are big numbers.

Another part of the problem surely is that the emphasis amongst scientists themselves, editors, funding agencies, and others is on doing new stuff, not reproducing published results.

Do not even propose trying to repeat or publish things that others have done or your grant proposal or paper will be rejected. One of the big criterion for NIH grants is Innovation after all.

Is repeating Ted’s experiment published last year going to be high on a study section’s priority list? No! You must say you are doing something very different and better than what Ted did.

This obsession with perceived novelty results in many published studies standing alone without ever having been recapitulated.

Of course if you are deeply interested in the same topic as another scientist, say Ann, then you may choose to try to repeat her experiment anyway and the experience of science over the years predicts that often you will get very different data than Ann.

What does that mean?

Most of the time it doesn’t mean anything bad about you or Ann.

Rather it is the nature of science that there is a lot of noise in most systems. Molecular machinery and cells themselves in biological systems in particular are often inherently variable with major stochastic elements.

Sometimes the key conclusions of papers are correct and the results in a general sense are reproducible.

Still I believe that science needs to also reward validation studies, not just what some view as sexy new stuff. Science also needs to reward studies that are themselves rock solid technically speaking and have done a rigorous job on replicating their own studies and used robust statistical evaluation.

At the same time we need to be mature and realistic about the fact that not every paper or project will produce binary, black and white results. Too often reviewers expect all data to march lock step together to produce one uniform conclusion and they attack a manuscript that includes subtleties, what I call the “grays” of science.

It may not be flat out insane to expect perfectly reproducible results and our goal should be reproducibility, but on the other hand let’s be mature and accepting of the fact that science and biology in particular in the real world often produce variable results.

We should do our best to minimize that variability by doing the most rigorous science that we can and keep expectations high, but at the same time a certain realism is healthy on an individual and field-wide basis. It should also motivate us to use extra caution before jumping into clinical studies where there is likely to be even more variability due to each person being so different.