Tea Leaves: Predicting Risk and Benefit in Translation

Every early phase trial begins with a series of predictions: that a new drug will show clinical utility down to road, that risks to study volunteers will be manageable, and perhaps, that patients in trials will benefit. Make a bad prediction here, and people potentially get hurt and resources wasted. So how good a job do we do with these predictions?

Hard to know, but given the high rate of failure in clinical translation, there are grounds for believing that various stakeholders go into early phase trials with an excess of optimism. In the current issue of PLoS Medicine, Alex London and I posit two problems with the way decision-makers make predictions in early phase trials. First, they underattend frequent and systematic flaws in the preclinical evidence base. Secondly, they draw on an overly narrow evidence base (what we call "evidential conservatism") that obscures an assessment of whether preclinical studies in a given research area are a reliable indicator of agent promise.

As an open access journal, readers are invited to view our article here. The article has garnered a decent amount of press- digestible summaries can also be found at the Scientist and Pittsburgh Gazette. Also check out a commentary commissioned by the journal editors. (photo credit: canopic 2010)

Conditions of Collaboration: Protecting the Integrity of the Scientific Enterprise

So what does it take to keep medical research a well-oiled enterprise that efficiently and effectively delivers cures? Lots of cooperation–or so I argue, along with co-authors Alex John London and Marina Emborg in a piece appearing in Science [a publicly accessible version of the essay is available at Science Progress]. Unfortunately, we argue, the way or system of drug development currently thinks about the ethics of clinical research does not presently place sufficient emphasis on the conditions necessary to sustain this cooperation.

Right now, oversight of clinical research is focused almost exclusively on protecting the personal interests of human subjects by obtaining valid informed consent and ensuring that risks are reasonable in relation to benefits. We suggest that this ostensibly private transaction between investigators and patient-volunteers has a public dimension in at least three ways. First, such private transactions inevitably draw on public resources. Second, such transactions have externalities- adverse events occurring on one trial have potential to disrupt collaborations elsewhere in the research system. Third, lax oversight of such private transactions creates conditions where consumers have difficulty identifying (and hence rewarding) producers of high quality goods (namely, trials that are well designed).

We suggest that, when considering whether to initiate highly innovative clinical trials that draw on such public goods, proper oversight and analysis must take into consideration factors that lie beyond the personal interests of human volunteers. (photo credit: McKillaboy, Cataglyphis velox 22, 2009)

Filing Cabinet Syndrome: The Effect of Nonpublication of Preclinical Research

Much has already been said about Filing Cabinet syndrome in medical research: the tendency of researchers to publish exciting results from clinical trials, and to stash null or negative findings safely away from public view in a filing cabinet. Nonpublication distorts the medical literature, because it prevents medical practitioners from accessing negative information about drugs. Recall that, back in 2004, attorney-general Eliot Spitzer sued Glaxo Smithkline for suppressing trial results that showed elevated risk of suicide for adolescents taking the antidepressant drug Paxil; this and several similar episodes led FDA, major medical journals, World Health Organization, World Medical Association, and others to require researchers to register clinical trials before they enroll any patients.

Yet important gaps remain. In the March 2010 issue of PLoS Biology, Emily S. Sena and coauthors provide the most detailed analysis yet of one of these gaps: nonpublication of preclinical (animal) studies. They aggregated results of 16 systematic reviews of preclinical studies involving acute ischaemic stroke, and used statistical methods to estimate the degree of publication bias, and the likely effect of publication bias on measured disease responses. Among other things, they found that 16% of animal experiments were not published, leading to a 31% overstatement of efficacy. The authors note: "we estimate that for the interventions described here, experiments involving some 3,600 animals have remained unpublished. We consider this practice to be unethical."

The authors urge that central registries of preclinical studies be established and maintained-- a call that is not likely to go heeded anytime soon by companies that have much at stake in the secrecy in preclinical research. But their proposal ought to be taken seriously by anyone committed not only to respecting animals used in medical research, but also protecting the welfare of human beings who might enroll in possibly unwarranted clinical research. (photo credit: amy allcock 2009)

Mice- Three Different Ones: Towards More Robust Preclinical Experiments

One of the most exciting and intellectually compelling talks thus far at the American Society of Gene Therapy meeting was Pedro Lowenstein's.  A preclinical researcher who works on gene transfer approaches to brain malignancies (among other things), Lowenstein asked the question: why do so many gene transfer interventions that look promising in the laboratory fail during clinical testing? His answer: preclinical studies lack "robustness."

In short,  first-in-human trials are typically launched on the basis of a pivotal laboratory study showing statistically significant differences between treatment and control arms. In addition to decrying the "p-value" fetish- in which researchers, journal editors, and granting agencies view "statistical significance" as having magical qualities- Lowenstein also urged preclinical researchers to test the "nuances" and "robustness" of their systems before moving into human studies.

He provided numerous provocative examples where a single preclinical study showed very impressive, "significant" effects on treating cancer in mice. When the identical intervention was tried with seemingly small variations (e.g. different mouse strains used, different gene promotors tried, etc.), the "significant effects" vanished.  In short, Lowenstein's answer to the question of why so many human trials fail to recapitulate major effects seen in laboratory studies is: we aren't designing and reviewing preclinical studies properly. Anyone (is there one?) who has followed this blog knows: I completely agree. This is an ethical issue in scientific clothing. (photo credit: Rick Eh, 2008)

STAIRing at Method in Preclinical Studies

Medical research, we all know, is highly prone to bias. Researchers are, after all, human in their tendencies to mix desire with assessment. So too are trial participants. Since the late 1950s, epidemiologists have introduced a number of practices to clinical research designed to reduce or eliminate sources of bias, including randomization of patients, masking (or "blinding") of volunteers and physician-investigators, and statistical analysis.

In past entries, I have rallied for extending such methodological rigor to preclinical research. This has three defenses. First, phase 1 human trials predicated on weak preclinical evidence are insufficiently valuable to justify their execution. Second, methodologically weak preclinical research is an abuse of animals. Third, publication of methodologically weak studies is a form of "publication pollution."

Two recent publications underscore the need for greater rigor in preclinical studies. The first is a paper in the journal Stroke (published online August 14, 2008; also reprinted in Journal of Cerebral Blood Flow and Metabolism). Many of the paper's authors have doggedly pursued the cause of preclinical methodological rigor in stroke research by publishing a series of meta-analyses of preclinical studies in stroke. In this article, Malcolm Macleod and co-authors outline eight practices that journal editors and referees should look for in reviewing preclinical studies. Many are urged by STAIR (Stroke Therapy Academic Industry Roundtable)– a consortium organized in 1999 to strengthen the quality of stroke research.

Their recommendations are:

1- Animals (precise species, strain, and details should be provided)

2- Sample-size calculation

3- Inclusion and exclusion criteria for animals

4- Randomization of animals

5- Allocation concealment

6- Reporting of animals concealed from analysis

7- Masked outcome assessment

8- Reporting interest conflicts and funding

There's an interesting, implicit claim in this paper: journal editors and referees partly bear the blame for poor methodological quality in preclinical research. In my next post, I will turn to a related news article about preclinical studies in Amyotrophic Lateral Sclerosis. (photo credit: 4BlueEyes, 2006)