The Absence of an Ideal Observer
Why Some Clinical Trials May Not Be What We Think They Are
According to Richard Firth Green, a major change occurred in our conceptualization of the truth during the reign of Richard II of England in the late 14th century.1 Before then, “trouthe” was an ethical concept that resided within individuals; afterward, truth became an objective reality that existed outside ourselves.1,2 If so, who could be trusted with identifying the truth? Green proposed the “ideal observer”; ie, we can know that “x is better than y” if this judgment were made by an observer who was “fully informed and vividly imaginative, impartial, in a calm frame of mind and otherwise normal.”3 It is interesting to note that Green never claimed that any ideal observers actually existed.
The ideal observer theory is appealing. We believe that investigators can describe an external reality that exists independently of their ethical compasses. We assume that stringent experimental conditions can reveal an unbiased truth. But does the scientific method always yield valid results? Before we answer, we should ask: when we execute a large-scale clinical trial, is there an ideal observer who is impartial, knowledgeable, rational and calm?
A large clinical trial typically involves a leadership committee, a sponsor, numerous geographically dispersed investigators, and a group responsible for operational functions. The leadership committee helps to define the trial hypotheses and the methods by which they are tested; however, its members personally make no observations and may not know exactly how observations are made. The sponsor invests substantial sums of money, without assurance that the hypothesis is valid and generally without direct involvement in the collection of data. The investigators are paid to recruit patients, but they may have little vested in the overall results of the trial. The operations group ensures that patients are recruited rapidly, knowing that it will be difficult to confirm whether patients were recruited appropriately or trial procedures were followed faithfully. Among these trial components, who is the ideal, fully informed and impartial observer? Could the parties (each acting in self-interest but within regulatory standards) collectively yield a flawed product? Here are a few hypothetical possibilities.
A pharmaceutical executive or academic leader wonders whether to propose a clinical trial for funding. The existing data supporting the drug are marginal, but if its development is not pursued, the leader’s position and the jobs of many colleagues are at risk. The team makes the best possible argument for investment, emphasizing data that are encouraging and minimizing difficulties that are likely to occur.
The trial is funded, and an internal operations group proposes a reasonable (but costly) process of oversight. Instead, the company seeks an outside vendor to execute the trial at minimal expense. It receives proposals that are patently inadequate to support even basic services; yet, a procurement office, disconnected from the study, reviews the bids solely based on cost.
The outside vendor must generate a profit for its owners, stakeholders, or sponsoring institutions. Accordingly, the expenditures (already inadequate) are reduced further to make the project financially worthwhile. Investigators are sought who recruit quickly and inexpensively, knowing that, after the trial’s completion, their individual efforts will hardly be recognized or criticized.
The site investigators must support themselves, their staff, and often their home institutions. Office procedures are structured to deliver a reliable flow of consenting patients; some personnel may be incentivized for recruitment. The site investigators justify the aggressive enrollment of patients with borderline eligibility, because they know trial patients receive close medical follow-up, regardless of their randomized treatment. The research team understands that its procedures will be only lightly scrutinized.
When the trial is completed, the sponsor and the leadership committee rarely find the results extremely satisfying; ie, the trial’s hypotheses are achieved in a robust manner, and the data are of very high quality. Much more commonly, the results disappoint. The primary end points are not met or the effect size is less than expected; positive signals emerge only after a creative search; new safety concerns arise; or the data quality is poor. Given all that has been invested, there is reluctance to admit failure; some propose a new path forward, knowing most will be skeptical, but hoping that their leadership roles can continue if someone can be persuaded to maintain a financial commitment.
The results of the trial are presented, and scavengers emerge from their hiding places to feast. Each has his/her own agenda, determined by their own personal biases or interests. Although external critics play a critical role in the digestion of new data, much of the commentary is self-serving. Some insist that the trial design (public and uncriticized for years) was always known to be flawed. Many advise that the results not be believed until they are reviewed by regulators (who often have no access to raw data) or until the database is made public (even though most would not know what to do with it if it were made available). If the trial disappoints, many will claim to have predicted its failure.
The results of the trial are published, but who reads the report? Ask physicians if they keep up with the medical literature, and they respond with bewilderment. Who has the time to read or the knowledge to understand the primary publication? Has the standard of care changed since the inception of the trial? If the experimental design is novel or confusing, is it because it was intelligent or innovative or because it was tailored to satisfy the needs of regulators rather than test a hypothesis important to clinicians? It is easy to simply wait for the chatter on the Internet, social media, or (years later) the official guidelines, but are these opinions truly informed and unbiased?
Under the rare circumstances where the trial shows unambiguous evidence of benefit, those responsible for paying for the new drug are unhappy. The drug will be overpriced, in part, because monies are needed to pay not only for its development, but also to subsidize the sponsor’s unsuccessful projects. Payers look for loopholes in the data, create artificial reasons for excluding patients from access to the therapy, and require cumbersome forms and processes for preauthorization, all to create barriers to prescribing the new treatment.
These descriptions are hypothetical, but how confident are we that they do not happen or take place all at the same time? If these were to occur, how would one know the truth? Who among these self-interested parties is the ideal observer? Which part of our disaggregated approach to conducting clinical trials and acting on their results is unbiased and clear headed?
We live in an era marked by stated devotion to high-quality evidence, but do we always deliver it? Is it always possible to sort through the self-interests of the involved parties to arrive at a true scientific conclusion? When you read about the results of the next trial, ask yourself: who was the ideal observer in the experiment? If you struggle to answer the question, you might feel a kinship with our brethren who lived before the 14th century. For them, truth was a deeply ethical concept, analogous to our ideas of integrity and dependability, that resided within individuals.2 To resolve our current difficulties, we must realize (even in the 21st century) that the original notion of truth still has validity.
Dr Packer has been the principal investigator of many international large-scale clinical trials, but most of these were completed and presented years ago. Within the last three years, Dr Packer has consulted for Admittance, Amgen, AstraZeneca, Bayer, BioControl, Boehringer Ingelheim, Cardiorentis, CardioKinetix, Celyad, Daiichi Sankyo, Ferring, Novartis, Relypsa, Sanofi, Takeda, and ZS Pharma.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
Circulation is available at http://circ.ahajournals.org.
- © 2017 American Heart Association, Inc.
- Green RF
- Fowler E
- Brandt RB