Carotid Stents: Unleashed, Unproven
Primarily on the basis of data derived from the Stenting and Angioplasty With Protection in Patients at High Risk for Endarterectomy (SAPPHIRE) trial,1 the US Food and Drug Administration (FDA) has approved the use of carotid stents (CASs) in high-risk patients. The SAPPHIRE trial was published and much heralded as a randomized trial demonstrating that CASs were not inferior to carotid endarterectomy (CEA). Yet, the more recent Endarterectomy Versus Angioplasty in Patients with Symptomatic Carotid Stenosis randomized trial of CASs compared with CEA had to be stopped because the stroke rate with stents was so high that it triggered the safety guidelines of the study design.2 How can we explain the striking difference in outcome between these 2 studies, and how did it happen that the FDA was so convinced of the quality and validity of the SAPPHIRE trial that it granted approval for CASs? An examination of the SAPPHIRE trial—its conduct, data collection and analysis, the circumstances of publication, the presentation to the FDA Advisory Panel, and its consequent approval—is the primary focus of this article. This is a case study of the flaws in our system for the evaluation and approval of medical devices that warrant serious reflection on our ability to properly create and act on accurate information and live up to our commitment to evidence-based decision making.
Response by Samuelson et al p 1601
As it now stands, existing studies leave us with the unfortunate but not unreasonable conclusion that no scientific basis exists for the use of CASs as approved by the FDA, and in the absence of change, there is every reason to doubt the capability of our current system to protect the public from unnecessary risk in the future. Although this article focuses on just 1 example of how our systems are flawed at multiple levels to provide a reliable assessment of CASs and other technology, readers seeking further examples can find a wealth of related information.3–5
The SAPPHIRE Trial
The SAPPHIRE trial was originally designed as a randomized trial involving 29 centers comparing the outcome of CEA with carotid stenting. The exact details of patient selection and the rationale for the patient assignment within the trial to CEA or CAS are not completely clear from the published data. It seems that patients were seen by both a vascular surgeon and an interventionalist. If a patient was deemed by both to be suitable for either procedure, the patient was randomly assigned to CEA or CAS. If the vascular surgeon did not believe that CEA could be safely performed but the interventionalist thought CAS was appropriate, the patient was not randomized but was assigned to a stent registry. Likewise, when the opposite was true, the patient was assigned to a surgery registry. This is, in essence, an opinion-based entry criterion rather than a protocol-based criterion. A total of 2294 patients were referred for evaluation, of whom 747 met the criteria for inclusion in the study. However, only 334 of the eligible patients underwent randomization; 406 were entered into the stent registry, and 7 were entered into the surgical registry. This further process of exclusion represents a tremendous opportunity for bias. However, because no data are provided as to why, after meeting the protocol-defined inclusion criteria, patients were deemed acceptable to CAS but rejected for CEA, there is no way to assess the bias pattern. Ultimately, the assignment was based on opinion and was not necessarily predictable. For example, it is possible that a vascular surgeon, eager to gain experience and meet a numbers requirement for CAS credentialing, might turn down a patient for randomization to perform CAS. Such are the uncertainties of an opinion-based entry criterion.
The original intent of the study was to randomize 600 to 900 patients with a maximum sample size of 2900. The study was set up with certain statistical objectives, including the method of statistical analysis, the timing of interim analyses, the conditions for termination, and a final test to determine the primary end point. As the study unfolded, however, all of these statistical protocols were violated. The trial was terminated because of a decrease in enrollment; the interim analyses were not performed; and an alternative statistical method was used to determine the end point. After a lengthy description of the original study design, the authors state, “In early 2002, the pace of enrollment in the trial abruptly slowed, because several nonrandomized CAS registries had become available.” One such registry was their own, and they chose, as permitted in the SAPPHIRE design, to assign more of the study-qualified patients to their stent registry than to the randomized study. For these investigators, an incentive or requirement no longer existed to determine whether CAS was effective as soon as they had the opportunity to use CAS without such evidence. As an unfortunate consequence, their study was underpowered to answer the question it posed as to whether CAS was inferior to CEA. Despite this shortcoming, the study was published and formed the primary basis for FDA approval.
The primary end point of the trial was the cumulative incidence of death, stroke, or myocardial infarction within 30 days of the procedure or death or ipsilateral stroke between 31 days and 1 year. No significant statistical difference existed with regard to the primary end point at 30 days. At the end of 1 year, the authors, using a statistical analysis that was an alternative to the study design, reported a significant difference in the primary composite end point of 12.2% for CAS and 20.1% for CEA (P<0.05). This difference, in favor of CAS, is highly attributable to the differing incidence of non–Q-wave myocardial infarctions, as determined by a 2-fold elevation in creatinine kinase with a positive MB fraction. No other criteria were required for the diagnosis of myocardial infarction. For this reason, the significance of the diagnosis cannot be compared with other studies of perioperative myocardial infarction, usually using the World Health Organization criteria.6,7 The inclusion of non–Q-wave infarction as an end point equivalent to death or stroke is highly controversial. The authors justified this decision by stating that a perioperative non–Q-wave infarction confirms an increase in the risk of myocardial infarction by a factor of 27 in the subsequent 6 months. That may be true; however, for some reason, that concern was not validated by their own data in this study because no subsequent Q-wave infarctions occurred in the CEA group between 31 days and 1 year despite the higher incidence of “enzyme” infarctions. Another reason for the higher incidence of enzyme infarctions may be that the assays were done more often in the CEA group than in the CAS group because patients were studied every 24 hours while in the hospital and the CEA group on average was in the hospital 1 day longer than the CAS group. Unless the number and timing of enzyme studies were identical between the 2 groups, the data are not valid. Thus, for a number of reasons, the significance of myocardial infarction and the differences between groups represent data that cannot be relied on for any conclusion. The key issues here are stroke and death. However, the study was underpowered to determine noninferiority in outcomes under the conditions defined in the protocol, as was eventually noted by the FDA.
Missing from the SAPPHIRE data is any reference to local or systemic complications of stent placement and administration of intense antiplatelet therapy. Is it possible that no such complications occurred? To put this question in perspective, at least 1 similar registry of CASs exists in which poststent cerebral hemorrhage in a patient on antiplatelet therapy is not counted as a stroke but as “other neurological.”8 This may or not have been the case in the SAPPHIRE study, but to have no anticoagulation-related local or systemic problems in a study this large and involving major arterial stent placement via percutaneous femoral artery puncture is distinctly unusual and at least warrants some comment or verification. Also missing from SAPPHIRE is any effort to identify “silent” cerebral ischemia, an outcome at least as relevant as silent myocardial ischemia. Subsequent studies have shown silent infarcts in ≈40% of patients even when a protective filter is used.9,10 These are hard-copy, irrefutable data that warrant attention. Are these infarcts truly silent? Should we look for cognitive or emotional effects? Does embolization continue after the procedure?
The SAPPHIRE study design also was unique among carotid studies in that no tracking of outcomes based on degree of stenosis took place. What was it, and was it equal in the randomized arms? If the randomization were truly applied to all eligible patients, it would be extremely unlikely that 20% of the patients in the randomized trial had recurrent stenosis as an indication. At the primary institution, the Cleveland Clinic, the reported volume of CEA between 1989 and 1995 was 371 per year.11 During the overlapping interval, 1989 to 1999, the volume of redo CEA was 20 per year, or 5.4%,12 a number that is consistent with other high-volume centers.13 Credibility is strained to say that this fraction is now 20%, and this discrepancy adds fuel to the argument that the opinion-based entry criteria are scientifically invalid. The reason is that recurrent stenosis is a smooth, fibrotic lesion that many believe is highly suited to CAS, with less risk of embolization during the procedure. A study that includes a large number of restenotic lesions may be favorable to CAS. Thus, we are saddled here with the appearance of bias. That bias would have been eliminated with protocol-based entry criteria.
These data were presented to the FDA at a meeting of the Circulatory System Devices Advisory Panel on April 21, 2004.14 The response of the FDA staff reviewers warrants praise and is encouraging. Heng Li, PhD, the FDA statistical reviewer, summarized the randomized trial statistical analysis. The recorded minutes of Dr Li’s findings stated, “Dr. Li provided a detailed explanation of the planned statistical methodology and the ‘stopping’ rule it incorporated. In his analysis, the evidence would not have indicated that the trial should have been stopped (and non-inferiority declared), if the original protocol had been followed.”
The minutes continued: “The sponsor made unplanned comparisons between the stent registry and the CEA arm of the randomized study. Because the patient characteristics in the two groups by definition are different, a straightforward comparison is not appropriate. To address this issue, the sponsor used a propensity score method to compare the two groups, attempting to make a post-hoc claim of non-inferiority of the stent registry to the randomized CEA arm.” In other words, because the data from the biased-but-randomized arm were insufficient to make the case for CAS, data from the nonrandomized arm were pulled in to an alternative statistical analysis.
Dr Li continued, “In summary, the original group sequential protocol was not followed, and the FDA was not informed of any change in protocol. Any non-inferiority claim based on the sponsor’s post-hoc propensity score analysis is problematic.”
Ronald Weintraub, MD, FDA consultant, reviewed the methodology of the SAPPHIRE pivotal study. He stated, “The sponsor’s study findings are limited because the pre-specified enrollment plan and study analysis was not carried to completion in the SAPPHIRE randomized study.”
Lisa Cannel, FDA lead reviewer, commented, “The sponsor terminated the pivotal trial early, citing too many competing studies, physicians’ reluctance to randomize, and surgeons’ unwillingness to refer patients. The competing studies involved CORDIS’s own devices and were facilitated by CORDIS.” Thus, the sponsor (CORDIS) effectively undermined its own study and placed the responsibility on its own investigators.
Despite these concerns, including the misgivings of the FDA statistician, the FDA panel of clinician experts voted 6 to 5 in favor of CASs as a legitimate alternative to CEA in both symptomatic and asymptomatic patients who fall within a broad definition of high risk. With the shortcomings of the study and the complete absence of study design or data comparing symptomatic and asymptomatic patients as identified by the FDA staff, how did this all come about? How does a study with these shortcomings warrant publication in a major medical journal, and how, in spite of this, did the FDA panel approve the device? Such a challenge to logic and science requires that we examine the FDA approval process.
At the FDA, the data were reviewed by an advisory panel consisting of 11 voting members: 6 cardiologists, 2 interventional radiologists, 2 vascular surgeons, and 1 neurologist. All of these people submitted disclosures. Six of the voting members were acknowledged to have current or past interests in firms at issue. These interests were waived because they were in matters not related to the specific agenda. In other words, these panelists had existing interests in the corporate sponsor, but those interests were with products or devices other than CASs. Others have examined the complexities that arise under these circumstances.3,4 It is notable that the FDA itself, under the Prescription Drug User Free Act, is funded by the corporate sponsor. Others have posed this question: Who then is their client, the corporation or the public?15,16
The lead author of the report was an inventor of the Angioguard embolic protection device used in the SAPPHIRE Trial and was a founder of the Angioguard Corp, purchased by Johnson & Johnson in 1999, for $40 million.17 In fact, of the 15 authors of the SAPPHIRE Trial, 10 acknowledged support in one form or another from the CORDIS Corp; in addition, 2 of the authors were employees of the CORDIS Corp. This level of disclosure of relationships with the corporate sponsor was unusually high but does not prove that any effective influence was exerted on the authors. The disclosure standard set by the National Institutes of Health and the Association of Medical Colleges was met.18 The underlying philosophy here seems to be that once the disclosure standard is met, readers can form their own opinions. Unfortunately, however, readers cannot expunge from the literature studies such as SAPPHIRE that do not follow scientific standards and, once published, can be used to create public policy. In this case, the clinical science would have to be of the highest standard to overcome the burden of competing interests; in this case, as will be shown, the science was far from that.
The specific role of the lead author is a little different. The recommendations of a recent roundtable on this very subject, including editors of the American Heart Journal and The Lancet,19 concluded: “The inventor must not be involved with the clinical studies and must be excluded from enrolling patients, analyzing the data, and writing the manuscript.” The publisher of SAPPHIRE, however, was not responsible for approval of the inventor as the principal investigator or lead author. That, in fact, was primarily the responsibility of the author’s institution and its internal review processes. Nonetheless, the publisher has the admittedly complex task of examining all aspects of a clinical study to ensure the quality of information through which (in this case) public policy was influenced. Why dwell on these concerns about compromised science leading to public policy? Because once a product is unleashed in the medical marketplace, it is extremely difficult to reverse that decision. An entire industry, along with jobs and political influence, has developed around carotid stenting. Before FDA approval, not much was at stake, but now a monetary force has been created. Once created, the industry interest can be protected and maintained by a moving target of slightly new devices, slightly different technology, and new target populations, all of which make it extremely difficult to force the removal of a product from the marketplace.
Another CAS trial that played a role in the FDA approval process is the ACCULINK for Revascularization of Carotids in High-Risk Patients (ARCHeR) trial.19a This trial is self-described as a series of 3 sequential, multicenter, nonrandomized, prospective studies. Thus, this was not even an attempt at a controlled trial. Instead, the authors used a historical control for carotid surgery when they estimated the combined adverse event rate to be an extraordinary 14.4%. Without going further, these facts alone eliminate this as a trial of sufficient scientific quality to justify a change in public policy or to bring a device to the marketplace.
Two recently published European trials favor CEA and raise serious questions about the safety of CAS. The Endarterectomy Versus Angioplasty in Patients with Symptomatic Carotid Stenosis trial was, in fact, a protocol-based randomized comparison of CAS with CEA, an essential scientific criterion not met by either of the above-mentioned trials. The trial was stopped prematurely by its safety committee for reasons of safety and futility. The rate of any stroke or death was 9.6% for CAS compared with 3.9% for CEA, with a relative risk of 2.5 (95% CI, 1.2 to 5.1; P<0.01). The flaws in this study included the limited carotid stenting experience requirements of the interventionalists. In addition, embolic protection devices were not used in the early phase of the trial. Nonetheless, the experimental design and conduct of this study were at a scientific standard well above the US studies. The Stent-Supported Percutaneous Angioplasty of the Carotid Artery Versus Endarterectomy (SPACE) trial20 was a multinational, multicenter, randomized controlled trial of CAS versus CEA with an outcome of any stroke or death of 7.68% versus 6.51% and a relative risk of 1.19 (95% confidence interval, 0.75 to 1.92). These 30-day outcome results were interpreted to fail to confirm noninferiority of CAS; longer-term results are pending. The study was stopped prematurely for funding reasons, and many would fairly argue that, at this point, it is inconclusive. However, it is also reasonable to ask whether approval would have been granted if data from these trials had been presented to the FDA.
As it stands today, the FDA has approved the use of CASs for symptomatic patients with >50% stenosis and for asymptomatic patients with >80% stenosis who are also “high risk.” As noted, no valid data are available on which to justify the use of stents in symptomatic patients from either the SAPPHIRE or ARCHeR trial. For asymptomatic patients, it is easy to suggest that a group of patients exists who are at such high risk for surgery that CAS is justified for stroke prevention. However, the immediate question then is whether such frail patients are better off with no intervention and modern drug management with platelet inhibitors and statins. CAS is not innocuous and has its own risk factors for periprocedural hemodynamic complications, stroke, and death.21,22 CAS also is associated with a significantly higher complication rate when a contralateral carotid occlusion is present, age exceeds 80 years, or ulceration of the carotid plaque is present.23 These are the very same risk factors used to define high risk for surgery. Radiation-induced carotid stenosis has been considered a lesion that is better treated with CAS, but recent data demonstrate an extraordinary restenosis rate of 80%.24 What then are the valid, protocol definitions of high risk that make it safer to do CAS than CEA? Currently, the Centers for Medicare and Medicaid Services is considering establishing reimbursement for CAS in high-risk asymptomatic patients and, instead of using a protocol, is proposing the following criterion: “that the determination of high-risk for CEA be performed by a surgeon credentialed to perform CEA.”25 This would be an embodiment of one of the primary flaws of SAPPHIRE, that is, an opinion-based criterion by a person who often can perform both CEA and CAS or may have other obligations that influence the decision. If bone fide medical criteria exist that define high risk pertaining only to surgery, it should be possible to specify them. If we do not have that information, we should perform the scientifically designed studies necessary. For the moment, no such criteria have been defined. The statement that CAS provides the opportunity for stroke prevention for patients who are too high a risk for CEA has no foundation; in fact, under these circumstances, there is reason to be concerned that CAS is harmful compared with medical therapy alone.
The bottom line here is that we need well-conducted, scientifically designed randomized trials to get answers about CASs. SAPPHIRE represents a failed opportunity. The only existing randomized trial in this country is the Carotid Revascularization Endarterectomy Versus Stent Trial (CREST),26 a National Institutes of Health–sponsored trial that began long before SAPPHIRE but is moving comparatively slowly now that the FDA has approved CAS and CAS registries. Regrettably, we have an organ of government, the National Institutes of Health, that does not have support from another, the FDA. It is, in essence, science versus commerce and a major flaw in our system. If the FDA had refused to allow any stent registries until CREST was completed, we would now have an extensive body of valid information of which we could be proud and on which safe public policy could be based. Instead, we have settled for unscientific studies designed to win expeditious FDA approval. As a result, CAS was approved for use in symptomatic patients, a subgroup for which we have virtually no data, and asymptomatic patients, a subgroup for which we have flawed data. This can be described as the commercialization of science. It is not what the public deserves from its faith in clinical scientists, its respect for prestigious journals and institutions, and its dependence on regulatory agencies.
In the end, no one can make a strong argument for CASs. To the contrary, the most scientifically valid data, the European studies, are unfavorable to CAS. Unfortunately, CASs, having FDA approval, are already in widespread use despite conflicting data about their safety. Going forward, physicians, editors, institutional review boards, governmental agencies, and readers must be more vigilant and critical of any commercialization of clinical research. The validity of our debate can be no better than the validity of the available data.
It should be noted that the biotechnology industry does play an essential and vital role in the advancement of medical care. As eager as industry might be to see a new proprietary device arrive successfully in the marketplace, no business entity can ultimately benefit from inaccurate information. Clinical scientists funded by industry must walk a difficult ethical line and maintain scientific rigor in the face of countervailing pressures. This balancing act protects both parties. In the case of the rush to FDA approval of CASs, the flawed data and the appearance of impropriety, whether it occurred or not, are such that a disinterested party would not agree that we have data on which public safety can depend. If we adhere to the scientific method and are committed to evidence-based medicine, we should cease debate until we complete a scientifically valid study such as CREST.
In the end, it may be that CASs are as effective as or more effective than CEA or noninterventional medical therapy. It is our role as clinician-investigators to fully use the resources available to us to design and implement sound clinical studies. Until we have done that, we have no basis for supporting the current use of CASs.
Mas JL, Chatellier G, Beyssen B, Branchereau A, Moulin T, Becquemin JP, Larrue V, Lievre M, Leys D, Bonneville JF, Watelet J, Pruvo JP, Albucher JF, Viguier A, Piquet P, Garnier P, Viader F, Touze E, Giroud M, Hosseini H, Pillet JC, Favrole P, Neau JP, Ducrocq X. Endarterectomy versus stenting in patients with symptomatic severe carotid stenosis. N Engl J Med. 2006; 355: 1726–1729.
Kassirer JP. On the Take: How Medicine’s Complicity With Big Business Can Endanger Your Health. New York, NY: Oxford University Press; 2005.
Angell M. The Truth About Drug Companies: How They Deceive Us and What to Do About It. New York, NY: Random House; 2004.
Hawthorne F. Inside the FDA: The Business and Politics Behind the Drugs We Take and the Food We Eat. Hoboken, NJ: John Wiley & Sons; 2005.
Guidant Endovascular Solutions. Clinical Update for Physicians: RX ACCULINK Carotid Stent System, PMA P040012. Santa Clara, Calif: Guidant Endovascular Solutions; March 20, 2006.
Maleux G, Demaerel P, Verbeken E, Daenens K, Heye S, Van Sonhoven F, Nevelsteen A, Wilms G. Cerebral ischemia after filter-protected carotid artery stenting is common and cannot be predicted by the presence of substantial amount of debris captured by the filter device. Am J Neuroradiol. 2006; 27: 1830–1833.
Mozes G, Sullivan TM, Torres-Russotto DR, Bower TC, Hoskin TL, Sampaio SM, Gloviczki P, Panneton JM, Noel AA, Cherry KJ. Carotid endarterectomy in SAPPHIRE-eligible high-risk patients: implications for selecting patients for carotid angioplasty and stenting. J Vasc Surg. 2004; 39: 958–965.
Summary minutes: Meeting of the Circulatory System Devices Advisory Panel, Open Session. April 21, 2004; Gaithersburg, Md.
How Independent Is the FDA? http://www.pbs.org/wgbh/pages/frontline/shows/ prescription/hazard/independent.html.
Conflict of interest ends in dismissal. Vascular News. September 1, 2006. 1–2.
Ohk T. Conflicts of interest and double standards. Endovasc Today. 2006; 5: 23–24.
Cayne NS, Faries PL, Trocciola SM, Saltzberg SS, Dayal RD, Clair D, Rockmand CB, Jacobowitz GR, Maldonado T, Adelman MA, Lamperello P, Riles TS, Kent KC. Carotid angioplasty and stent-induced bradycardia and hypotension: impact of prophylactic atrophine administration and prior carotid endarterectomy. J Vasc Surg. 2005; 41: 956–961.
Hofmann R, Niessner A, Kypta A, Steinwender C, Kammler J, Kerschner K, Grund M, Leisch F, Huber K. Risk score for peri-interventional complications of carotid artery stenting. Stroke. 2006; 37: 2557–2561.
Centers for Medicare and Medicaid Services. Proposed decision memo for percutaneous transluminal angioplasty (PTA) of the carotid artery concurrent with stenting (CAG-00085R3). Available at: http://www.cms.hhs.gov/mcd/viewdraftdecisionmemo.asp?id=194. Accessed March 1, 2006.
Hobson RW, Howard VJ, Brott TG, Howard G, Roubin GS, Ferguson RD. Organizing the Carotid Revascularization Endarterectomy Versus Stenting Trial (CREST): National Institutes of Health, Health Care Financing Administration, and industry funding. Curr Control Trials Cardiovasc Med. 2001; 2: 160–164.
Response to LoGerfo
L. Nelson Hopkins, MD
“Lies, damn lies, and statistics….” In today’s medical literature, it is possible to find “data” to support almost any position. In Endarterectomy Versus Angioplasty in Patients With Severe Symptomatic Carotid Stenosis (EVA-3S) and Stenting and Angioplasty With Protection in Patients at High Risk for Endarterectomy (SAPPHIRE), we have conflicting studies with significant flaws (not unlike most randomized studies). The unassailable fact is that carotid endarterectomy (CEA) is a good operation. However, many years’ experience and a body of literature demonstrate that many patients are at increased risk for CEA, as exemplified by the patients excluded from the North American Symptomatic Carotid Endarterectomy Trial (NASCET) and the Asymptomatic Carotid Atherosclerosis Study (ACAS) (identified as high-risk patients on the basis of the trial designers’ surgical experience) and the objective performance criteria (predicted risk of CEA for patients in high-risk carotid artery stenting [CAS] registries) developed by the Food and Drug Administration after rigorous literature review. As a surgeon who has performed CEA procedures over the last 30 years, who participated in NASCET, and who also has performed CAS procedures since 1994 (while participating in most Food and Drug Administration–sponsored trials), I find that nothing is more frustrating than facing a patient in whom the known risk for CEA is excessive and for whom no reimbursable alternative exists, a not uncommon scenario. In 2007, equipoise exists between CEA and CAS. The Cardiac Revascularization Endarterectomy Versus Stent Trial (CREST), like all the other CAS trials, will teach us much about CAS relative to CEA but will not eliminate one procedure or the other. Today, the important question is not “Which is better, CEA or CAS?” but rather “Which procedure is better for a given patient?” and “Are the risk factors excessive for revascularization (with either technique)?”—suggesting a role for best medical therapy.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.