doi: 10.1161/CIRCULATIONAHA.105.572826
(Circulation. 2005;112:2222-2225.)
© 2005 American Heart Association, Inc.
Editorial |
Anatomy of an Emerging Diagnostic Test
Computed Tomographic Coronary Angiography
From the University of Vermont, Burlington.
Correspondence to Timothy F. Christian, MD, Prof of Medicine and Radiology, University of Vermont, Baird 191, 111 Colchester Ave, Burlington, VT 05401. E-mail timothy.christian{at}uvm.edu
Key Words: Editorials • coronary disease • diagnosis • imaging • imaging
 |
Introduction |
|---|
 Top Introduction SegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
He that will divide a minute into a thousand parts and break but a part of the thousandth part of a minute in the affairs of love, it may be said of him that Cupid hath clapped him o’ the shoulder, but I’ll warrant him heart-whole.— —Shakespeare, As You Like It
The wheel is come full-circle.— —Shakespeare, King Lear
The study by Mollet et al on CT coronary angiography in this issue of Circulation is important for several reasons.1 The 64-multislice technology for acquiring tomographic x-rays of the coronary vessels during the arterial phase of a bolus of iodinated contrast introduces several technical advances: improved spatial resolution (thinner slices), improved temporal resolution (less time needed to acquire an image means less time for the vessel to move), and improved spatial coverage of the heart (reduction of registration errors). Unlike previous studies, the authors attempted to analyze the entire length of all of the major arteries, not just those in which image quality was optimal. All patients had the definitive end point comparison to invasive angiography, which was not prompted by the results of the CT examination. The authors report sensitivity for the detection of significant obstructive disease of 99% and a specificity of 95%, confirming a slight propensity of CT to generate false-positive lesions in the presence of vessel wall calcification. Although the results were excellent, it is clear that this study and others have struggled with the question as to how to best evaluate CT angiography.
Article p 2318
New tests face validation challenges that are both common to all emerging technologies and unique to the specific test. CT coronary angiography is a purely anatomical (at least at present) result of a structure that is variable between individuals. It is also noninvasive, therefore, it cannot be compared with a direct pathological analysis clinically. These conditions give rise to 4 issues that have a direct impact on the evaluation and use of this technology: division of a complex structure into inflexible units, evaluation of what cannot be seen, referral bias, and efficacy and the effectiveness of a test.
|
Segmentation |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
Cardiac imagers have invoked the concept of the segment to provide boundaries within the heart where none exist. They are an imaginary convenience with a price. There are 2 issues that are introduced when segmental analysis is invoked: sample size and interdependence.
The achievement of significance for conventional statistical tests is a function of the number of observations. As an example, consider a study for the detection of viability using 99mTc sestamibi perfusion imaging at rest in an animal model of myocardial infarction, in which the pathological infarct size is known in 6 animals. A linear correlation of 6 data points is unlikely to be significant in the absence of a close agreement between the studies. By segmenting the myocardium into the standard American College of Cardiology/American Heart Association myocardial model,2 the number of data points is expanded from 6 to 102 for the group. Now, even modest correlations are significant; however, the global amount of scar tissue (the truth) is unaltered, as is the difference between the measures.
Segmental paring does not impart independence. The segments from a coronary CT scan are linked. They were acquired with a certain degree of motion, with a given attenuation from body habitus, with the same contrast bolus timing and dosage, and with a specific coronary anatomy. It is conceivable that in a cohort of patients in whom the prevalence of disease is low, all of the abnormal (diseased) segments originated from a handful of patients, and all of the scans had excellent image quality. Conventional statistical tests ignore the interdependence of segments that originate from the same patient. More rigorous statistics are necessary to assess the impact of linked segments on the outcome of the comparison of the tests.3 Mollet et al performed a subanalysis of a random single segment from each patient to determine whether the accuracy was altered (single bootstrap), but they did not statistically adjust for interdependence.
Analysis of individual coronary segments can be misleading. For simplicity, consider 20 patients undergoing coronary CT in which the coronary vasculature is divided into a 17-segment model. A total of 10 segments are reported as abnormal. The number of patients with disease by noninvasive testing can then be anywhere from 1 to 10 (5% to 50% depending on the "nesting" distribution). Independent segmental analysis provides no information on the distribution of disease in the cohort. Segmentation separates the coronary tree from the heart as a whole.
|
Impact of Missing Segments |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
Earlier papers comparing CT and invasive angiography limited their analyses to segments that could be visualized by CT. This has important consequences when considering test accuracy. Unlike a test for the predilection to develop male pattern baldness, a missed coronary lesion has the potential for fatal consequences. Sensitivity must be high (false-negative tests must be minimal). Using the 17-segment model again, consider 17 patients undergoing CT angiography with a reported sensitivity of 100% in 272 of 289 visualized segments. It is immediately apparent that the missing 17 segments could contain lesions and may be distributed heterogeneously in the 17 patients. Taken to the extreme, all 17 patients may have a single missed obstructive lesion.
Missing segments are not reported, but 3% of the extant segments were "poorly" visualized.1 Poor visualization, despite the multiplication of detectors, will be a reality with this technique in certain patients and may be hard to predict in advance (eg, motion during the acquisition, poor bolus of contrast, premature beats) and the impact harder to deal with. Stress echocardiography may have lung interference and nuclear perfusion imaging may have bowel interference precluding the visualization of certain segments of myocardium. They generate other variables such as exercise duration, ECG response, exercise-induced chest pain, LV function, transient LV dilatation, and lung tracer uptake, which can be factored into the final impression and are strong prognosticators. CT angiography receives no such help at present.
What are the options for dealing with missed segments of anatomy? (1) Assume the missing segments are normal. (2) The test is invalidated because another test is now necessary to rule out disease (assume missing segments are positive). (3) The probability of disease in missing segments should be extrapolated from the prevalence in the other segments. (4) The probability of disease should be extrapolated from the condition of the other vessels. (5) The probability of disease should be estimated from the clinical risk factors and other available data.
None of these alternatives are entirely satisfactory. One may be tempted to make one of the above choices based on the location of the missing segment, but the prediction of the amount of myocardium supplied distal to a lesion is not overly accurate.4 Ultimately, clinicians will need to incorporate several of the above strategies into their decision algorithm.
|
Verification Bias |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
The evaluation of noninvasive diagnostic modalities is hampered by the presence of referral bias.5–8 After an initial learning curve, clinicians develop a confidence in the test in question and refer only patients with a positive test to the definitive end point (coronary angiography in this case). Negative results are largely trusted and no further testing for such patients occurs. It becomes difficult to get a handle on the false-negative rate of the test, but false positives are readily exposed. This results in an inflation of the sensitivity of the test and a dramatic apparent reduction in the specificity. Consequently, the only time a test can be accurately judged is early in its development. This period is likely to be brief for CT angiography. Clinicians are used to visually evaluating coronary anatomy.
What are the implications for referral bias on CT coronary angiography? The present study by Mollett et al has made a solid effort to avoid referral bias by including only patients already referred to angiography on the basis of other tests and historical variables. Selection bias is present as evidenced by the high prevalence of coronary disease in the cohort (75% of the group), and a large proportion of these with multivessel disease. It is unlikely that such a prevalence of disease will be present in the population that CT coronary angiography will be applied to clinically. By Baye’s theorem, the performance of a test is a function of both its inherent accuracy and the prevalence of disease in the population to which it is applied. It is unlikely that CT angiography will perform near the levels reported here in a screening population, where disease prevalence is low and true positives hard to find.
Another sequela of verification bias will be the difficulty in comparing CT angiography to clinical variables or other noninvasive imaging tests for coronary artery disease detection when the invasive angiogram is the end point. Papers on this subject will almost certainly appear in the literature, which creates a problem: Assume all patients had a stress echocardiogram study in this cohort of 51 patients and the decision to perform angiography was based on that test result. Because only positive tests were likely to be referred and the test performs fairly well (75% sensitivity in this hypothetical situation), and nothing of the negative tests is known, only true positives and false positives are measured, with a high resultant disease prevalence. Consequently, the sensitivity of stress echocardiography will be somewhat inflated and the specificity low (0% in this case). The CT angiogram was not used to decide to do angiography and will be more accurate because the specificity will not be artifactually lowered (ie, all of the 4 possible outcomes will be known). The same issue applies to clinical risk variables, which were used to select the study group end point. When comparing 2 methods in which the established test was used to select which patients went to the definitive end point, the new test will almost always appear to be more accurate.
|
Efficacy and Effectiveness |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
Efficacy studies are done under controlled circumstances and reflect the potential accuracy of the test. This is different from effectiveness studies, which reflect the performance of the test when applied in an uncontrolled clinical setting. In the study by Mollet et al,1 70 patients had a definitive end point to compare the CT angiography results. Patients with arrhythmias, renal insufficiency, and contrast allergies were excluded as were the results of one patient with a technically inadequate scan. The authors commendably outline those excluded for the CT-specific reasons listed above, which are made up of 10 patients (9 patients had scheduling difficulties). All 10 could have been done by stress echocardiography or SPECT perfusion and 5 could have been done by MRI. Consequently, the effectiveness ratio was 51/61 patients, or 84% of the study population. Because other noninvasive techniques have their own specific set of exclusion criteria, it would be helpful to report this value when considering the usefulness of all diagnostic tests.
Accuracy measures should be adjusted for technical failures. Both contrast and radiation were received and no vessels visualized in the case of one patient in this study. Given the high prevalence of disease in the cohort, this could be viewed as a false-negative examination because no lesions were seen. The issue must be decided by a second test or invasive angiography. In the present study, this has little impact, bringing the sensitivity down to 97% on a "per patient" basis (38 true-positive tests/38 true positive+1 false negative). It can be a much bigger factor for technically demanding methods in less-experienced hands.
There are 3 questions to be asked of all noninvasive studies from an effective reference point: How does the prevalence of disease in this study translate to clinical practice? What proportion of patients in clinical practice can the test be applied to? How are accuracy measures adjusted for technical failures and other missing data?
|
Conclusions |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
Can CT be fairly evaluated? The present study is an excellent start. Although it is difficult to completely avoid verification bias, it can be recognized and adjusted for.6–8 By conducting both efficacy and effectiveness studies in appropriate populations for measures of accuracy, clinicians can better judge how the test will perform in their practice. Missing data must be accounted for in the overall accuracy of the test. Finally, we need to move away from segment analysis and become "heart whole" and classify patients into multi-, single-, or no-vessel disease categories. Perhaps more relevant, we should adopt a standard to divide patients into high- and low-risk groups even in the presence of disease, based on previous prognostic angiographic data.9,10
There are issues for using CT angiography as a diagnostic tool. Other noninvasive myocardial imaging tests (SPECT, stress echo) have consistently shown that the overall extent of abnormality is the single most important predictor of cardiac-related death, and that such test results correlate with more extensive anatomical disease. These tests were incorporated into clinical practice, not only to detect atherosclerosis indirectly but also to provide important physiological information about its impact. This has held up over time for both techniques in the form of powerful prognostic studies. Both nuclear stress imaging and stress echocardiography have shown incremental prognostic value in patients when the coronary anatomy is known.11–15 These methods now form a strong symbiotic relationship with coronary anatomy, which can guide therapy. This is lost when CT is used in isolation.
Is a noninvasive anatomic picture what we want? For a time, invasive anatomy was the only imaging tool we had. Our concept of atherosclerotic disease has changed a lot since then. We know that a majority of myocardial infarctions occur because of plaque composition and structure rather than luminal obstruction.16 There is a difference in the atherosclerotic disease of diabetic patients that we are struggling to define.17 At times the angiogram can be misleading when judging the impact that a lesion imparts on the coronary circulation.18 It can be hard to estimate the amount of myocardium at risk for a specific vessel lesion4 and critical to know the viability of that zone. Finally, we know the impact of inflammation and other molecular mechanisms in the development of ischemic syndromes.19 To turn away from all that and go back to simple anatomy as our guide would be a step backward. This technology is remarkable and we should demand more from it. The noninvasive angiogram is an important step for CT, but it should not be the last stop. The wheel may come full circle, but it can also move forward.
|
Acknowledgments |
|---|
Disclosure
Dr Christian is the recipient of an American Heart Association Grant-in-Aid for high-field imaging of quantitative MRI perfusion.
|
Footnotes |
|---|
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
|
References |
|---|
|
TopIntroductionSegmentationImpact of Missing SegmentsVerification BiasEfficacy and EffectivenessConclusionsReferences |
|---|
1. Mollet NR, Cademartin F, Van Meigham CAG, Runza G, McFadden EP, Baks T, Serruys PW, Krestin GP, de Feyter PJ. High-resolution spiral computed tomography coronary angiography in patients referred for diagnostic conventional coronary angiography. Circulation. 2005: 112: 2318–2323.
2. Cerqueira MD, Weissman NJ, Dilsizian V, Jacobs AK, Kaul S, Laskey WK, Pennell DJ, Rumberger JA, Ryan T, Verani MS. Standardized myocardial segmentation and nomenclature for tomographic imaging of the heart. A statement for healthcare professionals from the Cardiac Imaging Committee of the Council on Clinical Cardiology of the American Heart Association. Circulation. 2002; 105: 539–542.
3. Sternberg MR, Hagdu A. A GEE approach to estimating sensitivity and specificity coverage properties of the confidence intervals. Stat Med. 2001; 20: 1529–1539.[CrossRef][Medline] [Order article via Infotrieve]
4. Huber KC, Bresnahan JF, Bresnahan DR, Pellikka PA, Behrenbeck T, Gibbons RJ. Measurement of myocardium at risk by technetium-99m sestamibi: correlation with coronary angiography. J Am Coll Cardiol. 1992; 19: 67–73.[Abstract]
5. Rozanski A, Diamond GA, Berman D, Forrester JS, Morris D, Swan HJ. The declining specificity of exercise radionuclide ventriculography. N Engl J Med. 1983; 309: 518–522.[Abstract]
6. Heijenbrok-Kal MH, Hunink MM. Adjusting for bias in diagnostic reports. Am J Med. 2002; 112: 322–324.[CrossRef][Medline] [Order article via Infotrieve]
7. Diamond GA, Rozanski A, Forrester JS, Morris D, Pollock BH, Staniloff HM, Berman DS, Swan HJ. A model for assessing the sensitivity and specificity of tests subject to selection bias. Application to exercise radionuclide ventriculography for the diagnosis of coronary artery disease. J Chronic Dis. 1986; 39: 343–355.[CrossRef][Medline] [Order article via Infotrieve]
8. Miller TD, Hodge DO, Christian TF, Milavetz JJ, Bailey KR, Gibbons RJ. Effects of adjustment for referral bias on the sensitivity and specificity of single photon emission computed tomography for the diagnosis of coronary artery disease. Am J Med. 2002; 112: 290–297.[CrossRef][Medline] [Order article via Infotrieve]
9. Califf RM, Phillips HR III, Hindman MC, Mark DB, Lee KL, Behar VS, Johnson RA, Pryor DB, Rosati RA, Wagner GS, et al. Prognostic value of a coronary artery jeopardy score. J Am Coll Cardiol. 1985; 5: 1055–1063.[Abstract]
10. Emond M, Mock MB, Davis KB, Fisher LD, Holmes DR Jr, Chaitman BR, Kaiser GC, Alderman E, Killip T III. Long-term survival of medically treated patients in the Coronary Artery Surgery Study (CASS) Registry. Circulation. 1994; 90: 2645–2657.
11. Borges-Neto S, Shaw LK, Tuttle RH, Alexander JH, Smith WT IV, Chambless M, Coleman RE, Harrington RA, Califf RM. Incremental prognostic power of single-photon emission computed tomographic myocardial perfusion imaging in patients with known or suspected coronary artery disease. Am J Cardiol. 2005; 95: 182–188.[CrossRef][Medline] [Order article via Infotrieve]
12. Marie PY, Danchin N, Durand JF, Feldmann L, Grentzinger A, Olivier P, Karcher G, Juilliere Y, Virion JM, Beurrier D, Cherrier F, Bertrand A. Long-term prediction of major ischemic events by exercise thallium-201 single-photon emission computed tomography. Incremental prognostic value compared with clinical, exercise testing, catheterization and radionuclide angiographic data. J Am Coll Cardiol. 1995; 26: 879–886.[Abstract]
13. Pollock SG, Abbott RD, Boucher CA, Beller GA, Kaul S. Independent and incremental prognostic value of tests performed in hierarchical order to evaluate patients with suspected coronary artery disease. Validation of models based on these tests. Circulation. 1992; 85: 237–248.
14. Ciaroni S, Bloch A, Hoffman JL, Bettoni M, Fournet D. Prognostic value of dobutamine echocardiography in patients with intermediate coronary lesions at angiography. Echocardiography. 2002; 19: 549–553.[CrossRef][Medline] [Order article via Infotrieve]
15. Rizello V, Polderman D, Schinkel AF, Biagini E, Elhendy A, Sozzi FB, Maat A, Crea F, Roelandt JR, Bax JJ. Long-term prognostic value of myocardial viability and ischemia during dobutamine stress echocardiography in patients with ischemic cardiomyopathy undergoing coronary revascularization. Heart. April 6, 2005. DOI: 10.1136/hrt.2004.055798. Available at: http://heart.bmjjournals.com.
16. Burke AP, Farb A, Malcom GT, Liang YH, Smialek J, Virmani R. Coronary risk factors and plaque morphology in men with coronary disease who died suddenly. N Engl J Med. 1997; 336: 1276–1282.
17. Mehran R, Dangas GD, Kobayashi Y, Lansky AJ, Mintz GS, Aymong ED, Fahy M, Moses JW, Stone GW, Leon MB. Short- and long-term results after multivessel stenting in diabetic patients. J Am Coll Cardiol. 2004; 43: 1348–1354.
18. Herrmann SC, El-Shafei A, Kern M. Currnet concepts in coronary physiology for the interventionalist. Int J Cardiovasc Intervent. 2003; 5: 109–131.[CrossRef][Medline] [Order article via Infotrieve]
19. Ridker PM, Rifai N, Rose L, Buring JE, Cook NR. Comparison of C-reactive protein and low-density lipoprotein cholesterol levels in the prediction of first cardiovascular events. N Engl J Med. 2002; 347: 1557–1565.
This article has been cited by other articles:
 |
|
 |
|
  T. F. Christian, S. P. Bell, L. Whitesell, and M. Jerosch-Herold Accuracy of Cardiac Magnetic Resonance of Absolute Myocardial Blood Flow With a High-Field System: Comparison With Conventional Field Strength J. Am. Coll. Cardiol. Img., September 1, 2009; 2(9): 1103 - 1110. [Abstract] [Full Text] [PDF]
|
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||