(Circulation. 1996;94:3251-3256.)
© 1996 American Heart Association, Inc.
Articles |
the Department of Biomedical Engineering (K.A.P., E.L., J.F.C.) and Department of Biostatistics and Epidemiology (N.A.O.), Research Institute, and Department of Cardiology (S.E.N.), The Cleveland (Ohio) Clinic Foundation; the Biomedical Engineering Center, Ohio State University, Columbus (K.D.M., C.G.H.); and Shiley Heart Valve Research Center, Irvine, Calif (J.L.H.).
Correspondence to Kimerly A. Powell, PhD, Department of Biomedical Engineering, Wb3, The Cleveland Clinic Foundation, 9500 Euclid Ave, Cleveland, OH 44195. E-mail powell@bme.ri.ccf.org.
| Abstract |
|---|
|
|
|---|
Methods and Results Twenty-two sets (12 intact valves, 10 SLS valves) of cineangiographic images were obtained from individuals whose valve status was subsequently verified ex vivo. Several quantitative measures of SLS were evaluated to identify possible loss of metal or gaps in the SLS legs. Two of these measures, decrease in pixel intensity (DIPI) ratio and gap half-width, are diagnostic metrics of SLS: ie, the maximum likelihood estimate of the area under the receiver operating characteristic curve was 0.892 (SD, 0.066) for a model based on DIPI ratio and 0.802 (SD, 0.093) for gap half-width.
Conclusions We have developed a postprocessing technique that can be used to objectively evaluate outlet strut integrity in cineradiographic images of BSCC heart valves. At an estimated specificity of 1.0, the estimated sensitivity of the objective review was comparable to that of a subjective expert review panel.
Key Words: imaging valves angiography radiography
| Introduction |
|---|
|
|
|---|
The valve is made up of a circular flange and two struts that retain a tilting-disk occluder. The inlet strut is an integral part of the valve assembly, whereas the smaller outlet strut is welded to the outer flange. OSF typically occurs near the weld where the outlet strut meets the flange. When both legs of the outlet strut fracture from the flange, the disk occluder escapes and the valve becomes completely incompetent. It is postulated that the cause of OSF is excessive bending stresses imposed at the weld by the tilting-disk occluder.2
An estimated 86 000 BSCC valves were implanted in patients worldwide; of these, 45 000 are implanted in patients living today. As of January 31, 1995, 564 OSFs have been reported to the Shiley Heart Valve Research Center. Approximately two thirds of those reported cases resulted in the patient's death. Because of the morbidity and mortality associated with reoperative valve replacement, prophylactic valve replacement has not generally been recommended.3 4
Three lines of evidence suggest that one leg of the outflow strut separates at some time before the second leg breaks: (1) 7 of 24 electively explanted valves were reported to have an SLS5 ; (2) completely fractured clinical valves show a rough surface, indicating a fresh fracture on one leg, and a wear-polished surface on the other leg, indicating an earlier fracture2 ; and (3) 11 valves with artificially and clinically induced SLSs remained functional in sheep for up to 14 months (Shiley Heart Valve Research Center, unpublished data, 1993). In an effort to identify those valves that may have one leg separated from the flange, a radiographic imaging technique is being developed that can be used to evaluate the integrity of the outlet strut legs in vivo.6 7 This technique was first implemented in a 315-patient cohort study performed at WBH in Royal Oak, Mich.8 Patient selection for the WBH study required patients with a mitral BSCC valve
29 mm in diameter having an estimated fracture risk of 0.46% per year.9 10 11 The study was conducted from August 1992 to August 1994. The patients were imaged at least twice during the course of the study. The cineangiographic images were subjectively evaluated by the WBH investigators and an external panel of six expert reviewers. The status of each valve was graded on a scale of 0 to 5. The subjective review criteria were as follows: grade 1, apparently normal valve; grade 2, minimally suspicious (appearance of SLS in one or two frames in one view only); grade 3, suspicious (appearance of SLS in several frames of one view only or one or two frames in two views); grade 4, probable SLS (appearance of SLS in multiple frames of one view or several frames in two views); and grade 5, definite SLS (appearance of SLS in several frames of two or more views). A grade of 0 was assigned to suboptimal examinations, defined as those in which there was poor image contrast or the strut weld was obstructed from view. Interobserver and intraobserver variability and reviewer bias are inherent problems in any subjective review process. Therefore, the goal of our study was to develop objective quantitative measures of outlet strut integrity from the radiographic images of BSCC valves by postprocessing techniques.
| Methods |
|---|
|
|
|---|
3 to 4 seconds of data was obtained for each view. The maximum spatial resolution of the image acquisition system was 3.7 line pairs/mm (object size, 0.135 mm).
Secondary Digitization
The cineangiographic images were digitized with a Nikon LS-3510 scanner interfaced to a Macintosh Quadra 650. The dynamic range was 8 bits, and the image size was 800x800 pixels, covering a region of interest that included both outlet strut legs and their flange intercept. The effective pixel width was calculated by counting of the number of pixels across the strut leg (nstrut) and dividing it into the actual width of the strut leg, which was
1 mm in diameter (1 mm/nstrut). The mean effective pixel widths of the images for the first and second halves of the study were 0.040±0.007 and 0.029±0.005 mm, respectively. The images were digitized in the middle of the imaging sequence, and at least three complete cardiac cycles were obtained for each patient. Image frames in which the weld was completely obstructed by the annulus were excluded from the analysis. The images were then transferred to UNIX workstations for further processing and analysis.
Temporal Registration
There was a significant amount of valve motion in the image frames of the radiographic sequence. Therefore, a semiautomated preprocessing program was developed to reference each frame of temporal data spatially to an internal coordinate axis system defined by the centerline axis of the outlet strut leg and the internal curve of the outer annulus. First, the outlet strut leg was segmented from the surrounding background tissue by a statistical classification method based on Bayes' theorem.12 Two sample reference points were identified by the user. One sample population (n=255 pixels) was chosen approximately in the middle of the strut leg, and one (n=255) was chosen nearby in the background of the image. A one-sided hypothesis test at a level of significance of
=.005 was used to classify the pixels as strut or background. Because of the large variation in background across each image, the classification was performed separately for each strut leg in the tunnel views. Only the "leg of interest" was processed for each oblique view. For the internal oblique views, this was the leg nearest the crossover point at which the internal face of the annulus "crosses" over to the external face of the annulus. For the external oblique views, this was the longer of the two struts. (The other leg was not processed in the oblique views because in most cases the strut weld was obstructed from view by the radiodense annular ring.) Second, the outlines of the strut leg and the flange were determined from the segmented image. The central axis of the strut leg was determined by least-squares fitting of lines to either side of the strut leg outline and calculation of the position halfway between these two fitted lines. A second-degree polynomial was least-squares fitted to the inside curve of the outer flange, excluding the outlet strut leg. The preprocessing program was developed in the "C" programming language and X/Motif (v 1.2) Windows system.
Quantitative Measures of SLS
On the basis of interpretation by the expert review panel, two main visual cues, gap and offset, appear to be used to identify SLS in radiographic images. Gap refers to the space observed when the outlet strut leg separates from the flange, ie, a decrease in metal density. Offset is used to describe the strut leg deviation from its central axis of insertion or offset from the base. Quantitative measures for both gap and offset were calculated for each frame of data in all cineangiographic sequences.
Gap
Three quantitative variables were used to describe the gap in the outlet strut legs of the valve: (1) the maximum DIPI observed near the strut leg intercept with the flange, (2) the width of the gap, and (3) the location of the gap on the strut leg. The maximum DIPI was calculated from the average intensity profile along the long axis of the strut leg. The average intensity profile was calculated by spatially averaging pixels across the width of the strut leg. The profile starts in the radiodense region of the outer annular ring, 1 mm beyond the inner annular curve, and extends toward the center of the valve. An example of an average intensity profile is shown in Fig 1
. A characteristic DIPI is observed in the average intensity profile of a fractured leg. This DIPI is observed immediately adjacent to normal intensity roll-off observed in the transition from the radiodense annular ring to the outlet strut leg. The DIPI ratio is measured as the maximum number of SDs the average intensity profile decreases from a mean pixel intensity measured in the unfractured portion of the strut leg (Fig 1
). The second derivative of the average intensity profile was used to locate the maximum DIPI. The search for the location of the maximum DIPI was restricted to 1.5 strut widths (1.5 mm) of the inner curve of the outer annular ring. The majority of OSFs occur in this region of the leg. The half-width of the gap was measured from the location of the maximum DIPI to the maximum gradient immediately after the DIPI. Finally, the location of the maximum DIPI was measured relative to the inner curve of the outer annular ring.
|
Offset
Offset was calculated from the geometric outline of the strut leg. The outline of the strut leg was obtained from a subimage of the strut leg that included a portion of the outer annular ring. The subimage was median-filtered (5x5 square kernel), and the edges of the strut leg were calculated by use of a Marr-filtered image (5x5 square,
=.589)13 and Kittler's automated segmentation.14 Each side of the strut outline was referenced to the centerline axis of the strut leg. A significant deviation in the outline, shown in Fig 2
, was determined in a manner similar to that described for the DIPI ratio. The maximum offset deviation was calculated as the number of SDs the strut outline deviated from a least-squares line fitted to the straight portion of the strut leg. The search for the maximum offset deviation was restricted to 3 mm from the flange roll-off. This differed from that of the gap measurement because of the oblique orientation of the outlet strut leg in these views. Visual inspection of the radiographic images indicated that offset is observed on either side of the strut leg. Therefore, the location and maximum offset deviation were recorded for both sides of the strut leg.
|
Mechanical Measurements
The width of the separation, its location, and the offset deviation were measured from the explanted valves in a static unloaded condition. Measurements were made both mechanically and from photographic prints of high-magnification radiographs (x25) of the valves imaged in air. The width of the separation was measured mechanically by insertion of metal foils of varying thickness between the separated surfaces. This measurement represents the minimum width as opposed to the mean width because of the uneven fracture, its unloaded state, and the limited thickness of the metal foil. The location of the separation was measured along the centerline axis of the strut leg relative to the inner curve of the outer annular ring both mechanically and from the high-magnification radiographs. The offset deviation was measured as the maximum displacement of the separated leg from the base of the strut.
Objective SLS Classification
In most cases, SLS is observed in a limited number of frames of a cineangiographic sequence. This could be due to the orientation of the valve in the image plane; for example, as the valve moves in and out of the image plane, the separation may be obstructed from view. It may also be that the size of the separation changes during the cardiac cycle and that it is large enough for visualization only at certain times within the cycle. The subjective grading scheme was based on the premise that SLS was observed in only a limited number of frames. For example, a greater level of suspicion was assigned to those valves in which SLS was observed in a greater number of frames. The same scheme was used in the objective classification technique. The greater the percentage of frames that exhibited a quantifiable characteristic, the greater the likelihood of strut separation. The measure of discrimination between SLS valves and intact valves was chosen as the percentage of frames that exceeded or fell below a threshold. The thresholds were determined independently for each quantitative variable. They were defined as the value at which the expected false-positive (intact legs classified as SLS) and false-negative (SLS legs classified as intact) error rates were minimum but equivalent. The mean value for gap half-width and location was determined over all the frames of each patient in each view (the intact legs were omitted for SLS tunnel views). The threshold was determined from a binormal model fit to the mean values from each patient. The threshold for DIPI ratio and maximum offset deviation was determined from a binormal model fit to the 95% quantile of the relevant frames for each patient. (The DIPI ratio was transformed to ln[DIPI ratio+2] for statistical analysis. This transformation was used because the data were positively skewed and contained zero values.) The thresholds for discriminating SLS and intact frames were DIPI ratio
9.61, gap location (mm) <0.585, gap half-width (mm) <0.969, and maximum offset deviation ratio
3.9.
ROC curves15 were used to characterize the diagnostic ability of each variable. The area under the curve was used to assess whether the variables have any diagnostic ability. An area of 0.5 indicates no diagnostic ability, whereas an area of 1.0 indicates perfect discrimination between SLS and intact patients. MLEs of the area under the ROC curve were computed by use of LABROC.16 A resampling approach was used to construct nonparametric confidence intervals for the area under the ROC curve.17 This approach was used because the sample size was small and thus the methods for hypothesis testing based on large-sample theory would provide questionable results.
Objective Versus Subjective SLS Classification
Finally, the diagnostic accuracy of the quantitative analysis was compared with the diagnostic accuracy of the subjective review panel and the reviewers at WBH. In addition to the clinicians performing the radiographic examination at WBH, six expert reviewers, including one cardiac radiologist and five cardiologists,8 rated each patient on the 0 to 5 grading scale. (For the purposes of this study, the panel review scores were adjusted for cases that were assigned a score of 0. This was done for two reasons: [1] no case was assigned a score of zero by more than one expert reviewer and [2] it was assumed that the diagnosis must be made for each of the 22 patients. Therefore, a score of 3 was reassigned for those cases. This corresponds to a point of uncertainty between SLS and intact valves.) Nonparametric estimates of the area under the ROC curve were then used to assess the diagnostic accuracy of each expert reviewer.18 The diagnostic accuracy of the subjective reviewers versus the quantitative analysis involved the comparison of a discrete and continuous variable. To make a valid comparison of sensitivities at a given specificity, the nonparametric method of Beam and Wieand was used.19 The sensitivities were compared at a specificity of 1.0 because of the high cost of false-positives (ie, morbidity and mortality associated with reoperative replacement of an intact valve).
| Results |
|---|
|
|
|---|
9.61. We then calculated the average percentage of frames for patients with SLS valves and intact valves. The mean percentages of frames satisfying the discrimination criteria for SLS from intact valves are listed in Table 1
|
The MLEs of the area under the ROC curve are summarized for each quantitative variable in the upper half of Table 2
. For DIPI and gap half-width, the 95% CIs do not contain 0.5. Therefore, we conclude that these variables have diagnostic ability. However, the 95% CIs for gap location and maximum offset deviation contain the value 0.5; on the basis of this small sample, there is insufficient evidence to suggest that gap location or maximum offset deviation is useful in discriminating SLS from intact valves. A plot of the fitted ROC curves for each of the quantitative variables is presented in Fig 3
. Since DIPI and gap half-width are not correlated and they have diagnostic ability, we considered their joint probability to distinguish patients with SLS versus intact valves. A logistic regression was used to identify the best-fit model for predicting SLS as a function of DIPI ratio and gap half-width. From this model, the probability of SLS was predicted, and the probability was treated as a potential discriminator between SLS and intact valves. The fitted model is given by logit(p)=-10.44+0.527 (% frames whose DIPI ratio is
9.61)+0.129 (% frames whose gap half-width is <0.969), where p is the probability that the patient has an SLS valve and is given by p=e-logit(p)/[1+e-logit(p)]. The MLE of the area under the ROC curve for a model that includes both DIPI and gap half-width is 0.917 (SE, 0.059). The CI does not contain 0.5; therefore, a model that includes DIPI ratio and gap half-width is diagnostic. Although the area under the ROC curve is greater for a model that includes both DIPI and gap half-width, the 95% CIs for the difference between the bivariate model and a model based on either one of the variables alone (DIPI or gap half-width) contain zero. Therefore, a model based on both variables does not appear to be a better discriminator of SLS than one based on DIPI or gap half-width alone.
|
|
Since offset appeared to be a strong visual cue used by subjective reviewers to identify SLS and the results of the quantitative analysis did not support this observation, we were compelled to investigate this factor further. As noted above, approximately halfway through the study, changes were made at WBH in patient image acquisition that resulted in a 1.3 times larger valve image. It appeared that the maximum offset deviation was a less sensitive indicator of SLS in these magnified images. Therefore, we evaluated the effect of magnification on the diagnostic ability of each quantitative variable. The mean percentages of frames that satisfied the classification criteria for valves imaged before and after magnification are listed in Table 1
. Although the sample becomes extremely small when it is divided into before and after magnification, there is a trend toward improved discrimination between SLS and intact legs before magnification for DIPI ratio as well as maximum offset deviation. Specifically for DIPI ratio, the difference, SLS-intact, in the mean percentage of frames satisfying the classification criteria before magnification is 8.7% compared with 3% after magnification. This difference does not reach statistical significance (P=.117; two-way ANOVA with interaction). For gap location, the difference in mean percentage of frames that satisfy the classification criteria before magnification is 10.7%, compared with 38.6% after magnification. This difference is marginally significant (P=.076). For gap half-width, the differences in mean percentage of frames that satisfy the classification criteria before and after magnification are 3.1% and 34.7%, respectively. This difference is statistically significant (P=.007). Finally, for offset, the difference in mean percentage of frames that satisfy the classification criteria before magnification is 33.1%, compared with 0.6% after magnification. This difference is also statistically significant (P=.007).
The nonparametric estimates of the area under the ROC curve and the 95% CIs are listed for each expert subjective reviewer in the lower half of Table 2
. One of the 95% CIs for the subjective readers contained the value 0.5. Therefore, all but one of the subjective reviewers were diagnostic. The estimated accuracy of the average reader, excluding the investigator at WBH, is 0.853 (SE, 0.067).
At an estimated specificity of 1.0 (ie, the average proportion of the intact valves assigned a score of
3 by the reviewers), the estimated sensitivity of the average reviewer is 0.42. This compares with the estimated sensitivity of 0.2 for DIPI ratio, 0.5 for gap half-width, and 0.3 for these variables combined. The 95% nonparametric CI for difference in sensitivities between the quantitative measures and the subjective reviewers contains zero. This indicates that there is no difference in average sensitivity of the quantitative measures versus the qualitative review.
The mean locations of the fracture measured mechanically and from the high-magnification radiographs were 0.31±0.14 and 0.40±0.09 mm, respectively. The mean gap widths measured mechanically and from high-magnification radiographs of the explanted SLS valves were 0.000±0.001 and 0.083±0.065 mm, respectively. The mean offset deviations measured mechanically and from the high-magnification radiographs were 0.150±0.047 and 0.097±0.050 mm, respectively.
| Discussion |
|---|
|
|
|---|
It is important to note that the SLS valves in this study were a selected set of the larger 315-patient WBH population. This set was made up of only those SLS valves explanted during the course of or after the WBH study. Although the sensitivity of objective review was comparable to that of the subjective review panel, it was not as high as that for the observers directly involved in the radiographic examinations (apparent sensitivity, 0.833; apparent specificity, 0.997).8 This is partly because the reviewers at WBH had real-time interaction during the image acquisition process and were able to manipulate the patient to obtain optimal views suitable for their evaluation. Since there is currently no noninvasive means for identifying possibly misclassified SLS valves in the patients who did not have their valves explanted, it is impossible to assess the actual false-negative ratio of the subjective review or the reviewers at WBH.
When SLS was present, it could be observed in only a small percentage (ie, DIPI in 4.7%, gap half-width in 13.4%) of the frames within an image sequence. This could be related to the optimum image view or the optimum time within the cardiac cycle. An attempt was made to correlate the suspicious frames (ie, DIPI ratio
9.61 or gap half-width <0.969) within a sequence to a specific time in the cardiac cycle, such as disk opening or closing. This was done by retrospectively gating an image sequence by the position of the tilting occluder disk. No correlations could be made between the position of the disk and SLS-classified frames. This was partly a result of problems in determining the exact position of the disk in several image frames of the tunnel view. However, there are two pieces of evidence that suggest that visualization of SLS may be correlated to cardiac cycle. The first is from previous mechanical studies that suggest that large bending stresses may be imposed on the outlet strut at disk opening or closing.1 2 The second is from the valves analyzed in this study that had been explanted and examined ex vivo. The fracture width and offset measured ex vivo in a static unloaded condition are smaller than the resolving capabilities of the cineangiographic image acquisition system. This suggests that the size of the separation may be greater in vivo when the valve is in a loaded state. Further studies are required to determine the optimum time in the cardiac cycle for visualization of SLS. These studies would use the ECG to gate the frame-by-frame image analysis and thus identify specific periods of the cardiac cycle at which SLS may be more apparent (ie, valve loading). Such an investigation would be relatively simple to implement, because it would require only simultaneous acquisition of the ECG signal with the cineangiographic image acquisition. Unfortunately, this information was not available in the WBH study.
The success of this semiautomated objective analysis technique could be further enhanced by ECG gating and direct digital image acquisition. ECG gating at image acquisition could be used to identify those time periods in the cardiac cycle when there is a high probability of observing SLS. Once these high-probability time periods had been established, cineangiographic images would be acquired only during these time intervals. This would reduce the amount of ambiguous and sometimes confounding information acquired in an image sequence as well as the amount of x-ray exposure to the patient. The time required to digitize and analyze one frame of data was
1 minute. Half of this time was required for secondary digitization of the x-ray film. Direct digital image acquisition would eliminate the need for this secondary digitization. It would also provide the platform for near real-time image analysis. A real-time image analysis system could be used by the clinician to aid in the assessment of the valve while the patient is being imaged.
The quantitative analysis of these 22 patients required digitization and postprocessing of 3872 frames of cineangiographic data. The bayesian classification technique used to segment the strut leg from the background was robust in segmenting the strut leg in all of these images, several of which had a significant variation in background pixel intensity. Other segmentation techniques, such as Kittler's automated threshold,14 did not perform as well as the bayesian classification. The bayesian technique did, however, require the user to interactively identify regions in the strut and background. This contributed to the processing time and prohibited fully automated processing. A technique for automatically identifying these regions has since been developed, implemented, and tested for the oblique views.20
The objective analysis of cineangiographic images provided valuable insight into the radiographic detection of SLS in vivo. It is a reproducible method of analysis that does not suffer from the inherent problems of subjective review. The ability to identify SLS in cineangiographic images is currently limited by the imaging technology and the expertise of the clinical reviewers. It requires optimized cineangiographic imaging equipment and specially trained clinical reviewers, neither of which is available at typical cineangiographic imaging sites. An objective automated analysis system could be used to train and aid clinical reviewers in detection of SLS if specially designated imaging sites were established throughout the world. The present study has identified several areas in which the objective postprocessing technique may be improved beyond that of the subjective expert review panel. However, the proposed enhancements will require further investigation.
| Selected Abbreviations and Acronyms |
|---|
|
| Acknowledgments |
|---|
Received April 15, 1996; revision received July 23, 1996; accepted August 7, 1996.
| References |
|---|
|
|
|---|
2.
Sacks SH, Harrison M, Bischler PJE, Martin JW, Watkins J, Gunning A. Metallurgical analysis of failed Bjork-Shiley cardiac valve prostheses. Thorax. 1986;41:142-147.
3. Birkmeyer JD, Marrin CAS, O'Connor GT. Should patients with Bjork-Shiley valves undergo prophylactic replacement? Lancet. 1992;340:520-523.[Medline] [Order article via Infotrieve]
4. Lindblom D. Long-term clinical results after mitral valve replacement with the Bjork-Shiley prosthesis. J Thorac Cardiovasc Surg. 1988;95:321-333.[Abstract]
5. de Mol BA, Kallewaard M, McLellan RB, van Herwerden LA, Defauw JJ, van der Graaf Y. Single-leg strut fractures in explanted Bjork-Shiley valves. Lancet. 1994;343:9-12.[Medline] [Order article via Infotrieve]
6. Abolfathi AH, Hirsch JL, Nissen SE, Wieting DW, Chandler JG. Effect of valve profiling and increasing kVp on in situ radiographic detection of single leg separations of the Bjork-Shiley convexo-concave heart valve. In: Proceedings, 16th Annual International Conference: IEEE Engineering in Medicine and Biology; 1994; Baltimore, Md.
7. Gordon RE, Castellani MD, Bakalyar DM, Beatty KC, Catlin RT, Chandler JG, O'Neill WW. Radiographic detection and successful prophylactic explantation of single leg separation in patients with Bjork-Shiley convexo-concave prosthetic mitral valve. J Am Coll Cardiol. 1994;23(suppl A1):291A. Abstract.
8.
O'Neill WW, Chandler JG, Gordon RE, Bakalyar DM, Abolfathi AH, Castellani MD, Hirsch JL, Wieting DW, Bassett JS, Beatty KC, Soltis MA, Timmis GC, Grines CL. Radiographic detection of strut separations in Bjork-Shiley convexo-concave mitral valves. N Engl J Med. 1995;333:414-419.
9. Chandler J, Pobedonostzeff A. Risk of outlet strut fracture of Bjork Shiley convexo-concave valves. Lancet. 1992;340:861. Letter.[Medline] [Order article via Infotrieve]
10. Schondube FA, Althoff W, Dorge HC, Voss M, Laufer JL, Chandler JG, Messmer BJ. Prophylactic reoperation for strut fractures of the Bjork-Shiley convexo-concave heart valve. J Heart Valve Dis. 1994;3:247-253.[Medline] [Order article via Infotrieve]
11. Grunkemeier GL, Chandler JG, Miller DC, Jamieson WR, Starr A. Utilization of manufacturers' implant card data to estimate heart valve failure. J Heart Valve Dis. 1993;2:493-503.[Medline] [Order article via Infotrieve]
12. Duda RO, Hart PE. Pattern Classification and Scene Analysis. New York, NY: John Wiley & Sons; 1973.
13. Marr D, Hildreth E. Theory of edge detection. Proc R Soc London. 1980;207:187-217.[Medline] [Order article via Infotrieve]
14. Kittler J, Illingworth J. Minimum error thresholding. Pattern Recognition. 1986;19:41-47.
15. Metz CE. Basic principles of ROC analysis. Semin Nucl Med.. 1978;8:283-298.[Medline] [Order article via Infotrieve]
16. FORTRAN program LABROC, available from C. Metz, Department of Radiology, University of Chicago, Chicago, Ill.
17. Efron B. The Jackknife, the Bootstrap and Other Resampling Plans. Philadelphia, Pa: Society for Industrial and Applied Mathematics; 1982:38. CBMS-NSF Regional Conference Series in Applied Mathematics.
18. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837-845.[Medline] [Order article via Infotrieve]
19. Beam CA, Wieand HS. A statistical method for the comparison of a discrete diagnostic test with several continuous diagnostic tests. Biometrics. 1991;47:907-919.[Medline] [Order article via Infotrieve]
20. Cesmeli E. Translational and Rotational Correction of Bjork-Shiley Convexo-Concave 60° and 70° Heart Valves. Columbus, Ohio: Ohio State University; 1994. Master's Thesis.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Circulation Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 1996 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |