Multiscale Motion Mapping
A Novel Computer Vision Technique for Quantitative, Objective Echocardiographic Motion Measurement Independent of Doppler: First Clinical Description and Validation
Background— Objective, quantitative, segmental noninvasive/bedside measurement of cardiac motion is highly desirable in cardiovascular medicine, but current technology suffers from significant drawbacks, such as subjectivity of conventional echocardiographic reading, angle dependence of tissue Doppler measurements, radiation exposure by computer tomography, and infrastructure requirements in MRI. We hypothesized that computer vision technology could represent a powerful new paradigm for quantification in echocardiography.
Methods and Results— We present multiscale motion mapping, a novel computer vision technology that is based on mathematical image processing and that exploits echocardiographic information in a fashion similar to the human visual system. It allows Doppler- and border-independent determination of motion and deformation in echocardiograms at arbitrary locations. Correctness of the measurements was documented in synthetic echocardiograms and phantom experiments. Exploratory case studies demonstrated its usefulness in a series of complex motion analyses that included abnormal septal motion and analysis of myocardial twisting. Clinical applicability was shown in a consecutive series of echocardiograms, in which good feasibility, good correlation with expert rating, and good intraobserver and interobserver concordance were documented. Separate assessment of 2D displacement and deformation at the same location was successfully applied to elucidate paradoxical septal motion, a common clinical problem.
Conclusions— This is the first clinical report of multiscale motion mapping, a novel approach to echocardiographic motion quantification. For the first time, full 2D echocardiographic assessment of both motion and deformation is shown to be feasible. Overcoming current limitations, this computer vision–based technique opens a new door to objective analysis of complex heart motion.
Received March 31, 2004; revision received June 30, 2004; accepted July 8, 2004.
Although the title of the very first description of echocardiography by Edler and Hertz1 in 1954 mentioned “recording of the movements of heart walls” as its goal, the objective and quantitative measurement of wall motion has remained difficult in clinical practice. Objective assessment of wall motion is of considerable importance in a broad range of patients: Objective documentation that contractility is normal in patients examined in a screening situation may be as important as the reliable quantification of ventricular function in heart failure and the detection of wall-motion abnormalities at rest or during stress in those with suspected ischemic heart disease. The presence of segmental variations in contractility adds an additional level of difficulty to this analysis. The causes of these difficulties are manifold. Cardiac motion is complex, encompassing active thickening, passive translation of contracting and noncontracting segments, rotational components (eg, apical twisting), and shear components, for example, between endocardial, epicardial, and pericardial structures. Furthermore, ultrasound images are typically noisy, tissue intensities are not constant, and endocardial borders lack continuity, which significantly limits the applicability of standard image-analysis approaches such as border-detection algorithms. The most successful current technique, tissue Doppler-based analysis of motion2–5 and deformation,6–8 is hampered by the critical dependence of both measurements from the ultrasound beam direction, which leads to signal dropout and velocity underestimation.
However, most cardiologists will agree that the human visual system is quite adept at distinguishing such complex motion patterns even in noisy data, although analysis remains subjective and semiquantitative. We hypothesized that application of some of the fundamental properties of the human visual system to echocardiographic quantification could result in robust quantification of motion in an objective manner. To this end, we developed and validated multiscale motion mapping, a computer vision–based approach to echocardiographic motion quantification.
Multiscale motion mapping is a computer vision technology based on advanced mathematical image data analysis. The information contained in echocardiographic loops can be exhaustively described as a large numerical data set that is usually rendered as an image sequence on a screen; it is used here for computation of local motion and deformation. The following method description is given for the nonmathematician, whereas mathematical details of the approach are given in a separate technical paper in an engineering journal.9 The technique has a number of key features, as outlined below.
Continuous Representation of Reality
Conventional echocardiographic technology, including Doppler echocardiography, relies on pixel- and frame-wise image processing: Data for each pixel are computed independently, and the overall frame is then constructed from the individual pixels, after which the next frame is computed. In contrast, real objects are smoothly connected in space and move continuously instead of jumping from frame to frame. Therefore, we use a continuous and coherent mathematical representation in space and time, a so-called spline model10 of the ultrasound data, an approach of proven value in other fields.
Human vision is strongly context dependent. The overall structure of an object determines which details our brains will expect and recognize, whereas apparent features that are incoherent within the larger context are suppressed; for example, when looking at a partner’s face in heavy snowfall, we will pay more attention to the overall facial expression than to some bright snowflakes in front of the face. In addition, our brains expect moving objects to be coherent in time, which makes human vision even more robust. We therefore implemented spatial context dependence and temporal coherence in a so-called multiresolution pyramid: Overall shape and coarse motion are used to refine the analysis of smaller structures and of more detailed motion in a recursive fashion, analogous to the hierarchical organization of the early image-processing stages in the human visual system.
Affine Model of Motion
Typical motion in echocardiograms includes translation, deformation (eg, myocardial shortening along the myocardial fiber axis accompanied by thickening of the same structure perpendicular to the fiber axis), rotation (eg, valvular motion, apical twisting), and shear motion (eg, inner versus outer myocardial planes, epicardium versus pericardium). Mathematically, all these motion patterns (Figure 1) can be locally described in a so-called affine model: equation
The local motion (u, v) at each location is determined by the translation (u0, v0) at the region center (x0,y0) and local rotation and deformation are coded by the derivatives ux, uy, vx, vy. The image region is chosen to be of such size that it contains enough information to solve this equation for all unknowns using linear algebra. At the heart of multiscale motion mapping is the insight that for each image region in an echocardiogram, knowledge of the numerical grayscale values in this region (readily available in the image raw data) is sufficient for computation of just that affine motion equation that fits best to this region, thereby predicting local translation, rotation, and deformation. This ability of multiscale motion mapping to fit complex heart motion is one of several differences from conventional “optical flow” techniques, to which it is superior.9
Multiscale motion mapping yields a large amount of motion information. A simple, intuitive display is therefore of particular importance. With simple mouse clicks, different motion representations can be displayed as shown in Figure 2. Figure 2a shows an arrow representation of 2D velocity at arbitrary locations and time points. The arrow direction corresponds to the motion direction; the arrow length codes the absolute velocity. Their numerical values can also be exported. Radial velocity toward a reference point can also be displayed as a color map (Figure 2a, right). This color map is Doppler independent and thus free of the angle-dependent dropouts typical of the latter. Radial inward motion can be represented in the same color independently of the ultrasound beam angle, which renders motion display more intuitive. As shown in Figure 2b, active myocardial contraction has 2 physiological components: shortening along the myocardial fiber direction, and thickening perpendicular to it. Both can be represented in an ellipse that indicates the 2 principal axes and the absolute values of 2D strain rate, which circumvents the problem of tissue Doppler in which the 2 strain rate components will cancel each other at certain beam angles.
Echocardiograms were acquired with a Philips SONOS 5500 in native digital format or with one of several DICOM-3–enabled machines from Siemens, General Electric, or Toshiba and were stored digitally. Processing was done on a standard personal computer, with key hardware requirements consisting either of a current Pentium IV or PowerPC G4 processor and 1 gigabyte of RAM, with processing times in the range of 1 minute per sequence.
Validation of measured versus true motion was first performed in 2 setups for which true motion is known: (1) synthetic echocardiograph sequences and (2) moving physical phantoms. Then, clinical exploration was done with a series of prototypical echocardiograms. Finally, the feasibility of the method in routine echocardiography was evaluated in a consecutive series of echocardiograms performed for clinical indications. Here, conventional subjective reading was done by 3 experts. The 16-segment nomenclature11 was used, in which each segment was assigned 1 of 5 subjective ratings from hyperkinetic to dyskinetic and each view was given an overall label of “normal” (hyperkinetic or normokinetic) or “abnormal” (hypokinetic, akinetic, or dyskinetic). Objective analysis by multiscale motion mapping was performed by 2 blinded observers, as well as by repetition of the measurements by the same observer after several weeks. Velocity measurements and subjective ratings were used to train an artificial neural network (3-layer structure, sigmoid response, error-backpropagation algorithm for training12) with the first half of the available data. The ability for automatic classification of patients as normal or abnormal using quantitative measurements was validated with the second half of the data.
Standard statistical methods were used with a 2-sided significance level of 0.05. Linear regression and Bland-Altman analysis13 were used to compare measured with correct motion in models with known motion. Interobserver agreement and intraobserver consistency were assessed with Bland-Altman analysis for continuous variables and the κ-test for classification.
In the following section, we present results obtained from synthetic ultrasound data, phantom experiments, and clinical echocardiograms. Motion can only be displayed for single frames in this print version; for a better dynamic perception, moving-image loop representations can be viewed in the online-only Data Supplement at http://www.circulationaha.org.
Overall Capabilities in Synthetic Echocardiograms
Multiscale motion mapping was first applied to synthetic echocardiogram loops to assess its capabilities in the case of simplified, clearly defined motion. The example shown in Figure 3a simulates an apical short-axis view. The applied motion consists of uniform radial expansion/contraction combined with global, periodic translation in a diagonal direction. The abilities of the algorithm to measure and visualize this motion are demonstrated in Figure 3b through 3d. The following representations were computed from a 2D grayscale loop: (1) A full-motion field indicating velocity arrows at all locations and all time points in an image loop is shown in Figure 3b. The arrow display reflects the superposition of radial contraction and global translation. (2) A color map indicating motion components relative to the ventricular centroid moving with the ventricle is shown in Figure 3c. The yellow/red color indicates uniform inward motion. (3) Figure 3d displays 2D deformation by strain rate ellipses. Myocardial thickening and circumferential shortening are indicated by the ellipses. The deformation map is independent of the underlying translational motion. In contrast to tissue Doppler–based strain rate analysis, deformation measurement does not depend on the deformation orientation.
Measured velocities were compared with known true motion and showed very good agreement. In Bland-Altman analysis, there was a minimal bias (mean velocity difference −0.005 pixel/frame, P<0.001, SD=0.056 pixel/frame [1.4% of measured range]) with a correlation coefficient of 1.00 (P<0.001; regression line y=1.04x−0.00), which documents the accuracy of the algorithm. Measured deformation also corresponded well to true deformation (for absolute strain rate, correlation coefficient 0.88 (P<0.001), mean difference 0.0 s−1 [P=NS]; SD=4.4% of measured range; correlation coefficient for strain rate direction 0.89, P<0.001).
The ability to analyze motion in real ultrasound data was tested in a phantom experiment. A cylinder-shaped, tissue-mimicking phantom (radius 4.0 cm) was placed inside a tube of water and rotated with constant angular velocity (1.1 rad/s). Figures 4a and 4b show 1 frame of the B-mode sequence and its rotational velocity field, respectively. For comparison, the color Doppler signal is shown in Figure 4c; here, the typical Doppler signal dropout at certain beam angles is evident.
Estimated velocities were in good agreement with true phantom motion. The correlation coefficient between true and estimated velocity magnitudes was 0.97 (P<0.001; regression line y=0.86x−0.01). Bland-Altman analysis showed a negligible bias (mean velocity difference: 0.108 cm/s, P<0.001, SD=0.292 cm/s [3.3% of the measured range]).
Application to Clinical Echocardiograms
The ability to display motion in clinical echocardiograms independently of the ultrasound beam angle is demonstrated in the short-axis view of a normal beating heart shown in Figure 5a. The ability of multiscale motion mapping to assess apical myocardial twisting, which is clearly appreciated visually, is illustrated in Figure 5b by the arrow display. The motion arrows document the clockwise apical twisting during early diastole. The complexity of the actual motion pattern becomes readily apparent in the fish-swarm–like appearance of the arrows, which illustrates the difficulties of motion assessment with conventional methods that are limited to detection of only a 1D motion component.
The color-coded radial velocity component toward the center point is shown in Figure 5c for mid systole and in Figure 5d for early diastole. Note that in systole, the entire circumference is color-coded in red, which indicates inward motion everywhere (whereas in tissue Doppler, there would be an entire spectrum of colors, rendering reading less straightforward).
Deformation maps of the myocardium during systole and diastole are shown in Figures 5e and 5f, respectively; the radial alignment and elongation of the ellipses indicate the myocardial wall thickening during systole. In diastole, the circumferential alignment of the ellipses documents the corresponding wall thinning.
Comparison With Tissue Doppler
Agreement between multiscale motion mapping and tissue Doppler velocities was assessed in 132 separate measurements in the center of each segment from 11 clinical echocardiograms for which B-mode and tissue Doppler signals had been acquired simultaneously. For comparison, only the motion vector component parallel to the ultrasound beam direction was used from multiscale motion mapping, because tissue Doppler is unable to measure the motion component perpendicular to the beam. There was a good correlation between multiscale motion mapping and Doppler, with a correlation coefficient of 0.94 (P<0.001; regression line y=0.87x−0.14). Bland-Altman analysis showed no significant bias (mean velocity difference 0.223 cm/s, P=NS, SD=1.527 cm/s [7.2% of measured range]).
Clinical applicability is a major requirement for a new technology. We collected prospectively a consecutive series of clinical echocardiograms from 125 hospitalized patients (mean age 64±14 years, 33% judged as normal by conventional diagnosis) referred for echocardiography for clinical indications. Image quality was deemed sufficient for quantification in 114 cases (91.2%). Multiscale motion analysis was applied independently by 2 blinded users and produced completely reproducible motion maps. For each data set, segmental motion measures were extracted independently by 2 users, who subjectively specified locations in the centers of different myocardial segments. Interobserver variability of manual velocity extraction (SD) was 0.896 cm/s (5.1% of measured range), with a mean velocity difference of −0.077 cm/s (P=0.04) and a correlation coefficient between the 2 observers of 0.83 (P<0.001). The intraobserver variability of velocity extraction (SD) was 0.568 cm/s (3.2% of measured range) with a mean velocity difference of −0.063 cm/s (P=NS). The corresponding correlation coefficient was 0.93 (P<0.001).
Automatic objective classification of the measured velocities by the artificial neural network led to the global, binary classification of normal or abnormal motion. Objective classification corresponded to the “majority vote” of subjective expert ratings in 84% of cases. Interobserver agreement of subjective readings (κ=0.617) was only moderate. Interobserver agreement for classification based on multiscale motion mapping was good (κ=0.740).
Comprehensive Objective Motion Analysis: Case Study
To exemplify the practical application of multiscale motion mapping, an example of a routine echocardiogram for which local echocardiography experts disagreed as to the presence or absence of motion abnormalities is given in Figure 6. Figures 6a and 6b show the native grayscale loop together with 2D motion arrows in the septum and lateral wall during systole and diastole, respectively. They show that there is paradoxical motion of the basal interventricular septum. Additional insight can be gained through separate display of transverse (toward the centerline) and longitudinal (along the centerline) ventricular motion in the same loop as shown in Figures 6c and 6d, respectively. The motion components at mid systole indicate that there are reduced longitudinal velocities of the basal left ventricle (P<0.01 versus normal), whereas transverse velocity of the lateral segments was in the normal range, and transverse velocity was negative in the basal septum. The 2D strain rate, however, was normal (peak systolic thickening/thinning strain rates in the basal septum 1.28 s−1 and −1.03 s−1; P=NS versus normal, with maximum thickening directed toward the mid ventricle, and a corresponding behavior in diastole).
These findings of objective, quantitative analysis—namely, paradoxical motion but normal thickening of the septum, combined with impaired long-axis but normal transverse function of the lateral segment—agree nicely with careful subjective analysis of the echocardiogram. Such comprehensive, objective evaluation of heart motion is not possible with any other echocardiographic technique.
This is the first clinical description of multiscale motion mapping, a novel computer vision–based approach for quantification of motion in echocardiograms. Based on comprehensive mathematical analysis of digital images, and using strategies similar in several aspects to the human visual system, multiscale motion mapping allows visualization and quantification of true 2D heart motion independent of borders, Doppler, and beam angles. From a single conventional grayscale data set, multiscale motion mapping is able to quantify absolute local velocity, 2D velocity direction, 2D strain rates, and principal axes of strain components.
The ability to objectively quantify heart function is much needed today for a number of reasons. The epidemic of heart failure calls for improved methods to diagnose impaired myocardial function at an early stage, to follow its course quantitatively, and to study the impact of new therapeutic strategies in a noninvasive but objective manner. The ever-increasing population segment in the Western world that is at risk for ischemic heart disease renders noninvasive diagnostic methods such as stress echocardiography more and more important but also exposes their Achilles heel, namely, subjectivity and limited interobserver agreement. Finally, noninvasive examinations may lead to invasive procedures; further improvements in the objectivity of echocardiography may thus also influence cost efficiency in cardiology.
Although reliability in the description of anatomy, of intracardiac flows, and of valve function has reached a high level in the 40 years since echocardiography was invented, determination of myocardial function has lagged behind for several reasons. First, cardiac motion is very complex; even at a single location, it consists of components that may either be independent from each other, such as passive heart translation and active myocardial thickening, or that are physiologically connected, such as shortening along the myocardial fiber axis and thickening perpendicular to the fiber axis. Motion models that are too simple may thus be insufficient to describe such motion correctly.
Second, in conventional 2D echocardiograms studied here, our visual system is able to detect full 2D motion patterns, whereas conventional analysis methods are typically 1-dimensional: Border-detection algorithms detect motion perpendicular to the endocardium, and tissue Doppler detects motion parallel to the ultrasound beam. (Clearly, out-of-plane motion also exists in echocardiography; this is probably best examined in dynamic 3D imaging. Although not a topic of this report, it is in principle also accessible to the approach described here). The key weakness of 1D methods is well known for conventional Doppler, in which motion near 90° to the beam cannot be detected, but it is less well known that this poses a particular problem for Doppler-based deformation imaging, because geometry predicts that there are 2 “blind spots” for Doppler deformation imaging of myocardial contraction near 45° and 135°.
An important determinant of actual use of new technology in echocardiography is the ease of performing and interpreting measurements. Here, multiscale motion mapping offers 2 new aspects. In the color display, velocities can be encoded such that inward motion, for example, takes the same color in multiple segments (Figures 5c and 5d), an evident simplification compared with tissue Doppler, in which Doppler angle dependency leads to a multitude of colors even though the same absolute velocity actually may be present. In the arrow display, simple computer mouse clicks allow determination, visualization, and quantitative export in a spreadsheet of both motion direction and absolute value. Multiscale motion mapping thus extends current technology by making available more detailed quantitative motion information than with either visual analysis, Doppler velocity imaging, Doppler deformation imaging, or border-detection strategies such as color kinesis,14 while readability is maintained.
Current limitations of multiscale motion mapping include limited availability of the method on commercial machines, a point easily addressable because no hardware changes are needed to implement the method on a digital echocardiograph machine (currently, it is implemented on an external analysis package). As for the human eye, there is a limit of image quality below which recognition and measurement of structures and motion becomes unfeasible. In our consecutive series of clinical echocardiograms, this was a problem in <10% of cases; this robustness probably originates in the exploitation of spatial and temporal coherence (see Methods); nevertheless, we expect that the ongoing echocardiography hardware evolution will further improve the applicability of computer vision approaches in echocardiography. Because this is a beat-to-beat analysis, the pitfalls of measuring nonrepresentative beats in arrhythmia but also the potential to compare beat-to-beat variability of contraction need to be addressed.
Clearly, this first clinical description needs to be followed by more thorough clinical validation and application in a wide variety of clinical situations. Clinical studies applying multiscale motion mapping to routine echocardiography, heart failure quantification, and stress echocardiography are ongoing.
We believe that multiscale motion mapping may have a significant impact on the practice of echocardiography. First, it allows exploration of complex features of heart motion inaccessible to current echocardiographic techniques, such as quantitative assessment of apical twisting motion, as well as direct comparison of axial shortening, radial motion, and local thickening of the myocardium. Second, it expands the toolbox of the cardiologist with a quantitative tool that could be profitably applied in routine echocardiography for improved definition of normal versus abnormal left ventricular function (probably the most frequent indication for echocardiography); likewise, improved quantitative analysis may overcome limitations of echocardiography in diagnosis and longitudinal assessment of heart failure and may reduce the interobserver variability of stress echocardiography. Because determination of global ventricular function (by integration of regional motility) and evaluation of asynchrony of contraction (by comparison of the time course of velocities) are done at the same time, the application of the method to patients with bundle-branch block and resynchronization therapy appears promising.
This first clinical report of multiscale motion mapping thus opens a new door for improved analysis of cardiac motion. Availability of full 2D information for both displacement and deformation, unaffected by the known limitations of Doppler, not only allows exploration of the complex features of cardiac motion that are inaccessible to current techniques but also appears useful for making a wide range of clinical echocardiography more objective and quantitative.
This study was supported in part by grant 3200-059517 from the Swiss National Science Foundation to Drs Unser and Hunziker and a grant from the Swiss Heart Foundation to Drs Unser and Hunziker.
↵*Michael Sühling and Christian Jansen participated equally in this work.
Presented in part at the Young Investigator’s Award Session, Sixth Annual Meeting of the Working Group on Echocardiography of the European Society of Echocardiography (EUROECHO6/02), Munich, Germany, December 4–7, 2002, and published in abstract form (Eur J Echocardiogr. 2002;3[suppl 1]:122).
Edler I, Hertz CH. The use of ultrasonic reflectoscope for the continuous recording of the movements of heart walls. Kungl Fysiografiska sällskapets i Lund förhandlingar. 1954; 24: 1–19.
Sutherland GR, Steward MJ, Groundstroem KW. Color Doppler myocardial imaging: a new technique for assessment of myocardial function. J Am Soc Echocardiogr. 1994; 33: 3141–3146.
Derumeaux G, Ovize M, Loufoua J, et al. Doppler tissue imaging quantitates regional wall motion during myocardial ischemia and reperfusion. Circulation. 1998; 97: 1970–1977.
Fleming A, Xia X, McDicken W, et al. Myocardial velocity gradients detected by Doppler imaging system. Br J Radiol. 1994; 67: 679–688.
D’hooge J, Heimdal A, Jamal F, et al. Regional strain and strain rate measurements by cardiac ultrasound: principles, implementation and limitations. Eur J Echocardiogr. 2000; 1: 154–170.
Sühling M, Arigovindan M, Jansen C, et al. Myocardial motion analysis from B-mode echocardiograms. IEEE Trans Image Processing. In press.
Schiller NB, Shah PM, Crawford M, et al. Recommendations for quantitation of the left ventricle by two-dimensional echocardiography: American Society of Echocardiography Committee on Standards, Subcommittee on Quantitation of Two-Dimensional Echocardiograms. J Am Soc Echocardiogr. 1989; 2: 358–367.
Hagan MT, Demuth HB, Beale MH. Neural Network Design. Boston, Mass: PWS Publishing; 1996.
Mor-Avi V, Vignon P, Koch R, et al. Segmental analysis of color kinesis images: new method for quantification of the magnitude and timing of endocardial motion during left ventricular systole and diastole. Circulation. 1997; 95: 2082–2097.