Performing surgery on the beating heart has significant advantages compared to cardiopulmonary bypass. However, when performed directly, it is very demanding for the surgeon. As an alternative, using a teleoperated robot for compensating the heart motion has been proposed. As an addition, this paper describes how stabilized images are obtained to create the illusion of operating on a stationary heart. For that purpose, the heart motion is tracked with a stochastic physical model. Based on correspondences obtained by motion tracking, image stabilization is considered as a scattered data interpolation problem. The proposed algorithms are evaluated on a heart phantom and in in-vivo experiments on a porcine heart, which show that there is very little residual motion in the stabilized images.
In order to track the three-dimensional motion of the heart, a trinocular camera system is used to locate landmarks on the
heart surface. Various approaches for tracking natural landmarks on the heart surface have been proposed, for example [
The general goal of image stabilization is to remove motion from a video stream while preserving changes to color and texture. A reference image from a certain time step is given and information from the current image is transformed to appear in the reference image. Image Stabilization can be performed by calculating an interpolation from two-dimensional point correspondences {(xi, yi, xt, yt) | 1 ≤ i ≤ m} between the current image and the reference image. As the image is supposed to be warped to the reference image, the function f should interpolate (or at least approximate) these corresponding points. The corresponding points are interpolated by f if the equation (xi, yi)T = f (xt, yt), i = 1, . . . , m
i i
is fulfilled. The problem of image stabilization is thus reduced to the problem of scattered data interpolation. The
function f can be chosen from different families of functions. Common examples are global affine transformations [
Alternatively, image stabilization can be performed in three dimensions: The point (xt, yt)T in the reference image is a
projection of a point (Xt, Yt, Zt)T on the current 3D surface. A function h : R3 → R3 then describes a mapping of
(Xt, Yt, Zt)T on the reference 3D surface to (X, Y, Z)T on the current 3D surface. This point (X, Y, Z)T is then projected back
to (x, y) in the current image. The function h can be derived from the physical model, which describes the displacement
of the heart surface at any point. This approach is introduced and evaluated in [
The presented image stabilization approaches have been evaluated both on a heart phantom and in an in-vivo experiment
on a porcine heart. In both cases, a trinocular camera system consisting of three Pike F-210C cameras [
The experimental setup for the ex-vivo experiment consists of a pressure regulated artificial beating heart, which is located approximately 50 cm below the trinocular camera system. A pressure signal with amplitude 100 hPa and frequency 0.7 Hz causes the motion of the artificial heart. For evaluation, an image sequence consisting of 400 frames with a frame rate of 23 fps was recorded.
The in-vivo experiment was performed at Heidelberg University Hospital. Markers were placed on the beating heart of a pig and a trinocular camera system was used to record a sequence of 337 frames at a frame rate of 31 fps. A cardiac catheter was used to measure the pressure inside the left ventricle. The heart was mechanically stabilized with the commercially available Octopus stabilizer. Since the motion of the real heart surface is affected by breathing and by the motion of all four heart chambers, the physical model was extended with an excitation based on Fourier series. To analyze the residual motion in the stabilized images, the average difference across all k frames I1, . . . , Ik to the reference frame Ireference was calculated for each pixel (x, y) and each color channel c ∈ {R, G, B} according to k t=1 |It(x, y, c) − Ireference(x, y, c)| . e = c∈{R,G,B} y error 1
I
p x
For the purpose of this evaluation, only points inside the convex hull of all landmarks are taken into account. The image
of average differences Ierror was converted to grayscale in the range [
The average error e across the entire image summed over all color channels can be obtained by
where p denotes the number of pixels inside the convex hull of all landmarks (see Table 1). For both experiments, the
unstabilized image can be compared to a simple stabilization, which is based on a global affine transformation, and to
a more sophisticated stabilization, which is based on multi-level B-Splines as described in [
unstabilized
affine B-Spline (x, y, c) As the evaluation of the proposed algorithms demonstrates, it is possible create a stabilized view of the heart that is almost completely stationary. The results clearly show the superiority of the B-Spline transformation to the global affine transformation. This is not surprising since the affine transformation cannot properly deal with non-uniform deformations. Image stabilization depends on reliable tracking of the heart motion. The presented model-based technique allows robust tracking even in the presence of uncertainties and occlusions. As the in-vivo experiment illustrates, reliable tracking is not only possible with the heart phantom but under more difficult real-life conditions as well.
Future work might include optimizations of stabilization accuracy in order to further reduce the amount of residual motion. Furthermore, integration with a surgical robot is required to create a system that can be used for surgery.
This work was partially supported by the German Research Foundation (DFG) within the Research Training Groups RTG 1126 “Intelligent Surgery - Development of new computer-based methods for the future working environment in visceral surgery”.
We thank Evgeniya Ballmann for her work on physics-based motion compensation and Szabolcs Pa´li for his contributions to the in-vivo experiment. 5