In various clinical scenarios, as e.g. minimally invasive surgery (MIS), endoscopy-based diagnostics in the upper and lower GI tract or clinical motion analysis of organs, the view onto the site is usually restricted by endoscopic apertures. Nevertheless, computer based (surgical) assistance systems often rely on manually defined and initiated landmarks, which must be tracked during the intervention. A typical application of this type is for example the tracking of landmarks on the beating heart within open or minimal invasive heart surgery to enable the augmentation of the view with matched and warped pre-operative image data [1,2,3,4]. Also for clinical motion analysis as the determination of movements of heart valves, manually defined landmarks have to be tracked [5,6]. Nevertheless, due to the slight but constant change of the underlying anatomical structure as well as changes of endoscopic orientation and illumination, classic template matching approaches such as correlation or SSD are neither stable nor sufficient [1,2] for such applications. Alternatively, feature tracking methods are used to determine movements in similar image data [7,8]. The literature dealing with tracking of medical image data from monocular image sequences describes such approaches, which are designed to automatically detect newly appearing local features in the scene and are hence not designed to deal with externally provided landmarks. To solve this problem and to exploit the advantages of combining robust feature descriptors with manually defined landmark points, a local feature tracking approach for endoscopic motion analysis is presented and evaluated on various types of endoscopic imagery.
Robust feature descriptors and the corresponding matching algorithms are well known and established methods to identify and track prominent identical points in sequences of consecutive image frames. These methods have been optimized to track feature points with specific characteristics in monocular image sequences. In contrast, manually initiated landmarks for tracking are unlikely to coincide with optimal (in the sense of feature tracking) features. Thus, depending on the visibility of interesting anatomical structures or landmarks and their movement and velocity within an endoscopic image sequence, a region of interest (ROI) can be defined around a manually initiated landmark point. As one side condition the ROIs must be defined sufficiently small and in such a way that clearly visible landmarks and the interrelated anatomy lies within the region and independent movements of adjacent ROIs do not interfere with each other. Additionally, the ROIs have to be large enough to cover the possible displacements of consecutive image frames based on the occurring movements. Now, within each manually initiated region, local feature detection and tracking is applied using well-established feature tracking approaches, which are promising for clinical (real-time) applications: the Kanade-Lucas-Tomasi (KLT) feature tracker [9,10] and speeded up robust features (SURF) [11]. SURF tracking is realized by descriptor comparison within the ROI. Within our evaluation framework, these matching algorithms are applied to the manually selected and initiated ROIs to recognize corresponding feature points in consecutive frames. Assuming that detected feature points vary from the given landmarks, but still describe the surrounding anatomic region of interest, the resulting movements can be used to estimate the movement of the original landmark.
To evaluate the proposed approach, it has been applied to five monocular endoscopic video image sequences of different organs. These image sequences cover the surface of a human liver, the view into the human bladder, tissue of the colon, a native and as well as an artificial heart valve, cf. Figure 1. The movements within those sequences vary from almost pure translation of the endoscope (e.g. liver) up to complex deformations of the organ (e.g. heart valves). To prove the principal concept and the correctness of the implementation, a well-behaving test sequence with only one moving item (a ball-pen) was recorded. Specifically, this rigid object being different from the organic structures and being displaced over time with a constant and stable background has been chosen to be independent from application related restrictions in the tracking task, where several movements of landmarks, organs and tissue background are overlaying each other. As ground truth data for comparison and evaluation, the selected landmarks have been labeled manually in all frames of the sequences and the corresponding coordinates have been stored. Due to the intention to estimate the movement of a certain anatomical structure and not to track the exact landmark, the spatial distance between the ground truth points and the tracked features has not been considered as a meaningful error measure. Instead, the movement m of the tracked Points Tr in frame i, relative to the initial key frame, is determined as
For each of the endoscopic image sequences, the movement of the ground truth (GT) was compared to the detected motion using both the SURF as well as the KLT descriptors. The resulting trajectories of all six sequences are shown in Figure 2. Ideal tracking with respect to the manually labeled ground truth data would result in identical trajectories. The more parallel the trajectories of the tracking approaches are, compared to the trajectory of the ground truth, the better the result can be regarded. In addition, both tracking approaches were evaluated by manually rating the consistency of feature tracking. For each sequence, we counted the number of frames, in which the organic structure (i.e. initially selected feature) was tracked correctly, independently of the pixel based ground-truth distance. Table 1 depicts the results obtained.
As can bee seen in Figure 2(a), as well as in Table 1, upper row, the movements within the reference ball-pen test sequence could be tracked with only small deviations. Thus, the principle approach of exploiting robust feature descriptors for manually defined and initiated landmarks seems to be promising. However, applying the proposed tracking method to real medical endoscopic image data, the results differ. For the liver (Fig. 2(c)) and both heart valve sequences (Fig. 2(d+e)), the movements of the manually selected landmarks can only be roughly estimated. In the case of the bladder (Fig. 2(b)), the difference between the ground truth and the tracked movement is increasing, while for the colon (Fig. 2(f)), the original movement is hardly identified in the tracking trajectories. Interestingly, in some sequences (bladder, native heart valve) the KLT tracker yields closer results to the ground truth while in other sequences (colon, artificial heart valve) the SURF features seem better. The rating depicted in Table 1 confirms these results. Although none of the approaches yielded better results throughout all image sequences, the KLT approach showed better stability than the SURF approach in most cases. Dealing with medical image data leads to various possible sources of error, e.g. low contrast images with significant noise, substantial organ deformations, as well as structured surfaces and specular highlights. Figure 3 shows an example for a tracking result applying the KLT tracker to a recording of a native heart valve. As can be seen, the cusps are tracked during the opening phase of the heart valve but lost while closing the orifice. Even though the proposed approach is a promising concept and a good start for further research, a clinical use of landmark based tracking in medical monocular endoscopic image data will demand substantial further improvements. Besides applying and evaluating enhanced tracking and correspondence methos for landmark tracking in monocular endoscopic images, a technical alternative for such scenarios could be the use of stereoscopic endosopes [12]. Nevertheless, such stereoscopic imaging have much larger diameters and can thus not be applied for all applications, and are currently only available as rigid endoscopes.