Automated image analysis and interpretation within computer assisted minimally invasive surgery (MIS) most often depend and rely on manually defined landmarks, visible in endoscopic views. More specific, within many types of applications, such landmarks must be tracked automatically during the intervention. Typical feature tracking approaches are able to track slightly changing landmarks over time, as they occur in endoscopic image sequences, but are originally most often designed to track automatically detected salient points. In this contribution an approach is presented, where the advantages of feature descriptors and corresponding matchers can be used to track manually defined landmarks. Based on such initiated landmark points, local feature detection and tracking utilizing SURF or KLT features as descriptors, is executed. Within the region of interest as a constraint, movements of the detected features can be used to approximate the original landmark movements.
Robust feature descriptors and the corresponding matching algorithms are well known and established methods to
identify and track prominent identical points in sequences of consecutive image frames. These methods have been optimized
to track feature points with specific characteristics in monocular image sequences. In contrast, manually initiated
landmarks for tracking are unlikely to coincide with optimal (in the sense of feature tracking) features. Thus, depending on
the visibility of interesting anatomical structures or landmarks and their movement and velocity within an endoscopic
image sequence, a region of interest (ROI) can be defined around a manually initiated landmark point. As one side
condition the ROIs must be defined sufficiently small and in such a way that clearly visible landmarks and the interrelated
anatomy lies within the region and independent movements of adjacent ROIs do not interfere with each other.
Additionally, the ROIs have to be large enough to cover the possible displacements of consecutive image frames based on the
occurring movements. Now, within each manually initiated region, local feature detection and tracking is applied using
well-established feature tracking approaches, which are promising for clinical (real-time) applications: the
KanadeLucas-Tomasi (KLT) feature tracker [
Within our evaluation framework, these matching algorithms are applied to the manually selected and initiated ROIs to recognize corresponding feature points in consecutive frames. Assuming that detected feature points vary from the given landmarks, but still describe the surrounding anatomic region of interest, the resulting movements can be used to estimate the movement of the original landmark. 3
To evaluate the proposed approach, it has been applied to five monocular endoscopic video image sequences of different organs. These image sequences cover the surface of a human liver, the view into the human bladder, tissue of the colon, a native and as well as an artificial heart valve, cf. Figure 1. The movements within those sequences vary from almost pure translation of the endoscope (e.g. liver) up to complex deformations of the organ (e.g. heart valves). To prove the principal concept and the correctness of the implementation, a well-behaving test sequence with only one moving item (a ball-pen) was recorded. Specifically, this rigid object being different from the organic structures and being displaced over time with a constant and stable background has been chosen to be independent from application related restrictions in the tracking task, where several movements of landmarks, organs and tissue background are overlaying each other. As ground truth data for comparison and evaluation, the selected landmarks have been labeled manually in all frames of the sequences and the corresponding coordinates have been stored. Due to the intention to estimate the movement of a certain anatomical structure and not to track the exact landmark, the spatial distance between the ground truth points and the tracked features has not been considered as a meaningful error measure. Instead, the movement m of the tracked Points Tr in frame i, relative to the initial key frame, is determined as For each of the endoscopic image sequences, the movement of the ground truth (GT) was compared to the detected motion using both the SURF as well as the KLT descriptors. The resulting trajectories of all six sequences are shown in Figure 2. Ideal tracking with respect to the manually labeled ground truth data would result in identical trajectories. The more parallel the trajectories of the tracking approaches are, compared to the trajectory of the ground truth, the better the result can be regarded. (d) (e) (f) Figure 2: Trajectories of the movements for the six evaluated recordings for Speeded Up Robust Features (SURF, +), Kanade-Lucas-Tomasi (KLT, ×), and the annotated ground truth (GT, *): ball-pen test sequence (a), view inside a bladder (b), surface of a liver (c), native heart valve (d), artificial heart valve (e), and colon tissue (f).
Ball pen Bladder Liver Native heart valve Artificial heart valve Colon
KLT 100 % 32 % 75 %
4 % 25 % 13 %
SURF 100 % 14 % 5 % 9 % 0 % 29 % In addition, both tracking approaches were evaluated by manually rating the consistency of feature tracking. For each sequence, we counted the number of frames, in which the organic structure (i.e. initially selected feature) was tracked correctly, independently of the pixel based ground-truth distance. Table 1 depicts the results obtained. 4
As can bee seen in Figure 2(a), as well as in Table 1, upper row, the movements within the reference ball-pen test sequence could be tracked with only small deviations. Thus, the principle approach of exploiting robust feature descriptors for manually defined and initiated landmarks seems to be promising. However, applying the proposed tracking method to real medical endoscopic image data, the results differ. For the liver (Fig. 2(c)) and both heart valve sequences (Fig. 2(d+e)), the movements of the manually selected landmarks can only be roughly estimated. In the case of the bladder (Fig. 2(b)), the difference between the ground truth and the tracked movement is increasing, while for the colon (Fig. 2(f)), the original movement is hardly identified in the tracking trajectories. Interestingly, in some sequences (bladder, native heart valve) the KLT tracker yields closer results to the ground truth while in other sequences (colon, artificial heart valve) the SURF features seem better. The rating depicted in Table 1 confirms these results. Although none of the approaches yielded better results throughout all image sequences, the KLT approach showed better stability than the SURF approach in most cases. Dealing with medical image data leads to various possible sources of error, e.g. low contrast images with significant noise, substantial organ deformations, as well as structured surfaces and specular highlights.