Problem Statement 3-D endoscopy gained high attention as it enables new applications to minimally invasive surgery [1]. Besides structured light [2] and stereo vision [3], Time-of-Flight (ToF) technology was manufactured into a first hybrid 3-D endoscope prototype, recently. In comparison to stereo vision, ToF is independent of texture information. Hence, the endoscope acquires range images with a constant resolution of 64×48 px. As the ToF sensor is manufactured into a conventional endoscope we additionally acquire high-resolution color images with 640×480 px through a common optical system using a beam splitter. Both range and complementary color information can be used to develop robust algorithms for image guided surgery. Haase et al. [4] proposed a tool localization framework that exploits range and color information for increased robustness. Nevertheless, as ToF technology exhibits a low signal-to-noise ratio, preprocessing is a required first step. Different preprocessing techniques have been proposed for ToF range images [5,6], recently. For instrument segmentation approaches based on geometric information [7] or color information [8] have been investigated. This segmentation result can thereupon be used for further applications, e.g. the avoidance of risk situations as proposed in [9]. However, in comparison to purely 2-D driven approaches we are able to incorporate 3-D surface data as well as 2-D photometric data to improve robustness. Our preliminary framework describes a first approach towards entire instrument segmentation on 3-D surface information using a ToF/RGB endoscope. We propose a multi-sensor instrument segmentation framework using super-resolution to denoise ToF data and increase spatial resolution [6]. Our framework exploits a data fusion for range and color images [10]. After upsampling ToF data, segmentation is performed on both modalities and then the results are consolidated into a common segmentation mask.
The proposed segmentation framework is illustrated in Fig. 1 and is subdivided into (a) super-resolution and (b) multisensor segmentation. The preprocessing is applied according to our previous publication [6]. Our approach requires data fusion of range and color images. As the prototype uses a beam splitter to deliver signal to the RGB and the ToF sensor, we map color information to the surface data using a homography that is estimated beforehand using a modified checkerboard [10]. This mapping results in a high-quality RGB image that is aligned to the range image up to a scale-factor.
We cope with the low signal-to-noise ratio of the ToF images by applying super-resolution as described in [6]. The super-resolution approach is subdivided into motion estimation, range correction and numerical optimization. Multi-frame super-resolution employs subpixel displacements between consecutive frames as a cue to obtain a super-resolved image from multiple low-resolution frames. These displacements are induced by navigating the endoscope. Our objective function to obtain a maximum a-posteriori (MAP) estimate � � for a high-resolution image � is described by:
The first sum denotes the data term and the second sum is a regularizer based on a pseudo Huber loss function � � of a high-pass filtered version of the input image �. � weights the regularizer and � denotes the number of low-resolution input frames and � denotes the number of pixels in the super-resolved output image. The data term describes the distance of the k th low-resolution input frame � ��� and a mathematical model of our image acquisition.
The system matrix � ��� incorporates blur induced by the point spread function, downsampling and the displacement field of a high-resolution image �. As the low signal-to-noise ratio of ToF data limits the accuracy of displacement field estimation, we exploit data fusion to estimate a high-quality displacement field in the color domain using optical flow [11] and transfer it into the range domain. As we acquire images from different angles and distances we have to correct the range data, to have all low-resolution range images in the same plane. This correction is modeled by � � ��� and � � ��� .
For more details considering the multi-sensor super-resolution see Köhler et al. [6].
Based on the output of the preprocessing we apply instrument segmentation on data of both modalities. We distinguish between instruments and background by different thresholding techniques [8]. For our segmentation we exploit the fact that instruments are usually closer to the sensor and that instruments are usually grayish. Due to the data fusion in our hybrid 3-D endoscope, we can not only exploit the range data but also incorporate the color information into the segmentation process similar to [9]. Range values � are considered as instruments pixels if � � � � ��� ���. In the color domain we exploit the value and the saturation channel of the HSV color space to segment the instrument. Here, pixels � are considered as instrument pixels if � � � � � ��� �� � � and � � � � � ��� �� � �, where � � and � � denote the saturation channel and the value channel of the color image, respectively. Both binary results are then consolidated into a common segmentation mask by multiplication. For outlier removal caused by noisy data we apply morphological operators to close small holes and remove separated areas with less than 1000 instrument pixels as false instrument candidates.
Our algorithm is evaluated on real data with a realistic liver phantom. Data was acquired with a ToF/RGB endoscope manufactured by Richard Wolf GmbH, Knittlingen, Germany. We assembled realistic scenarios including two different endoscopic instruments. For evaluation we investigated the results in two different scenarios, for 6 frames each. The upsampled images had a resolution of 240×160 px. Our instrument segmentation is compared to segmentation for both modalities separately. For ground truth data, the endoscopic instruments were manually segmented by an expert in the color domain. The threshold parameters were set empirically to , and by analyzing the first frame. This frame was excluded for further evaluation to separate between training and evaluation data.
For quantitative evaluation, we analyzed the sensitivity, the specificity and the F-score of our approach in Table 1. Here, we compare our segmentation results to the results of our framework for a purely range driven approach based on superresolution and for a purely color driven approach.
For qualitative evaluation we illustrate the results of all three approaches in Fig. 2. The benefit of super-resolution for our noisy range data is shown in Fig. 3 with the color overlay encoding the segmentation result. Discussion Table 1 illustrates that our approach results in the best specificity, i.e. only few background pixels are considered as instrument pixels. However, both single-sensor approaches result in satisfying sensitivities, i.e. only few instrument pixels are missing. Nevertheless, the F-score as a measurement of the accuracy indicates a more reliable performance of our approach. Furthermore, as our approach consolidates both modalities, we have a higher robustness considering different threshold parameters. Oversegmentation in one modality can be compensated by the other modality. The qualitative results confirm the comparison in Tab. 1 and highlight that both modalities oversegment the image in areas close to the sensor with surface normals pointing directly to the camera. In those areas the instruments are too close to the tissue to be distinguished in the range image, but specular highlights exclude the use of the color image, likewise. Our approach achieves a reasonable compromise, where only few instrument pixels are missed and oversegmentation is reduced. The 3-D reconstructions show that most parts of the instruments are segmented correctly in our approach and that preprocessing is required to provide any intuitive visualization.
Summary In this paper we proposed an instrument segmentation framework for 3-D ToF/RGB endoscopy. Our method applies robust multi-sensor super-resolution based on motion estimation on high-resolution RGB images to upsample and to denoise low-resolution range images. Due to improved signal-to-noise ratio of the range images we apply instrument segmentation using thresholding techniques and consolidate the results of both modalities. Compared to purely color driven segmentation we improved the F-score from 0.61 to 0.73. Future work will consider different segmentation techniques and refinement of our super-resolution for further denoising. For the consolidation of both sensor results additional weighting factors will be taken into account as proposed in [4]. In experiments on real organs, we will investigate the robustness of our segmentation in real medical scenarios.
We gratefully acknowledge the support by the Deutsche Forschungsgemeinschaft (DFG) under Grant No. HO 1791/7-1. This research was funded/ supported by the Graduate School of Information Science in Health (GSISH) and the TUM Graduate School. The authors gratefully acknowledge funding of the Erlangen Graduate School in Advanced Optical Technologies (SAOT) by the DFG in the framework of the German excellence initiative. We thank the Metrilus GmbH for their support. This project was supported by the research training group 1126 funded by the DFG.