Detector of Interest Point within Region of Interest on NBI Endoscopy Images Dmitry M. Stepanov1 , Vyacheslav V. Mizgulin2 , Vsevolod V. Kosulnikov1 , Radi M. Kadushnikov1 , Evgeny D. Fedorov3 , and Olga A. Buntseva4 1 SIAMS Ltd. 2 Ural Federal University named after the first President of Russia B. N. Yeltsin 3 Pirogov Russian National Research Medical University (RNRMU) 4 Lomonosov Moscow State University Abstract. This paper presents a new method for detecting scale in- variant interest point from region of interest on NBI endoscopy images. This work is related to design of decision support systems in the area of gastrointestinal endoscopy with image classication facility. The use of a computer-based system could support the doctor when making a di- agnosis and help to avoid human subjectivity. Our method is based on computing a skeletons of gastric mucosa pit-patterns. As a result of ap- plying the proposed approach to improve the accuracy of a selection of distinctive points, this allow a selection region of interest for endoscopy images in real time. These points are invariant to scale, rotation and translation as well as robust to illumination changes and limited changes of viewpoint. Keywords: image analysis, endoscopy, cancer diagnostic, decision sup- port, point detector 1 Introduction Gastric cancer is the second most lethal cancer in the world. Cancer causes 20% of deaths in the European Region, being the second most important cause of death and morbidity in Europe after cardiovascular diseases with more than 3 million new cases and 1.7 million deaths each year. In many cases cancer can be avoided, and early detection substantially increases the chance of cure. Develop- ments in the field of medical equipment contribute to improvements in quality of diagnosis. Modern endoscopic technologies allow obtaining high-resolution im- ages with distinguishable thin structure of neoplasms. Experts can use these endoscopic images experts to predict histologic structure of a tumor. Clinical decision making systems based on computer-aided image analysis are also ac- tively used to improve diagnostics. One of the most common methods employed by the said systems is image classification on the base of selected visual features. Miyaki et al [8] investigated possibility of applying computer-based analysis to endoscopic images received using zoom-magnification chromoscopy in order to al- low distinction of benign tumors and early stomach cancer. Lee et al [4] presented preliminary results of analyzing stomach suspicious neoplasm images obtained with zoom-magnification narrow band imaging (NBI) endoscopy. Authors used neural network to classify images of stomach mucous membrane. Coimbra et al [10, 11] presented application of classification methods for endoscopic images obtained using chromoscopy. The authors classified affected mucosal areas using suggestions by Dinis-Ribeiro [1]. Tamaki et al [12] made a research in the area of classifying the endoscopic images obtained with zoom-magnification narrow band imaging endoscopy. Authors used the NBI magnification findings classifi- cation scheme, presented in [3]. However, due to complexity of processed images existing solutions require manual selection of interest area. That, in term, does not allow implementation of a real-time decision support system In order to allow automated selection of local features within the interest area it is neces- sary to find an optimal interest point detector. Such a detector should provide the greatest amount of points captured within an interest area, and satisfy the following criteria introduced in [13]: – repeatability; – distinctiveness/informativeness; – locality; – quantity; – accuracy; – efficiency. The purpose of the work was to develop a decision support system for gas- trointestinal endoscopy. It was suggested that this system should possess the following features: – self-training capabilities in order to allow diagnosis of various pathologies using images obtained with various endoscopic methods; – real time operation, in order to allow decision-making in course of inspection, and not after wards; – completely automated implementation work of analysis algorithms that does not require additional operator training. The following article describes the approach taken for detection of local fea- tures within the interest areas of source images obtained with zoom-magnification NBI endoscopy. This approach will allow implementation of image classification that does not require selection of interest areas. Application of a presented in- terest point detector will allow implementing the real-time operation mode for clinical decision support system. 2 Point detection Analysis of endoscopic images is complicated by the presence of various artifacts that appear in course of image acquisition, including highlights (that appear because of an almost concentric arrangement of illuminator and camera, and wet mucous surface of stomach as well), floating scale, geometric deformations (associated with use of a wide-angle lens), and uneven brightness. The presence of artifacts requires image preprocessing, image-specific selection of analysis pa- rameters, and use of invariant methods. Information relevant for the purpose of image classification is contained in several fragments of an analyzed image. Therefore, it is necessary to consider local features that belong to a given region of interest. Basing upon the results of textual analysis presented in [9] that allowed creating the list of key words describing key objects present on endoscopic images, and properties of the said objects, the authors specified properties of the key points that should be selected on endoscopic images, and determined properties of the objects that are impor- tant for decision-making. To highlight these points of interests, authors propose a detector that is invariant for the most distortions of an image, and uses the principle of constructing skeletons for gastric mucosa pit-patterns. The first step of using this detector involves application of image convolution with the Gaus- sian kernel, and allocation of pit-pattern ridges using Difference of Rotating Half Smoothing Filters [5]. The use of similar methods was proposed for extracting features of endoscopic images in [6, 9]. The pit-patterns skeleton was then built using fast fully parallel thinning algorithms [2], and nodes of this skeleton were considered to be interest points. The resulting points were invariant to uneven brightness, geometric deformation and rotation. Scale was selected using meth- ods proposed by Mikolajczyk and Schmid [7]. In this method, a set of images with different resolution represented a so-called scale-space. Different resolution levels were obtained by varying the value of sigma parameter in course of image convolution with Gaussian kernel. The authors then calculated Laplacian re- sponse function values for the selected points of interest at different scale levels. Local maximum of the function was taken to represent a characteristic scale. Fig. 1. Example selection skeleton 3 Experimental results Image collection and annotation and collection software was designed and imple- mented. Endoscopy specialists were able to upload images, manually annotate and enter results of histology. For testing purposes, a dataset consisting of 204 images was collected. Figure 2 displays the expert-annotated image, and the set of points for this image selected using our detector. Fig. 2. The expert-annotated image, and the set of points for this image selected using our detector Results of comparing several popular interest point detectors are presented in table 1. Detectors were implemented using the lip-vireo library5 . Table contains average values of results for all images within the test set. Names of the detectors are given in the first row of the table. The second row demonstrates percentage of detected points. It can be observed that the highest amount of points was detected by the DoG solution. The ratio of correct points to detected points is given in row three, with The Skeleton Nodes detector providing the highest value. Table 1. Row 2: percentage of points for which a characteristic scale is detected. Row 3: percentage of points for which a correct scale is detected with respect to detected points. Difference of Laplacian of Harris- Hessian of Fast Skeleton Gaussians (DoG) Gaussian (LoG) Laplacian Laplacian Hessian Nodes detected 0,84% 0,10% 0,11% 0,10% 0,14% 0,15% correct/ 35,64% 35,58% 39,65% 35,89% 40,6% 45,88% detected 5 http://pami.xmu.edu.cn/˜wlzhao/lip-vireo.htm The results demonstrate that the detector provides good accuracy of selecting points. High share of detecting informative points allows image classification without manual selection of interest regions, and visualization of expected region of interest as shown in figure 3. Fig. 3. Example selection Region of Interest by interest points 4 Conclusions and perspectives Existing solutions require manual selection of interest region, not real-time oper- ation of decision support system. The authors, using the results of analyzing text conclusions, proposed a detector of key points to be used for automatic selection of visual features from the region of interest. The said detector provides the great- est amount of allocated key points from the region of interest. These points are invariant to uneven brightness, geometric deformation, rotation and scale. Im- age annotation and collection software was designed and implemented, allowing medical specialists to upload endoscopy images, manually annotate and input histology results. Further research goals include collection of training samples for all diagnoses with histological confirmation and application of classification methods for endoscopic images. Acknowledgements. The work was done within the framework of the project performed by SIAMS company, and supported by the Ministry of Education and Science of the Rus- sian Federation (Grant agreement 14.576.21.0018 dated June 27, 2014). Project (applied research) unique ID RFMEFI57614X0018. References 1. Dinis-Ribeiro, M., da Costa-Pereira, A., Lopes, C., Lara-Santos, L., Guilherme, M., Moreira-Dias, L., Lomba-Viana, H., Ribeiro, A., Santos, C., Soares, J., et al.: Magnification chromoendoscopy for the diagnosis of gastric intestinal metaplasia and dysplasia. Gastrointestinal endoscopy 57(4), 498–504 (2003) 2. Guo, Z., Hall, R.W.: Fast fully parallel thinning algorithms. CVGIP: Image Un- derstanding 55(3), 317–328 (1992) 3. Kanao, H., Tanaka, S., Oka, S., Hirata, M., Yoshida, S., Chayama, K.: Narrow- band imaging magnification predicts the histology and invasion depth of colorectal tumors. Gastrointestinal endoscopy 69(3), 631–636 (2009) 4. Lee, T.C., Lin, Y.H., Uedo, N., Wang, H.P., Chang, H.T., Hung, C.W.: Computer- aided diagnosis in endoscopy: A novel application toward automatic detection of abnormal lesions on magnifying narrow-band imaging endoscopy in the stomach. In: Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual In- ternational Conference of the IEEE. pp. 4430–4433. IEEE (2013) 5. Magnier, B., Montesinos, P., Diep, D.: Ridges and valleys detection in images using difference of rotating half smoothing filters. In: Advanced Concepts for Intelligent Vision Systems. pp. 261–272. Springer (2011) 6. Majewski, P., Jedruch, W.: Endoscopy images classification with kernel based learning algorithms. Innovations in applied artificial intelligence pp. 167–180 (2005) 7. Mikolajczyk, K., Schmid, C.: Indexing based on scale invariant interest points. In: Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on. vol. 1, pp. 525–531. IEEE (2001) 8. Miyaki, R., Yoshida, S., Tanaka, S., Kominami, Y., Sanomura, Y., Matsuo, T., Oka, S., Raytchev, B., Tamaki, T., Koide, T., et al.: Quantitative identification of mucosal gastric cancer under magnifying endoscopy with flexible spectral imaging color enhancement. Journal of gastroenterology and hepatology 28(5), 841–847 (2013) 9. Mizgulin, V.V., Stepanov, D.M., Kamentsev, S.A., Kadushnikov, R.M., Fedorov, E.D., Buntseva, O.A.: Hybrid classification approach to decision support for en- doscopy in gastrointestinal tract. In: Analysis of Images, Social Networks and Texts, pp. 218–223. Springer (2015) 10. Sousa, A., Dinis-Ribeiro, M., Areia, M., Coimbra, M.: Identifying cancer regions in vital-stained magnification endoscopy images using adapted color histograms. In: Image Processing (ICIP), 2009 16th IEEE International Conference on. pp. 681–684. IEEE (2009) 11. Sousa, R., Ribeiro, M.D., Pimentel-Nunes, P., Tavares Coimbra, M.: Impact of svm multiclass decomposition rules for recognition of cancer in gastroenterology images. In: Computer-Based Medical Systems (CBMS), 2013 IEEE 26th Interna- tional Symposium on. pp. 405–408. IEEE (2013) 12. Tamaki, T., Yoshimuta, J., Kawakami, M., Raytchev, B., Kaneda, K., Yoshida, S., Takemura, Y., Onji, K., Miyaki, R., Tanaka, S.: Computer-aided colorectal tumor classification in nbi endoscopy using local features. Medical image analysis 17(1), 78–100 (2013) 13. Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors: a survey. Foun- dations and Trends R in Computer Graphics and Vision 3(3), 177–280 (2008)