On Fuzzification of Color Spaces for Medical Decision Support in Video Capsule Endoscopy V. B. Surya Prasath Computational Imaging and Visualization Analysis Lab Department of Computer Science University of Missouri-Columbia Columbia MO 65211 USA Abstract Advances in image and video processing algorithms and the availability of computational resources has paved the way for real-time medical decision support sys- tems to be a reality. Video capsule endoscopy is a di- rect imaging method for gastrointestinal regions and produces large scale color video data. Fuzzification of color spaces can improve contextual description based tasks that are required in medical decision support. We consider abnormalities detection in video capsule en- doscopy using fuzzy sets and logic theory on different colorspaces. Application in retrieval of bleeding detec- tion and polyp vascularization are given as examples of the methodology considered here and preliminary re- sults indicate we obtain promising retrieval results. Introduction Figure 1: Each VCE exam involves a human operator (gas- Video capsule endoscopy (VCE) is a revolutionary imag- troenterologist) looking at various abnormal frames and ing technique which paved the way for unprecedented di- classifying the impressions using linguistic terms. Auto- rect visualization of the gastrointestinal tract without much matic classification using a mapping of such linguistic terms discomfort to patients. A typical colon VCE exam produces to colors using fuzzy logic can be used for various tasks such around 8 hours of color (RGB) video data. For example, Pill- as image retrieval. cam Colon capsule endoscope (Given Imaging, Yoqneam, Israel) produces approximately 30, 000 frames per patient and more than 1.6 million patients worldwide have used capsule endoscopy over the past 10 years. Automatic algo- ample in HSI (Hue, Saturation, Intensity) space is provided rithms can help augment computer aided diagnosis (CAD) in for obtaining a correspondence between colors and human VCE medical decision support systems and can help reduce impressions. In this work, we adapt the ground-work done the burden on gastroenterologists (Rey 2008; Niwa et al. in (Shamoi, Kawanaka, and Inoue 2014) to the VCE color 2008). For example, polyp detection (Figueiredo et al. 2010; space fuzzification for abnormality classifications. We pro- 2011), mucosa surface identification (Prasath, Figueiredo, vide an overview of different color spaces (RGB, CMYK, and Figueiredo 2011; Prasath et al. 2012; Prasath and Del- HSV, La*b*) and their fuzzifications to organize all pos- hibabu 2015b), contrast enhancement (Prasath and Del- sible human operator color perceptions of abnormalities in hibabu 2015a). Nevertheless, designing automatic methods VCE images. Using operator defined impressions expressed for automatically analyzing the VCE imagery via image in terms of linguistic terms we provide retrieval examples. processing and computer vision techniques pose significant We note the overall framework is general in the sense that it challenges as we are dealing with big data. can be expanded with domain knowledge for various related Image processing involves uncertainty quantification tasks. and fuzzy techniques are effective in handling various The rest of the paper is organized as follows. Next section tasks (Kerre and Nachtegael(eds) 2000; Vlachos and Ser- introduces different color spaces useful in VCE imagery and giadis 2006; Shamoi, Inoue, and Kawanaka 2014b; 2014a). fuzzification techniques for representing color perceptions Recently, Shamoi et al (Shamoi, Kawanaka, and Inoue 2014) of human operators using fuzzy sets. Next, we provide some used fuzzification of HSI color space for apparel coordina- example classification results on VCE images for bleeding tion. In (Shamoi, Kawanaka, and Inoue 2014) a case ex- regions. Figure 2: Fuzzy logic and mass assignment theory based mapping of different colors and human (operator) impres- sions of them. Image adapted from (Shamoi, Kawanaka, and Inoue 2014). Color Fuzzification for Abnormalities in video Capsule Endoscopy Fuzzification Appearance of different abnormal regions such as polyps, adenomas, and bleeding in VCE videos under different color spaces provide different linguistic terms for description. (a) RGB (b) CMYK (c) HSV (d) La*b* This can be utilized in the fuzzy logic framework advocated in (Shamoi, Kawanaka, and Inoue 2014), see Figure 2. All Figure 3: Different color spaces can be utilized in identifying three components of a medical decision system, namely, dif- abnormalities in VCE exams. In this example, a bleeding ferent color spaces, impressions based on linguistic terms, region (from a Pillcam R Colon capsule image) is visualized and mapping between them, are all interpretable using fuzzy in different color spaces and the proposal in this paper can logic. In particular we consider an example of bleeding de- play a crucial rule in medical decision support system for tection in VCE, see Figure 3. Note that various color spaces VCE. First row shows RGB, CMYK, HSV, La*b* images can be utilized in the fuzzification framework and we con- and next rows their corresponding individual channels. For sider standard color spaces such as RGB, CMYK, HSV and CMYK space, we only show magenta, yellow, key images La*b*, we refer to (Wyszecki and Stiles 1982) for corre- and omit cyan as it did not provide any visual information in sponding definitions and formulae. Each give a different per- this example. spective of an abnormality, see Figure 3 for an example of bleeding region in RGB, CMYK, HSV and La*b* spaces. Advantages of color (spectral) information can be exploited bleeding regions respectively. Note that for different tasks for different diagnostic decision purposes (Figueiredo et these context dependent color impressions are utilized as a al. 2011; Prasath and Delhibabu 2015a). In this particu- query, for example light bleeding (see Figure 5(a)). lar case of bleeding detection, gastroenterologists tend to mark bleeding regions using linguistic terms such as dark Retrieval red/medium red or pale red. For example, in RGB color Following (Shamoi, Kawanaka, and Inoue 2014) we utilized space (see Figure 3(a)) the bleeding region is darker in green a taxonomy of color impressions adapted for VCE imagery and blue channels and the overall appearance can be charac- based medical decision support systems. Here we describe terized as dark red in the RGB spectrum. Thus, fuzzy logic it for bleeding regions and Table 1 provides the taxonomy and mass assignment theory based mapping of different col- of color impressions in the RGB - Value case, and a similar ors and human (operator) oriented impressions can be uti- table is generated for the polyp vascularization with RGB - lized in making a medical decision support system. Density. Using these taxonomy we follow basic formulae in Figure 4 shows RGB and HSV fuzzy sets which are fuzzy logic such as the intersection (minimum), union (max- used to fuzzify different bleeding regions. In contrast to imum) of two membership functions, the apparel coordination application considered in (Shamoi, Kawanaka, and Inoue 2014), here we use only RGB color (µA ∩ µB)(x) = min{µA(x), µB(x)} space and the Value (from HSV) fuzzy membership func- tions. Hence, the {Red, Green, Blue} and {Dark, Medium, (µA ∪ µB)(x) = max{µA(x), µB(x)} Pale} are the spectral and linguistic terms, respectively. We and α-cut utilized a ground truth marked histogram from two expe- fα = {x : µf (x) ≥ α}. rienced gastroenterologists for various bleeding regions and Figure 6 shows some examples with Red and three linguistic These basic fuzzy formulae are used to fuzzify color spaces terms. Context dependent color impressions in the bleeding and interprets linguistic impressions of colors for composite scenario are light, strong which indicate lighter or stronger cases. We used these formulae along with a map between (a) RGB (a) Input (b) Groundtruth (b) HSV - Value Figure 4: Example fuzzy sets for color spaces RGB and HSV (c) Bleeding based on linguistic labels in bleeding regions classification in VCE. Note for the HSV space we only show the Value channel. color impressions and color in RGB, Value in ranking simi- lar images for bleeding region identification. Similar interpretations are done in the polyp vasculariza- tion and density, tortuosity are used as context dependent impressions. Figure 5 provides example results in bleed- ing and polyp vascularization using linguistic queries alone. (d) Input (e) Groundtruth Figure 6 shows the corresponding ranking mechanism based on linguistic impressions for the bleeding the vasculariza- tion cases. As can be seen, histograms are utilized to iden- tify the top three ranked nearest images matching the Dark red interpretation and the retrieval results are accurate as per gastroenterologist ground truth markings. All the retrieved bleeding regions are from the jejunum area of the gastroin- testinal tract and we only show top three results according to (f) Polyp vasularization color histogram matching, see Figure 6(a). A similar analy- sis with RGB space and polyp vascularization density is un- dertaken and the the query image (Figure 5(d)) is described Figure 5: Two example results in bleeding and polyp vas- as Pale Red Dense and the ranking given in Figure 6(b) ranks cularization. (a): For a given input image of bleeding spot the resultant images according to the density of vasculariza- which is described as (b) Dark Red the system retrieves sim- tion. All the retrieved vascular regions contain dense ves- ilar images containing bleeding regions with (c) decreasing sels and are malignant. We utilized 400 Pillcam R Colon ranks (matching). (d): For a given input polyp image which capsule images for bleeding detection, these were obtained is described as (e) Dense tortuous the system retrieves sim- from 5 different patients, and marked by two gastroenterol- ilar polyp images with (f) decreasing density. Note we only ogists who provided ground truth regions along with bound- the top three retrieval results and the ground truth regions aries separating bleeding from normal mucosa tissue. For are marked by human (operators) and used in training the polyp vascularization based retrieval we used 100 images impressions and their mappings. which are benchmarked against an automatic segmentation method (Prasath, Pelapur, and Palaniappan 2014) for calcu- lating the density of vascularization in polyps. Table 1: Taxonomy of color impressions in the RGB case for bleeding regions in VCE images. Level Impressions Comment I Red, Black Atomic context independent colors II Dark, Medium, Pale Atomic context independent, dependent colors III Combinations of I & II Composite context independent, dependent colors Conclusion In this paper, we considered fuzzification of different color spaces for medical decision support systems in gastroin- (a) Bleeding testinal diagnosis using video capsule endoscopy. Follow- ing (Shamoi, Kawanaka, and Inoue 2014) we utilized fuzzy sets and logic, color space theory for VCE imagery inter- pretation and used it for retrieval tasks. Our preliminary re- sults in bleeding region detection and polyp vascularization in various VCE images indicate promise for using fuzzifica- tion techniques in a medical decision support system. Future works include introducing shape (e.g. polyp appearance), (b) Polyp vascularization texture (e.g. pit patterns) features along with fuzzification framework studied here for different VCE videos. Moreover, increasing the number of experts (in our case study gastroen- Figure 6: Ranking based on linguistic terms and (a) color terologists) and quantifying/enlarging the linguistic impres- histograms in the bleeding region identification (b) Density sions is an important avenue to be explored. We also believe of vascularization in the polyps in decreasing order. the framework considered here will help in identifying trash, bubbles for uninformative frame classification. Prasath, V. B. S., and Delhibabu, R. 2015a. Automatic con- Acknowledgment trast enhancement for wireless capsule endoscopy videos The author thanks the Gastroenterologists Dr. R. Shankar, with spectral optimal contrast-tone mapping. In Compu- Dr. A. Sebastian from Vellore Christian Medical College tational Intelligence in Data Mining - Volume 1, 243–250. Hospital, India for their help in interpreting VCE imagery Springer SIST (eds.: L. Jain, H. S. Behera, J. K. Mandal, D. and Radhakrishnan Delhibabu (Kazan Federal University P. Mohapatra). & Innopolis, Kazan, Russia) in helping with data collec- Prasath, V. B. S., and Delhibabu, R. 2015b. Automatic im- tion/organization. This work was done while the author was age segmentation for video capsule endoscopy. In Compu- visiting the Center for Scientific Computation and Mathe- tational Intelligence in Medical Informatics, SpringerBriefs matical Modeling (CSCAMM) at the University of Mary- in Applied Sciences and Technology, 73–80. Springer CIMI land, MD, USA. (eds.: N. B. Muppalaneni, V. K. Gunjan). References Prasath, V. B. S.; Figueiredo, I. N.; Figueiredo, P. N.; and Palaniappan, K. 2012. Mucosal region detection and 3D Figueiredo, I. N.; Prasath, S.; Tsai, Y.-H. R.; and Figueiredo, reconstruction in wireless capsule endoscopy videos using P. N. 2010. Automatic detection and segmentation of active contours. In 34th IEEE/EMBS International Confer- colonic polyps in wireless capsule imagess. Technical Re- ence, 4014–4017. port 10–37, University of Coimbra, Portugal. Figueiredo, P. N.; Figueiredo, I. N.; Surya Prasath; and Tsai, Prasath, V. B. S.; Figueiredo, I. N.; and Figueiredo, P. N. R. 2011. Automatic polyp detection in pillcam colon 2 cap- 2011. Colonic mucosa detection in wireless capsule endo- sule images and videos: Preliminary feasibility report. Di- scopic images and videos. In Congress on Numerical Meth- agnostic and Therapeutic Endoscopy 2011:7pp. Article ID ods in Engineering (CMNE 2011). 182435. Prasath, V. B. S.; Pelapur, R.; and Palaniappan, K. 2014. Kerre, E., and Nachtegael(eds), M. 2000. Fuzzy techniques Multi-scale directional vesselness stamping based segmen- in image processing. Heidelberg: Physica-Verlag. tation for polyps from wireless capsule endoscopy. In Niwa, H.; Tajiri, H.; Nakajima, M.; and Yasuda, K. 2008. Figshare. New Challenges in Gastrointestinal Endoscopy. Springer. Rey, J.-F. 2008. Future perspectives for esophageal and colorectal capsule endoscopy: Dreams or reality? In New Challenges in Gastrointestinal Endoscopy. Springer. 55–64. Shamoi, P.; Inoue, A.; and Kawanaka, H. 2014a. Fuzzy color space for apparel coordination. Open Journal of Information Systems 1(2):20–28. Shamoi, P.; Inoue, A.; and Kawanaka, H. 2014b. Percep- tual color space: Motivations, methodology, applications. In Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Sympo- sium on Advanced Intelligent Systems (ISIS), 1354–1359. Shamoi, P.; Kawanaka, H.; and Inoue, A. 2014. Fuzzifica- tion of HSI color space and its use in apparel coordination. In Proc. of 25th Modern Artificial Intelligence and Cognitive Science Conference, 11–17. Spokane, WA, USA: CEUR- WS.org. online CEUR-WS.org/Vol-1144/paper2.pdf. Vlachos, I. K., and Sergiadis, G. D. 2006. A heuristic ap- proach to intuitionistic fuzzification of color images. World Scientific. chapter 108, 767–774. Wyszecki, G., and Stiles, W. 1982. Color Science: Concepts and Methods, Quantitative Data and Formulas. New York, NY, USA: Wiley.