NLM at ImageCLEF 2015: Biomedical Multipanel Figure Separation

Abstract

This paper summarizes the participation of the National Library of Medicine (NLM) in the imageCLEF 2015 biomedical multipanel gure separation task. In this task, our method uses two di erent techniques that are employed on the basis of characteristics of the gures: 1) stitched multipanel gure separation; and 2) multipanel gure separation with homogeneous gaps. Fusion of the two techniques achieved an accuracy of 84.64%.

Keywords

• Biomedical articles
• multipanel gure separation
• contentbased image retrieval

Motivation

Medical image retrieval has been considered as an important research domain over the past 20 years [ 1, 2, 6, 15, 17, 21, 22 ]. Figures in the biomedical publications are often composed of multiple panels. Multipanel gures are used as an aid for grouping related visual artefacts for human consumption. However, they may comprise of images from di erent modalities (such as x-ray, MRI, CT, microscopy, graphics). In [ 4, 13 ], authors report an increasing use of visual material in biomedical publications. The average number of gures per article in the reputed biomedical journals ranges from 6 to 31 [7, 23]. More importantly, according to [11, 12, 16], multipanel gures represent about 50% of the gures in the biomedical open access image data sets such as those used in the imageCLEF (URL: http://www.imageclef.org) benchmark. Mixed modality in multipanel gures pose a challenge for image retrieval [ 1, 2, 15, 17 ] and modality classi cation systems [8, 19, 21]. We also note that these gures are not commonly available in biomedical publication datasets as standalone entities that could be readily used by automated systems since rarely do publishers require authors to submit gures (and captions) in separate les for easy access. In other words, most of the gures packaged as a single image le in the article thereby adversely a ecting their accessibility by automatic multimodal indexing systems such as the National Library of Medicine's OPENi system (URL: http://openi.nlm.nih.gov) [14]. In this context, multipanel gure separation is considered as a crucial step for high quality content-based image retrieval (CBIR) [ 3, 5, 6, 14 ]. Therefore, we call this step `a precursor' to biomedical CBIR.

The remainder of the article is organized as follows. Our method is explained in Section 2, where we provide details on two di erent panel-splitting techniques and their fusion. In Section 3, we present testing results and analysis. Finally, we summarize the paper in Section 4. 2 2.1

Methods

Outline

Uniform-space-separated multipanel gures comprise a signi cant subset of the imageCLEF benchmark data. These include regular (images) and graphical (illustrations, charts, plots) type gures. Pixel intensity pro le-based and homogeneitybased (for crossing bands) methods are commonly used (and often su cient) to separate the panels [ 3, 5, 5, 18 ]. Other methods uses optical character recognition (OCR) for stitched or fully connected multipanel gures [ 3, 14 ]. But, their solution is sensitive to common errors generated by the OCR and are rigid about the alignment of sub gure panel labels relative to each other. To the best of our knowledge, no methods have been reported that separate stitched multipanel gures purely from an image analysis standpoint. A primary challenge for image analysis-based techniques is that no clear boundaries and homogeneous gaps exist between fully connected panels. In this imageCLEF 2015 participation [10, 22], we combine two di erent techniques, operating separately to separate both stitched multipanel gures and the multipanel gures with homogeneous gaps. As a preliminary step, we overlook automating gure type selection (fully-connected and with homogeneous gaps), and focus on developing automatic techniques for separating the panels. Automatically detecting the gure types is left for future work. We manually separated the two types of multi-panel gures in the data set (see Fig. 1). Fig. 2 shows an example of stitched multipanel gure and two examples having homogeneous gaps between the panels.

Dataset

Data

Separation

Stitched multipanel figures

Multipanel figures with gap

For stitched (i.e., fully connected) multipanel gures, we apply our previously reported technique [20]. The steps for stitched multi-panel gure separation can be summarized in the following two steps: 1) Line segment detection, and, 2) Line vectorization.

Details on this technique can be found in [20]. For completeness, we summarize the major steps below.

Line segment detection. The line segment detector (LSD) is designed to detect local straight contours (i.e., line segments), from the zones where the grey level changes from dark to light or vice-versa [9]. It uses edge pixel gradients to detect level lines for separating stitched panels. Fig. 3 shows an output of line segment detection.

Line vectorization. This step connects all prominent broken lines along the panel boundaries while eliminating unwanted line segments within the panels. Like other state-of-the-art techniques, it uses pro le-based concept to connect lines from end to end (horizontal and vertical). Projection pro les from a 2D image f (x; y) of size m n can be computed as p = =2 = P1 x m f (x; y) and p =0 = P1 y n f (x; y): To eliminate dominant line segments that are typically resulted from the objects within the panels, we compute their corresponding pro le transform (i.e., p2), which is then normalized by using their mean and standard deviation. As a consequence, the magnitude of the line segments along panel boundaries are more pronounced. To make it e cient, line segments are rst ltered in two orthogonal directions: 0 and 2 , as shown in Fig.3. (a) Line segments (b) Filtered line segments (c) Output (in red)

Multipanel gure separation with homogeneous gaps

Since majority of the multipanel gures in the ImageCLEF 2015 dataset are separated by homogeneous horizontal or vertical crossing bands of uniform color, we apply our previously reported method [ 3 ]. It contains ve distinct modules: 1) Text label extraction, 2) Panel subcaption extraction, 3) Panel segmentation, 4) Panel label extraction, and, 5) Combination of all outputs from the previous modules.

Note that in this participation, considering the dataset, the rst two modules are not included since no gure caption text is provided.

Panel segmentation. The aim of this module is to identify homogeneous gaps (or crossing bands) for separating panels along them. Speci cally, it is composed of ve major steps: 1) image overlay/markup removal; 2) homogenous crossing band extraction; 3) border band (homogenous band that is located on the boundary of the panel) identi cation; 4) low gradient band (a band that does not have a sharp boundary line) removal; and 5) image division based on crossing bands. For images where the homogeneous gaps do not cross end-to-end, two iterations are required. For example, in Fig. 2 (b), the rst iteration (that goes vertically) results three panels, which are still multipanel gures.

Panel label extraction. This module is designed to detect panel labels from each individual panel. It comprises of three steps: 1) panel label segmentation connected components (CCs); 2) CC recognition using OCR; and 3) re nement of OCR results to get panel labels. The module results several candidate sets of panel labels. In the combination module, the panel label candidate sets (obtained via panel label extraction) are matched with the panels (obtained via panel segmentation). The results of panel segmentation can help selecting the best label set while the results of panel label extraction can help splitting a panel further if multiple labels are found within it. For more detailed description, we refer readers to our previous work [ 3 ]. 3

Experiments

3.1

Dataset and evaluation protocol

The imageCLEF 2015 panel segmentation dataset comprises two parts: training and test, composed of 3403 and 3381 images, respectively. It is important to note that our method does not use the training set. It separates every single image independently from test set without training. From the test set, we manually selected 145 images in the category of stitched multipanel gures. For more details about datasets and evaluation protocol, we refer to [10]. 3.2

Results: comparative study

Following the method described in Section 2, we have submitted two di erent runs (designated as run1 and run2). In both runs, stitched multipanel gure separation (see Section 2.2) is combined. As described in Section 2.3, in run1, panel separation is used while in run2, panel label extraction is integrated with panel separation.

Table 1 shows an overall performance evaluation of our system and a comparison with other participants. Our results are reported as 79.85% and 84.64%, for run1 and run2. Since the performance of the stitched multipanel gure separation remains the same in both runs, the performance di erence of approximately 5% in run2 is attributed to panel label extraction. Panel label extraction does not only help improving the panel separation, but can be used to link the panel with its relevant caption fragment. Out of the two runs, we have received a best multipanel separation rate of 84.64%.

Summary

We have participated in imageCLEF 2015 biomedical multipanel gure separation task. We have submitted our test results by combining two di erent techniques. Our rst technique separated panels from stitched multipanel gures, which is motivated by the fact that no state-of-the-art techniques reported any solutions. Our second technique focused on other remaining multipanel gures that are having homogeneous gaps between the panels. Based on the evaluation protocol designed by the organizer [10], our test outperforms the other participants by more than 35%.

Both techniques perform automatically but, their fusion is not since we have manually separated the dataset for them. As next steps, we plan to automatically categorize multipanel gures based on their characteristics into stitched multipanel and multipanel gures having homogeneous gaps.

Acknowledgements

This research was supported by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC). The authors would like to thank Dr. Daekeun You (currently at the University of Michigan Health System) for his prior contributions that are part of the method used. 7. Cooper, M.S., Sommers-Herivel, G., Poage, C.T., McCarthy, M.B., Crawford, B.D., Phillips, C.: The zebra sh fDVDg exchange project: A bioinformatics initiative 77, 439 { 457 (2004) 8. Demner-Fushman, D., Antani, S., Simpson, M.S., Thoma, G.R.: Design and development of a multimodal biomedical information retrieval system. Journal of Computing Science and Engineering 6(2), 168{177 (2012) 9. Grompone von Gioi, R., Jakubowicz, J., Morel, J.M., Randall, G.: LSD: a Line

Segment Detector. Image Processing On Line 2, 35{55 (2012) 10. Garc a Seco de Herrera, A., Muller, H., Bromuri, S.: Overview of the ImageCLEF 2015 medical classi cation task. In: Working Notes of CLEF 2015 (Cross Language Evaluation Forum). CEUR Workshop Proceedings, CEUR-WS.org (September 2015) 11. de Herrera, A.G.S., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S., Muller, H.: Overview of the imageclef 2013 medical tasks. In: Forner, P., Navigli, R., Tu s, D., Ferro, N. (eds.) Working Notes for CLEF 2013 Conference , Valencia, Spain, September 23-26, 2013. CEUR Workshop Proceedings, vol. 1179. CEUR-WS.org (2013) 12. Kalpathy-Cramer, J., Muller, H., Bedrick, S., Eggel, I., de Herrera, A.G.S., Tsikrika, T.: Overview of the CLEF 2011 medical image classi cation and retrieval tasks. In: Petras, V., Forner, P., Clough, P.D. (eds.) CLEF 2011 Labs and Workshop, Notebook Papers, 19-22 September 2011, Amsterdam, The Netherlands. CEUR Workshop Proceedings, vol. 1177 (2011) 13. Licklider, J.C.R.: A picture is worth a thousand words: And it costs... In: Proceedings of the Joint Computer Conference. pp. 617{621. AFIPS '69 (Spring), ACM, New York, NY, USA (1969) 14. Lopez, L.D., Yu, J., Arighi, C.N., Tudor, C.O., Torii, M., Huang, H., Vijay-Shanker, K., Wu, C.H.: A framework for biomedical gure segmentation towards imagebased document retrieval. BMC Systems Biology 7(S-4), S8 (2013) 15. Muller, H.: Medical (visual) information retrieval. In: Information retrieval meets information visualization, winter school book. Springer LNCS, vol. 7757, pp. 155{ 166 (2013) 16. Muller, H., de Herrera, A.G.S., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S., Eggel, I.: Overview of the imageclef 2012 medical image retrieval and classi cation tasks. In: Forner, P., Karlgren, J., Womser-Hacker, C. (eds.) CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Rome, Italy, September 17-20, 2012. CEUR Workshop Proceedings, vol. 1178 (2012) 17. Muller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications - clinical bene ts and future directions. I. J. Medical Informatics 73(1), 1{23 (2004) 18. Murphy, R.F., Velliste, M., Yao, J., Porreca, G.: Searching online journals for uorescence microscope images depicting protein subcellular location patterns. In: Proceedings of the 2nd IEEE International Symposium on Bioinformatics and Bioengineering. pp. 119{128. BIBE '01 (2001) 19. Rahman, M.M., You, D., Simpson, M.S., Antani, S., Demner-Fushman, D., Thoma, G.R.: Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identi cation and classi cation. Int. J. Multimed.

Info. Retr. 3(3), 131{146 (2014) 20. Santosh, K.C., Antani, S., Thoma, G.: Stitched biomedical multipanel gure separation. In: International Symposium on Computer Based Medical Systems (2015) 21. Simpson, M.S., Demner-Fushman, D., Antani, S., Thoma, G.R.: Multimodal biomedical image indexing and retrieval using descriptive text and global feature mapping. Inf. Retr. 17(3), 229{264 (2014) 22. Villegas, M., Muller, H., Gilbert, A., Piras, L., Wang, J., Mikolajczyk, K., de Herrera, A.G.S., Bromuri, S., Amin, M.A., Mohammed, M.K., Acar, B., Uskudarli, S., Marvasti, N.B., Aldana, J.F., del Mar Roldan Garc a, M.: General Overview of ImageCLEF at the CLEF 2015 Labs. Lecture Notes in Computer Science, Springer International Publishing (2015) 23. Yu, H.: Towards answering biological questions with experimental evidence: automatically identifying text that summarize image content in full-text articles. pp. 834{838 (2006)

References

1. Aigrain , P. , Zhang, H., Petkovic , D. : Content-based representation and retrieval of visual media: A state-of-the-art review . Multimedia Tools and Applications 3 ( 3 ), 179 { 202 ( 1996 )

2. Akgul, C.B., Rubin , D.L. , Napel , S. , Beaulieu , C.F. , Greenspan , H. , Acar , B. : Content-based image retrieval in radiology: Current status and future directions . J. Digital Imaging 24 ( 2 ), 208 { 222 ( 2011 )

3. Apostolova , E. , You , D. , Xue , Z. , Antani , S. , Demner-Fushman , D. , Thoma , G.R. : Image retrieval from scienti c publications: Text and image content processing to separate multipanel gures . Journal of the American Society for Information Science and Technology 64 ( 5 ), 893 { 908 ( 2013 )

4. Aucar , J.A. , Fernandez , L. , Wagner-Mann , C. : If a picture is worth a thousand words, what is a trauma computerized tomography panel worth? The American Journal of Surgery 6 ( 194 ), 734 { 740 ( 2007 )

5. Cheng, B. , Antani , S. , Stanley , R.J. , Thoma , G.R. : Automatic segmentation of sub gure image panels for multimodal biomedical document retrieval . In: Agam, G. , Viard-Gaudin , C . (eds.) Document Recognition and Retrieval XVIII - DRR 2011 , 18th Document Recognition and Retrieval Conference, part of the IS & TSPIE Electronic Imaging Symposium , San Jose, CA, USA, January 24 - 29 , 2011 , Proceedings. SPIE Proceedings , vol. 7874 , pp. 1 { 10 ( 2011 )

6. Chhatkuli , A. , Markonis , D. , Foncubierta-Rodr guez , A., Meriaudeau , F. , Muller, H.: Separating compound gures in journal articles to allow for sub gure classi - cation . In: SPIE, Medical Imaging ( 2013 )