Figures that are included in biomedical publications play an important role in understanding essential aspects of the paper. Much work over the past few years has focused on gure analysis and classi cation in biomedical documents. As many of the gures appearing in biomedical documents comprise multiple panels (sub gures), the rst step in the analysis requires identi cation of compound gures and their segmentation into sub gures. There is a wide variety ways to detect compound gures. In this paper, we utilize only visual information to identify compound vs non-compound gures. We have tested the proposed approach on the ImageCLEF 2015 benchmark of 10; 434 images; our approach has achieved an accuracy of 82:82%, thus demonstrating the best performance when compared to other systems that use only visual information for addressing the compound gure detection task.
These authors contributed equally to this work.
work can be categorized into two main schemes: The rst is based on the
analysis of peak region detection within the image; the peak region is then used as a
reference to nd separating lines for segmentation [
The second scheme is based on connected components analysis [
The rest of the paper is organized as following. In Section 2, we provide an overview of the datasets. The proposed compound gure detection approach is discussed in Section 3. The analysis of component connectivity based scheme is discussed rst, followed by a presentation of the peak region detection scheme and the fusion scheme. Section 4 presents the experimental results submitted to the ImageCLEF 2015. In the end, conclusions are given in Section 5. 2
In our experiments, we use the dataset provided in the ImageCLEFmed 2015
benchmark. We refer the reader for more details to the respective task
description [
In this report, we rst discuss the proposed compound gure detection scheme, where we illustrate the details of our detection method, utilizing only visual information. As shown in the ImageCLEF15 comparison of the results with those obtained by other systems, our approach achieves the highest level of performance among schemes that use only visual information, while its accuracy is only 2:57% lower than that of the top performing scheme, which combines visual and textual information. 3.1
The rst part of our compound gure detection scheme is based on the analysis
of component connectivity of sub gures in an image [
The general scheme of connected component analysis used in our work is
as follows: First, RGB images are converted into grayscale images. Then we
rescale the pixel intensities in the whole image into values in the range [
After that, we scan the resulting, simpli ed image pixel-by-pixel (top to bottom and left to right). Connected regions in which adjacent pixels share a similar range of intensity values [v0; v1] are identi ed. The connected components labeling operator scans the whole image by moving along each row until it reaches a pixel p which has not been previously labeled. If pixel p is not labeled in the previous stage, we examine two p0s predecessor neighboring pixels directly up (denoted pu), and to the left (pl). The label value assigned to pixel p is based on the comparison with these two neighboring pixels.
After scanning the whole image, each detected component in the gure is
labeled with a di erent value. An important issue for compound gure detection
is to minimize the in uence of false positive area where non-compound regions
are misclassi ed as compound regions. Most of these false positive areas are
caused by the connected text. To address this issue, rather than directly removing
text from the images [
The illustration of the whole scheme is presented in Fig. 4. If more than one sub gure is detected, the given gure is classi ed as compound gure, otherwise not. To handle compound gures separated by black rather than white regions, we invert all images and perform sub gure detection using the same procedure. 3.2
Besides the connected component analysis method described above, we also test
the performance of directly using pixel intensity to segment the gure as used in
works [
Consider an image I represented as a matrix I(x; y), where x is the row index and y is the column index; let W and H be the total number of rows and columns, respectively. Assuming that the sub gures are separated by white margins, the rst step is pixel projections operation along the x-axis and along the y-axis as indicated in Fig. 2. Formally:
Ix = min I(x; y) y2 [1; :::; H ]; Iy = min I(x; y) x2 [1; :::; W ]; (1) that is, Ix is a candidate separating row and Iy is a candidate separating column. The next step is to nd a peak region within Ix and Iy. The peak region indicates an area located within the continuous region whose pixel value is greater than a prede ned threshold. In this work, considering the noise and other in uential factors, the threshold is set to 0:85 times of the maximum pixel intensity in the whole image. By comparing with the threshold, we can nd the peak region along Ix and Iy vector. From this, we obtain the index and the region width. These peak regions are regarded as the margin between sub gures. Based on these detected margins, the sub gure region is then calculated.
For a speci c testing image, to get rid of false positives and minimize the in uence of the text region, two di erent post-processing steps are applied. First, we set a threshold on the minimum area of a detected peak region. Another criterion is to measure the ratio value calculated between the current segmented area and the maximum segment detected. If the ratio is smaller than 0:3, the detected segmentation region is classi ed as false positive. If more than one sub-region is detected, input gure is classi ed as compound, otherwise not. As before, this method assumes that the separation between sub- gures consists of white pixels. To also consider black separators, if the gure is not classi ed as compound, we invert the image and go through the processing steps discussed above again. 3.3
In our work, we also fuse the above two di erent schemes { connected component analysis is used as the rst step and if no compound gure is detected, peak region detection is applied as the second step.
An illustration of the proposed scheme is shown in Fig. 3. Our results showed that connectivity component analysis is good at removing false positives caused by text regions. However, it is not as e ective for detecting compound gures consisting of graph images (e.g. line graphs or diagrams). These types of compound gures can be detected using peak region detection approach. We conducted a standalone comparison between the proposed di erent schemes to evaluate their respective performance. 4
We evaluated the proposed approach on ImageCLEF 2015 benchmark, which includes 10,434 di erent gures. Several illustrations of the experimental results are provided in Fig. 4. The overall accuracy is calculated as accuracy = Cg C 100%, where Cg represents the number of correctly detected gures and C is the total number of samples in the set. In addition, we also consider the well-known recall and precision measures, as shown in Table 1. The latter two measures are calculated as:
P recision =
Recall =
T P T P + F P
T P T P + F N ; ; (2) where T P is the number of true positives (compound gures) detected by the proposed scheme, F P is the number of false positives ( gures that are noncompound, but labeled as compound by our scheme), and F N is the number of false negatives ( gures that are compound, but not detected as such by the proposed approach).
As listed in Table 1, the connected component analysis based scheme performs better than the peak-region detection based scheme. By combining the two di erent schemes, we have obtained an accuracy of 82:82% on the test dataset.
For the sake of completeness, we also demonstrate several cases, shown in
Fig. 5, in which our system fails to detect or to correctly segment a compound
gure. As illustrated by Fig. 5(a), when the boundaries between sub gures are
thin, although our algorithm can correctly classify the given compound gure,
the sub- gure segmentation does not work well. Moreover, segmenting diagrams
remains a challenge, as indicated in Fig. 5(b). Over-segmentation is still a
common problem for this kind of non-compound gures [
In this work, we have studied the problem of compound gure detection. Two di erent schemes, as well as an integration of the two, are evaluated. Our integrated scheme outperforms the other systems that use only visual information, participating in this challenge, by more than 10%. In this challenge, the only system outperforming this system (by 2.57%) used a combination of textual and visual information. 6
This work was supported by NIH Award 1R56LM011354-01A1.
Fig. 3. Illustration of the proposed fusion scheme for compound gure detection.