Figures play an important role within biomedical publications. A typical and essential rst step toward using images is the detection of compound gures and their separation into panels. In ImageCLEF'16 our team has participated in the compound gure detection and separation tasks, where we utilized a method based on connected component analysis (CCA) to detect and to separate compound gures, while extending CCA in several ways to improve correct detection of subgures while avoiding over-fragmentation. We have also participated in the Sub gure Classi cation task, where we employed an array of global image characteristics and a merge-split strategy coupled with neural network classi ers. We describe here the methods used in each task and analyze the performance of our system.
Our team has participated in the ImageCLEF'16 Medical Compound Figure
Detection, Figure Separation and Sub gure Classi cation tasks [
About 40% to 50% of gures in biomedical documents are compound gures
consisting of multiple panels [
In contrast, rather than look for separators we employ Connected Component
Analysis (cca) [
The above methods do not address stitched compound gures, that is,
gures in which panels directly touch one another and are not separated by gaps.
Santosh, et al [
Compound gures consist of several panels, typically separated by gaps, which
appear as vertical or horizontal light/dark bands; however, such gaps may be
blurry or too thin to recognize. We rst preprocess compound gures by
resizing, adjusting, and cropping them to make the gaps in the images clearer and
broader. We then apply Connected Component Analysis (cca) to separate
compound gures into constituents panels. As part of this step, (to which we refer
throughout the rest of this report as cca), we set thresholds to eliminate small
objects and keep only the main components as individual panels, as we have
done before [
Notably, this process may not be e ective in several cases, namely: individual panels whose contents may not be well-connected, very blurry panels, and stitched compound gures. For the rst two, which often occur in displayed graphics, we apply susan edge detection1 on the original compound gure, thus sharpening the blurry components and keeping the connectivity within the panels. For stitched compound gures, we apply susan edge detection to nd the 1 We have experimented with several edge-detection methods and found susan to have the best performance in this context. positions where adjacent pixels are sharply changed. The boundary can then be detected by nding the peak value of the sum projections that are calculated by summing the pixel values along the horizontal and the vertical directions. Some compound gures are separated by partial gaps, where boundaries do not cross the entire image; we separate such gures by nding the projection with the highest peak value along only one direction. The projections are then calculated for the separated sub- gures, and the separation is recursively repeated until the highest peak of projections falls below a threshold.
As a last step, we employ a separation quality assessment step that we have developed to prevent over- and under-segmentation. It consists of ve steps: 1) Merge overlapping panels; 2) Temporarily eliminate small disconnected components; 3) Recover missing panels; 4) Separation of potentially overlapping regions; 5) Small component recovery. The complete framework is shown in Fig. 1, and is further described below. Image Preprocessing Gaps in compound gures typically separate sub-panels into clear individual components. However, some panels may be positioned too close to one another, or a thin gap may be noisy or blurred, making separation hard. To address this issue, we rst scale-up the original image I, of size m n, to 2m 2n, using nearest neighbor interpolation, which broadens the separating gaps. Notably, gaps in compound gures are not always white or black, that is, the intensity of pixels in the gaps is non-binary. To make the gap clearer, we adjust the intensity of gures, remapping pixel intensity that is below a lower threshold Tlow or above an upper threshold Thigh in the resized image Iresized to 0 and 1 values, respectively. This enhances contrast in the image so that gaps, which are the lightest or the darkest bands in the gure, will be clearer. In our experiments, we set Tlow to 0.05 and Thigh to 0.95.
Connected Component Analysis Compound gure segmentation implies
identifying panels that can be separated by boundaries and contents. We assume
that the gaps among panels are white (which can be reversed later by inverting
pixel values). We thus generate a mask of image foreground M by setting a
threshold t and mapping all grayscale pixels whose value is below t to 0 and all
those whose value exceeds t to 1. In our experiments the threshold t is set to
0:95. We then detect the connected components in M by using the Connected
Component Labeling method [
Using connected component analysis may generate some small boundingboxes due to small and unconnected objects in the image, such as text. We thus set two thresholds to initially eliminate bounding boxes of very small height or width: theight = height=20, twidth = width=20, where width and height are the total gure width and height.
cannot be separated through the step above, we employ a classi er that labels it as either a stitched compound gure or a compound gure with gaps. We de ne here a gap as a row or a column whose minimum gray value is higher than 0.95. If a gap is found in the gure, the latter is labeled as a compound gure with gaps. Otherwise, it is labeled as stitched.
Two kinds of images with gaps cannot be directly separated by the cca method: images that have very blurry components and images that have components with very low internal connectivity. To address the rst issue, we apply susan edge detector to the preprocessed image, which sharpens blurry components in the image. Still, the corresponding susan edge image, denoted Isusan, may have poor connectivity. To enhance connectivity of components in Isusan, we dilate the connected regions within the susan edge image using the minimum gap-width in the image as the dilation factor. After dilating, the connectivity of connected-regions within the dilated susan edge image is increased. We then apply the cca method to the dilated image.
Projection on SUSAN edge image For stitched compound gures separation, identifying panel boundaries is the main challenge. To detect these boundaries we propose a separation method based on the application of susan edge detector to the gure. The edge detector is rst applied to the preprocessed image, which highlights the boundary between panels. Thus the objective becomes that of detecting boundaries shown in the resulting susan-edge image. Given a susan-edge image Isusan, if I(x; y) is a detected edge we have Isusan(x; y) = 1.
Therefore, summing the pixel value along the horizontal and the vertical directions gives rise to two projections: P roj0 and P roj90 . The panel separation is done along the horizontal or the vertical line that goes through the highest projection position. For gures with complex layout, the boundary between panels may not cross the whole gure; in this case we recursively separate the gure along one direction at a time, where the projection peak value is at least 0.7 of the height or the width of the region currently considered for separation. Separation Quality Assessment a. Merge Overlapping Panels: Because cca is based on the connectivity of components in the image, some bounding boxes may overlap. For example, the bounding box of legends may overlap with the bounding box of corresponding line graphs. Therefore, we choose to merge two bounding boxes when their overlap ratio is larger than 0:1. b. Temporarily Eliminate Small Disconnected Components: In the cca separation we try to eliminate noise and text by setting a threshold for the size of bounding boxes. In this step we eliminate the sub gures that are too small when compared with the biggest bounding box from the separation result. When the height or width of a sub gure is less than the 1/5 of maximum height or width of all sub gures, this sub gure will be removed. c. Recover Missing Panels: Due to blurriness in some of the compound gures, some panels will be recognized while others may be omitted/missed in our separation process. Therefore, we propose a missing sub gure recovery method. We assume that the missing panel is similar in size and symmetric in position to present panels. We thus check for each present panel whether there is enough space for another bounding box to its left, right, above or below. The candidate panels are expected to have similar black area ratio, and the boundary of candidate panels are veri ed by checking if all pixels in the boundary have a variance less than 5. d. Separation of Overlapping Regions: We calculate the dark area overlap. If the overlap-rate is less than 0.5 we consider the separation to be incorrect. e. Small Component Recovery: While we have eliminated text and some relatively small regions from our separation result, there may be some essential parts to be eliminated. We will merge those bounding boxes to their closest quali ed bounding box. To do this we employ several rules. 1. If merging changes both height and width of a quali ed bounding box { do not merge. 2. If merging changes more than 20% of height or width of a quali ed bounding box { do not merge. 3. If the change of height or width for a quali ed bounding box is more than 20%, this quali ed bounding box keeps its original size. 4. Each small bounding box can only be merged once.
Last, we experiment with an automatic cropping scheme that simply subtracts the position of the top-left corner of the top-left bounding box, from the coordinates of all bounding boxes. This ensures that all top-left panels are anchored at position [0; 0], which re ects the bounding boxes position in the training data. 2.2
Here we describe each of our submitted runs for the Figure Separation task: { FS.run1: Connected Component Analysis. { FS.run2: Combination of the image preprocessing and Connected Component Analysis. { FS.run3: Combination of separation quality assessment and Connected Component Analysis. { FS.run4: Combination of image processing, separation quality assessment and Connected Component Analysis. { FS.run5: Combination of image preprocessing, separation quality assessment, Connected Component Analysis and Connected component analysis method over results of susan edge image. { FS.run6: Combination of image preprocessing, separation quality assessment,
Connected Component Analysis and projection on susan edge image.
{ FS.run7: The framework proposed in Fig. 1 without adopting small objects
(small component recovery).
{ FS.run8: The framework proposed in Fig. 1 with small component recovery.
{ FS.run9: The results from FS.run8 with automatic cropping employed.
Results from all our runs are shown in Table 1. The simple Connected
Component Analysis approach (originally used in [
This task was rst introduced in 2015 [
Text-based compound gure detection Captions in compound gures often contain speci c delimiters, separating the description of each panel. That is, gure caption that contain at least two of these delimiters are often captions of compound gures. Therefore, we employ a text-based compound gure detection method by extracting these delimiters from the captions. To select the delimiters, we manually analyzed captions of gures from the training set, and identi ed a list of delimiters that frequently occur in compound gure captions. These delimiters are used to detect compound gures, that is, a gure is classi ed as compound if the corresponding caption has more than two of these delimiters. A list demonstrating the typical delimiters used is shown in Table 2.
ure detection method we employ uses the result from the gure separation method (see Section 2) to classify the gures as compound vs. non-compound. If a gure can be separated into multiple panels, it is labeled compound, otherwise, it is labeled non-compound. In addition to the separation-based decision, we extract three features for compound gure detection: { First, the bounding box with maximum area divided by the area of the image (Maximum Bounding box Area f1). { Second, the sum of the number of solid columns and rows in the image (Number of Separating Lines f2). { Last, we form two binary-vectors: a row-based vector where the value 1 is placed at each position in which a solid column starts (0 in all other positions), and a column-based vector where the value 1 is placed at each position in which a solid row starts (0 in all other positions). We multiply each of these vectors by a Gaussian whose standard deviation is proportional to the image size, thus giving more weight to solid rows and columns that are at the center of the gure (as opposed to the margins). We sum all elements of the weighted vectors and take the result as a third feature, (which we call the solid axis matrix, denoted f3).
Combined methods As both caption and image provide important information for deciding whether a gure is compound, we use several combinationmethods based on features extracted from both captions and images.
One combined method is a simple logical 'OR' (union) of the two predictions. In this case, if either the text-based detection method or the image-based method labels a gure as compound, the gure is assigned the compound label.
The second combined method employs a Decision Tree classi er on the
textbased and image-based prediction results. Each gure is represented as a
twodimensional binary vector hr1, r2i, in which r1 is the label assigned by the
text-based method (where 0 denotes non-compound and 1 compound). Similarly,
r2 is the label assigned by the image-based method. All gures in the training
set are used by the J48 algorithm on Weka [
The third combined method employs a Decision Tree classi er on the textbased prediction results, image-based prediction results along with the three additional image features listed before. Thus, each gure is represented as a 5dimensional vector hr1; r2; f1; f2; f3i. r1 and r2 are as in the above decision tree, while f1 , f2 and f3 are the three additional features described earlier. All gures in the training set are used by the J48 algorithm (Weka implemntation) to build a decision tree model. 3.2
We have submitted 10 runs in total for the compound gure detection subtask, as follows: { FD.run01: Uses only the text-based detection method. { FD.run02: Uses the gure separation result from FS.run4 (see Sec. 2.2). { FD.run03: Uses the gure separation result from FS.run7 (see Sec. 2.2). { FD.run04: Represents each gure in the test-set as a 5-dimensional vector based on the results of FD.run2, and the three features mentioned in Sec. 3.1 (Image-Based detection), then employs the Decision Tree classi er built over the training set. { FD.run05: Represents each gure in the test-set as a two-dimensional vector based on the results of FD.run1 and FD.run2, then employs the Decision Tree classi er built over the training set. { FD.run06: Uses logical OR (Union) over the results from FD.run1 and FD.run2. { FD.run07: Represents each gure in the test-set as a two-dimensional vector based on the results of FD.run1 and FD.run3, then employs the Decision Tree classi er built over the training set. { FD.run08: Uses the logical OR relationship (union) over the results from
FD.run1 and FD.run3. { FD.run09: Represents each gure in the test-set as a ve-dimensional vector based on the results of FD.run1, FD.run2 and the three additional features, then employs the Decision Tree classi er built over the training set. { FD.run10: Represents each gure in the test-set as a ve-dimensional vector based on the results of FD.run1, FD.run3 and the three additional features, then employs the Decision Tree classi er built over the training set. 3.3
Our results are given in Table 3. We achieve an accuracy of 90:74% using our combined textual and visual approach that represents images as 5 dimensional vectors using feature extraction, showing that the combined method improves upon the purely visual methods. This result is among the very top achieved on this task this year. { Training a hierarchical classi er using merge-split approach.
{ Fine tuning a pre-trained deep-network.
As our deep learning approach has not proven e ective in this setting, we will not discuss it further in the interest of focusing on the hierarchical approach. 4.1
The complexity of a classi er is typically proportional to the number of classes in the data and the amount of training data required is proportional to the complexity of the classi er. Due to limited training data available, and classi er limitations, some of the classes are not easily distinguished from one another. To combat this we have augmented the ImageCLEF training dataset with images collected from the web. Furthermore we have developed an approach to target speci c cases of easily mis-classi ed images. Given a classi er and training data we use a merge-split approach, in which classes that tend to be easily confused are merged into a single class, and the classi er is re-trained accordingly. Another classi er is then trained to classify (split) each merged class into its constituents. Details of the image features and the classi er used are discussed in the following subsections.
Image Features The global image feature used is a combination of 4 types
of feature-vectors that capture color distribution, contrast, gradient orientations
and distribution of local patterns in an image, as follows:
{ The rst type of feature vectors consist of information regarding the mean
and co-variance of the color histogram. We compute histograms with 10 bins
for each color channel and compute the mean and co-variance across these
histograms. This results in a 10 and a 9-dimensional vector, respectively,
both of which are normalized.
{ The second set of components is based on a Histogram of Oriented Gradients
[
Before feature extraction, the images are preprocessed to remove bright and
dark borders with constant intensity as they do not contribute to image
information content. The cropped images are then re-sized to be of size 64 64.
Classi er and Training We use a neural network multi-class classi er [
The merge-split training approach was used to train a set classi ers. 10 di erent classi ers were trained using random initialization and the one that did best on the training data was used on the ImageCLEF test data. We used two approaches for doing this: { Sequential: the training steps described in Section 4.1 were used sequentially to train 10 di erent classi ers and the best among them was picked; { Parallel: the merge step was repeated 10 times and the best classi er was chosen before performing the split step 10 times with the selected set of classes. We trained models on both the original training data and our augmented training data. The results on imageCLEF 2016 test data are shown below.
We submitted 4 runs with classi ers trained using the merge-split method: { Run 1: Sequentially trained classi er on ImageCLEF training data. { Run 2: Sequentially trained classi er on augmented training data. { Run 3: Parallel trained classi er on ImageCLEF training data. { Run 4: Parallel trained classi er on augmented training data. We also submitted 3 runs with classi ers trained on the augmented dataset using our deep learning approach (Runs 5-7). These are di erent models that were trained in the same manner, with varying number of iterations. Table 4 shows results on ImageCLEF test data.
We have presented our approach for addressing the Compound Figure Separation, Compound Figure Detection, and Sub gure Classi cation tasks at ImageCLEF'16 Medical. Our approach for gure separation is based on connected component analysis, and extends it via preprocessing steps, susan edge detection for improved segmentation and for addressing stitched- gure segmentations, and separation quality assessment. For compound gure detection, we have developed and employed text-based, visual and combined methods. The text based method leverages common delimiters typically found in captions of compound gures. The visual separation techniques employ our gure separation approach to identify images that can or cannot be split into panels. We use decision trees to combine the methods. We were the only team to submit an entry to the Figure Separation task, and our results on the Compound Figure Detection task are among the very top this year.
For sub gure classi cation we developed an approach based on global features extracted from the images. The features include statistics derived from color, gray level, edge orientation and local patterns. Once images were represented through these features, we used a merge-split scheme and trained neural network classi ers. All the techniques presented throughout this paper o er important capabilities to document analysis approaches that seek to extract information from gures.
This work was partially supported by NIH grant R56LM011354A.