In this paper we present the strategies that were designed and applied by the Institute of Biomedical Engineering and Technology (BMET) team to the liver image annotation task of ImageCLEF 2014. This was the rst year this challenge was held and as such our strategies form the basis for future exploration in this area. The major challenge of the liver annotation task was limited training data, with some annotation labels having very few positive samples and other labels having no samples at all. We propose two strategies for annotating the liver images given the unbalanced nature of the training dataset. Our rst method uses multi-class classi cation scheme where each label has a classi er that is trained to separate it from the other labels. Our second method uses the similarity scores from an image retrieval algorithm as weights for a majority voting scheme, thereby reducing the inherent bias towards labels that have a high number of samples. We also investigate the performance of our methods using di erent feature sets. In total, BMET submitted 8 runs to the ImageCLEF liver annotation task. All of our runs achieved high scores (> 90%) during evaluation. We also achieved the highest score out of all submissions to the ImageCLEF 2014 liver annotation task.
ImageCLEF [
One of the major challenges was the limited amount of training data compared to the number of annotations that needed to be recognised. In particular, there were some annotations that did not occur at all in the dataset. Similarly, there were also instances where all of the training samples had the same annotation. Our methods were thus designed to account for the unbalanced dataset by reducing the bias towards annotations with more samples.
In this paper, we outline our submission to ImageCLEF 2014 liver
annotation challenge. We present two methods for the annotation of liver CT images,
one based on multi-class classi cation and another using image retrieval. Our
aim was to perform the annotation using visual features only. As such, we do
not utilise any of the information in the ONtology of the LIver for
RAdiology (ONLIRA) [
The following terminology is employed in the remainder of this paper. A question refers to a speci c annotation task, i.e., an element of the structured report that needs to be automatically lled. A label is a possible annotation that can be assigned to a question. An answer is the label that is assigned to the question. 2
The training dataset contained 50 CT volumes cropped to the region around the liver; the volumes had varied resolutions (x: 190{308 pixels, y: 213{387 pixels, slices: 41{588) and spacings (x, y: 0.674{1.007mm, slice: 0.399{2.5mm). A mask of the liver pixels and the bounding box for a selected lesion was provided for each image in the training dataset. The training data also included a set of 60 well-established image features (with a total dimensionality of 458) that had been extracted from the images in the dataset. The answers to 73 questions was provided for each training image.
The test dataset contained 10 CT volumes, with varied resolutions and pixel spacings, cropped to the region around the liver. The test data also included a mask of the liver pixels, a bounding box for a lesion and a set of 60 wellestablished image features (with a total dimensionality of 458) for each image in the dataset.
Our aim was to investigate annotation using variations of two methods: support
vector machine (SVM) classi cation [
Method 1: Two stage classi cation using SVMs with linear kernels. Method 2: Two stage classi cation using SVMs with radial basis function (RBF) kernels.
Method 3: Content-based image retrieval.
Method 4: Content-based image retrieval with feature selection. The two-stage classi cation method is described in Section 3.3 and the image retrieval method is described in Section 3.4.
We also investigated the e ect of expanding feature set on all these methods. The feature set expansion is described in Section 3.2.
Our method was applied to a subset of the annotations; seven questions where the label sets were unbounded (e.g., measurements in millimetres or counts of particular objects) were excluded. We also excluded one question that accepted multiple labels as the answer. In total, we answered 64 of the 73 questions. 3.2
We used two feature sets for the annotation challenge. These were: Feature Set 1: The well-established features included with dataset after cleaning as described below (total dimensionality = 446).
Feature Set 2: Feature Set 1 + a bag-of-visual-words (BoVW) features, constructed as described below (total dimensionality = 1446).
Dataset Features The image features in the dataset included features
extracted from the liver, the hepatic vasculature of the liver, and the selected
lesion. Features extracted globally across all lesions were also included. The
features described object shape properties (e.g., volume, surface area, sphericity,
solidity, convexity, Hu shape invariants [
We cleaned the feature data by removing feature dimensions that had a nota-number (NaN) value or that were used to scale other features. We removed the Anatomical Location feature (5 dimensions) of the lesion since one training image had NaN values for this feature. The Hu Moments feature (3 dimensions) of the lesion was also removed for the same reason. We also removed the rst two dimensions (upper and lower bounds) of the Histogram feature of the lesion and the HistogramOfAllLesions (a feature extracted across all lesions). The cleaned feature set had a dimensionality of 446. Readers are directed to the task documentation for more information about the image features. Train Stage 1
Classifiers Training Data
Train Stage 2 Classifiers
A vs Rest A vs B
STAGE 1: 1-v-all classifiers vBs ...
Rest
Multiple Positives?
Yes STAGE 2: 1-v-1 classifiers vAs ...
C Y vs
Z No Answer
Bag-of-Visual-Words We extracted Scale Invariant Feature Transform (SIFT)
descriptors [
We created a visual codebook from the SIFT descriptors extracted from the
training dataset. We randomly sampled 5% of the descriptors and grouped the
subsampled data using k-means clustering with k = 1000. The cluster centres
were treated as the visual words in the codebook. A single visual word was
assigned to every descriptor in both datasets; this assignment was determined
by nding the visual word whose cluster centre with the minimum Euclidean
distance from the descriptor. A BoVW descriptor was then created for every
image using a 1000-bin histogram of the visual words in that image [
Our two stage classi cation approach for image annotation is shown in Figure 1. Each stage consisted of a bank of several SVM classi ers. This two stage approach was repeated separately for each question. Due to the unbalanced training dataset, we expected that the classi ers for labels with low samples would have relatively low accuracy. For this reason, we used the two stage approach to introduce further discriminative power, especially in the case of ties.
Let be a question. Also let L be the set of labels for with jL j = l. For every label A 2 L, we trained a A-vs-rest (1-vs-all) SVM classi er, hence forming l 1-vs-all classi ers. We also trained A-vs-B (1-vs-1) SVMs for every pair of labels A; B 2 L where A 6= B, forming a total of l2 l =2 1-vs-1 classi ers. For every question, our rst stage was composed of the 1-vs-all classi ers and the second stage was composed of the 1-vs-1 classi ers.
After the classi ers have been trained, our annotation process proceeded as follows. An un-annotated image (from the test dataset) was classi ed using the rst stage. If only one of the 1-vs-all SVMs returned a positive classi cation (i.e., there was no tie) then the label corresponding to that classi er was adopted as the answer. If the classi ers in the rst stage assigned multiple labels (i.e., multiple 1-vs-all classi ers returned positive results) then the second stage was activated.
Let L+ L be the set of labels given positive responses by the rst stage of classi ers. During the second stage, we classi ed the un-annotated image using the 1-vs-1 classi ers for all the labels in L+ (i.e., the 1-vs-1 classi ers for the tied labels). A majority voting scheme was used to select the answer.
Two tiebreaker situations remained after both classi cation stages were completed. The ties included the case where the rst stage did not return a positive label and when there was a tie in the vote during the second stage (multiple labels had the highest majority vote). In both of these situations, we set the answer to \other", noted in the task description as the label selected when the radiologist was unsure of the correct annotation. For such ties in questions where \other" 2= L , we selected the label \N/A" if it was available or "false" for questions that expected a boolean answer.
During training, we tested our algorithm on various SVM kernels (linear, quadratic, radial basis function (RBF), multilayer perceptron, polynomial) and parameters using 10-fold cross validation. We discovered that the best overall accuracy was achieved by the RBF kernel with scaling factor equal to 1. There were only ve questions in which the RBF kernel was beaten by other kernels and in each of these cases the di erence was not signi cant. We therefore selected the commonly-used linear kernel and the RBF kernels for our classi cation approach (Methods 1 and 2, respectively). 3.4
Our image retrieval based approach for annotation used the most similar training images to select the answers for an un-annotated image. While the classi cation approach (Section 3.3) trained separate classi ers for each question, the retrieval approach attempted to answer all of the questions together. An overview of the method is shown in Figure 2.
We de ned the similarity of the the unannotated image (U ) and a training image (T ) as the Euclidean distance between their respective feature vectors: vu d s (U; T ) = tuX (ui i=0 ti)2 (1) Training
Data
Calculate Similarity Select n Similar Calculate
Weights
Answers where ui was the i-th feature in the feature vector of U , ti was the i-th feature in the feature vector of T , and d was the dimensionality of the feature set (see Section 3.2). Under this formulation, lower values of s indicated greater similarity with s (U; T ) = 0 implying that U and T were exactly similar.
We selected the n most similar images from the training data. Let S = fs1; :::; sng be the similarity values of these images (sorted from most similar to least similar). A weighted voting scheme was used to select the answer for each question. The weights for the voting scheme were determined from the set of similarity values. The weight wi for the i-th most similar image was calculated as: wi = c
si s1 (2) where c was a constant scaling factor and si 2 S was the similarity value of the i-th most similar image. In our experiments, we empirically set n = 10 and c = 10. Our weighting scheme adjusted the value of the vote based on the calculated similarity. Images with a higher similarity would thus have a stronger vote compared to images with a lower similarity. The weighting was necessary due to the unbalanced nature of the dataset. If a majority voting scheme was used then the labels that had a higher frequency in the dataset would have a higher chance to be selected as the answer (depending on the value of n).
In the case of a tie (multiple labels having the same weighted vote), we set the answer to \other". When \other" was not part of the label set, we selected the label \N/A" if it was available or "false" for questions that expected a boolean answer.
We also designed an alternate question-based version of our retrieval scheme
for annotation; this is noted as Method 4 in Section 3.1. In the alternate scheme,
we applied sequential feature selection [
We submitted eight runs to the ImageCLEF 2014 liver annotation challenge. The runs were created using a combination of the four methods listed in Section 3.1 and the two feature sets listed in Section 3.2. The runs were evaluated on completeness, the percentage of questions that were answered, and accuracy, the percentage of completed questions with a correct answer. Only 65 questions formed part of the evaluation; questions with unbounded labels (e.g., measurements) were not evaluated as part of the 2014 challenge. Table 1 shows the mean completeness and accuracy of our eight runs.
There were 20 registered participants for the ImageCLEF 2014 liver annotation challenge. However, only three groups (including ours) submitted runs for evaluation. A comparison of the groups is shown in Table 2. In 2014, our submission achieved the highest score of all the participants.
The results show that all of our runs achieved high scores (> 0:92). We achieved a completeness score of 0.98 for every run because we always answered 64 of the 65 questions. The question that we excluded from our submission was 1 These results are from http://www.imageclef.org/2014/liver#Results. one that accepted multiple labels as the answer. We could answer this question and achieve a perfect completeness score by removing the tiebreaker from both of our annotation methods. This would be an exception for only this particular question, i.e., all other questions would still go through a tiebreaker process if necessary.
In general, there were no large di erences between the two variants of the classi cation method. That is, the accuracy of two classi cation methods (Method 1 and Method 2) were approximately the same. This suggests that the choice of kernel was not a major factor in the overall accuracy of the annotation. This outcome was contrary to the training stage (as stated in Section 3.3) where our selection of the RBF kernel (Method 2) was due to its higher accuracy compared to the other kernels. We attribute this di erence to the unbalanced training dataset, which may not have re ected the labels of the test dataset. However, our high scores (> 0:93) demonstrate that our classi cation approach for annotation performs well despite the unbalanced training dataset.
The score of the retrieval method with feature selection (Method 4) was equal to or less than that of the retrieval method with no feature selection (Method 3). This result is counter-intuitive as the expectation is that feature selection would improve the accuracy of the annotation. One explanation for this could be that the feature selection reduces the similarity scores calculated during retrieval (since fewer features are used), which in turn negatively impacts the weighted vote by returning voting power to labels with a larger number of training samples.
It is interesting to note that the classi cation methods performed best when using the expanded feature set (Feature Set 2) while the retrieval methods performed best when using the normal feature set (Feature Set 1). This suggests that one of the major considerations in the annotation of the liver is the combination of features and methods. That is, the ndings indicate that one cannot choose a method for annotation without considering which features will be used, and vice versa. 5
This paper described the methods and results of the BMET group's submission
to the liver annotation task of ImageCLEF 2014. Our eight runs investigated
di erent combinations of methods and feature sets. While all of our runs achieved
high scores they also revealed the areas in which our method could be optimised.
Our future work will investigate building associations between image features
and ONLIRA terms to create classi ers for labels with no samples in the training
dataset [
Acknowledgments. The authors would like to thank the organisers of the ImageCLEF 2014 liver annotation task for their assistance in procuring the detailed results.