This paper presents the algorithms and results of our participation to the medical image annotation task of ImageCLEFmed 2008. Our previous experience in the same task in 2007 suggests that combining multiple cues with different SVM-based approaches is very effective in this domain. Moreover it points out that local features are the most discriminative cues for the problem at hand. On these basis we decided to integrate two different local structural and textural descriptors. Cues are combined through simple concatenation of the feature vectors and through the Multi-Cue Kernel. The trickiest part of the challenge this year was annotating images coming mainly from classes with only few examples in the training set. We tackled the problem on two fronts: (1) we introduced a further integration strategy using SVM as an opinion maker. It consists in combining the first two opinions on the basis of a technique to evaluate the confidence of the classifier's decisions. This approach produces class labels with “don't know” wildcards opportunely placed; (2) we enriched the poorly populated training classes adding virtual examples generated slightly modifying the original images. We submitted several runs considering different combination of the proposed techniques. Our team was called “idiap”. The run using jointly the low cueintegration technique, the confidence-based opinion fusion and the virtual examples, scored 74.92 ranking first among all submissions.
The rapid development of new medical image acquisition techniques and the widespread of computerized equipment to save transfer and store medical imagery in digital format have led to the need for new methods to manage and archive this data. Average-sized radiology departments produce nowadays several tera-bytes of data annually. Automatic image annotation systems turn out to be important tools to manage big databases, in avoiding manual classification errors and helping in image retrieval. The ImageCLEFmed challenge is an international event which gives the possibility to different research groups to benchmark their image annotation approaches. The aim is to find out how well current techniques can identify image modality, body orientation, body region and biological system examined based on the images.
In 2008 the ImageCLEFmed annotation task provided participants with 12076 x-ray images as training data spread across 197 classes. The task consisted in assigning the correct label to 1000 test images. To recognize these images, an automatic annotation system have to face two major problems: the intra-class variability vs inter-class similarity, and the data unbalance. The first problem arises from the fact that images belonging to the same visual class might look very different while images that belong to different visual classes might look very similar. The second one is connected to the natural disposition of organs in the human body and the frequency of diseases, which causes that some parts of the body are more likely to be object of image acquisition. The ImageCLEFmed organizers decided to focus on this second problem introducing in the training set 82 classes with a maximum of 6 images each and preparing a test set mainly with images from this low populated classes.
This paper describes the algorithms submitted by the “idiap” team as its second participation
to the CLEF benchmark competition1. Last year we proposed different cue-integration approaches
based on Support Vector Machine (SVM, [
This year we decided to reuse both the above described methods changing the selected features
into two different types of local descriptors: Scale Invariant Feature Transform (SIFT, [
We submitted several runs, the results show that the classification performance increases passing from the use of a single cue (idiap-LBP score 128.58; idiap-SIFTnew score 100.27) to that of multiple cues (LOW lbp siftnew score 93.20), from the use of a hard decision (idiap-MCK pix sift score 313.01; LOW lbp siftnew score 93.20) to a soft decision through confidence based opinion fusion (idiap-MCK pix sift 2MARG score 227.82; LOW 2MARG score 83.79) and gets even better adding virtual examples in low populated classes (idiap-LOW MULT 2MARG score 74.92).
The rest of the paper is organized as follows: section 2 describes the feature extracted from images and the two methods used to combine them. Section 3 gives details on the confidence based opinion fusion, while section 4 explains how we multiplied images to create virtual examples. Section 5 reports the experimental procedure adopted and the results obtained. Conclusions and outlook are given in Section 6.
1In 2007 the name was “BLOOM” due to our sponsors.
The aim of the automatic image annotation task is to classify images into a set of classes based
on the hierarchical IRMA code [
We propose to extract a set of features from each image and to use then SVM to classify them.
In the previous editions of the challenge, top-performing methods were based on the assumption
that images consist of parts which can be modelled more or less independently. That methods used
local features which thus seem to be the most discriminative cues for medical image annotation
[
In 2007 for the medical annotation task we defined and used a modified version of the classical
SIFT descriptor that we called modSIFT [
The two runs based on this feature ranked third and fourth in 2007, so we decided to reuse
it doing only a slight modification, inspired by the approach in [
Another approach that we explored was considering local texture features. As the x-ray images
do not contain any color information, texture features play an important role for this task and in
the past challenge editions they were used by several groups [
Our preliminary results on a validation set showed that the best way to use LBP on the medical image database at hand was combining in a two dimensional histogram LBPr8i,8u2 together with LBPr1i6u,122 and concatenating it with the two dimensional histogram made by LBPr1i6u,128 together with LBPr2i4u,222. In this way a feature vector of 648 elements is obtained. Each image is divided in four parts, one vector is extracted from each subimage and from the central area and then they are concatenated producing a vector of 3240 elements (see Figure 2). 2.2
In the computer vision and pattern recognition literature some authors have suggested different
methods to combine information derived from different cues. They can all be reconducted to
one of these three approaches: high-level, mid-level and low-level integration [
Considered our results in the ImageCLEF 2007 [
The Multi-Cue Kernel is a linear combination of kernels each dealing with a single feature. Suppose that for each image Ii, we extract a set of P different cues, Tp(Ii), p = 1 . . . P . Hence we have P different training sets and a corresponding set of P kernels Kp, p = 1 . . . P . The Multi-Cue Kernel between two images, Ii and Ij , is defined as If now we think of the confidence as a measure of unambiguity of the decision, we can define it as the difference between the maximal and the next largest distance: 3
As previously described, the evaluation scheme for the medical image annotation task addresses the hierarchical structure of the IRMA code by allowing the classifier to decide a “don’t know” at any level of the code, independently for each of the four axes. To effectively support this scheme, models which estimate the classifier’s confidence in its decision could be useful, a fortiori if we consider the high unbalancing of the classes in the training set.
Discriminative classifiers usually do not provide any out-of-the-box solution for estimating confidence of the decision, but in some cases they can be transformed in opinion makers on the basis of the value of the used discriminative function. This gives the possibility to derive confidence information and hypothesis ranking from the produced opinions. In case of SVM, it can be done considering the distances between the test samples and the hyperplanes. The evaluation results very efficient due to the use of kernel functions and does not require additional processing in the training phase.
In the One-vs-All multiclass extension of SVM, if M is the number of classes, M SVMs are trained each separating a single class from all remaining ones. The decision is then based on the distances of the test sample, x, to the M hyperplanes, Dj (x), j = 1 . . . M . The final output is the class corresponding to the hyperplane for which the distance is largest:
KMC (Ii, Ij ) =
P X apKp(Tp(Ii), Tp(Ij )) .
p=1 where ap ∈ <+ are weighting factors found through cross validation while determining the optimal separating hyperplane.
On the other hand, in the low-level scheme, the single features vectors are combined in a unique vector, that is normalized to have sum equal to one.
j∗ = argmax Dj (x) .
j=1...M
C(x) = Dj∗ (x) − j=1.m..Max,j6=j∗ Dj (x) . 2.3
For the classification step we used an SVM with an exponential χ2 as kernel, for both the local structural and textural approaches and the cue-integration methods:
K(X, Y ) = exp
i=1 −γ X (Xi − Yi)2 ! .
N
Xi + Yi
The parameter γ was tuned through cross-validation. This kernel has been successfully applied
for histogram comparison and it has been demonstrated to be positive definite [
Even if the labels are hierarchical, we have chosen to use the standard multi-class approaches. This choice is motivated by the finding that, with our features, the error score was higher using an axis-wise classification. (1) (2) (3) (4) The value C(x) can be thresholded for obtaining a binary confidence information. Confidence is then assumed if C(x) > τ for threshold τ .
Hence what we did was considering the first two margins produced by SVM corresponding to the distances of the test samples from the two closest hyperplanes. If the decision is not confident, that is C(x) < τ , then the label corresponding to the first two opinions are compared and where they differ we put a “don’t know” term. We looked for the best threshold considering the results obtained in the preliminary validation phase and we adopt that for the subsequent experiments. 4
To achieve good results in machine learning based classification, it is important to use training
data which are sufficient not only in quality but also in quantity. For an SVM, working with
classes very sparsely populated means that during the training phase, it is forced to individuate
the best hyperplane which separate classes with few examples, to all the rest of the training set.
Obviously the work done by the classifier in that condition can’t be considered really reliable.
To improve the reliability of the classification, we thought to enrich the poorly populated classes.
Using virtual examples, i.e. artificially created images, is a well known method to expand the
training data in an automatic way on the basis of a prior knowledge [
In one of their publications [
Thus for each of the images belonging to poorly populated classes we produced 17 copies using matlab scripts.
Before starting our validation experiments, we studied in-depth how to divide the released database to consider the high unbalancing between classes. We decided to separate the training images in: • rich set: images belonging to classes with more than 10 elements. A total of 11947 images divided in 115 classes; • poor set: images belonging to classes with less than 10 elements. A total of 129 images divided in 82 classes.
From the first group we built 5 disjoint sets, rich traini/rich testi, each with of 11372/575 images, where the test sets were created randomly extracting five images for each of the 115 classes. On the other hand, we used the whole poor set as a second test set. In this way, although the classes with few images are not considered in the training phase, we can evaluate the performance of the classifier to assign to that images the corresponding nearest class in the hierarchy. So we trained the classifier on the rich traini set and tested both on the rich testi and on the poor set, for each of the 5 splits. The error score was evaluated using the program released by the ImageCLEF organizers. The score values were normalized by the number of images in the corresponding test set, producing two average error scores. They were then multiplied by 500 and summed together to produce the value of the score on the test set of the challenge hypothesizing that it would have been constituted half by images from the rich set and half by images form the poor set. The expected value of the score is then defined as the average of the scores obtained on the 5 splits. Each parameter in our methods was found optimizing this expected score.
Our validation experiments started considering the local structural and textural approaches separately, applying the proposed experimental setup. We used the above procedure to select the best kernel parameters for the single-cue SVMs, giving the lowest combined error score described above. We adopted the same procedure for our validation experiments with the two cue-integration schemes.
On top of these experiments we applied, as second step, the confidence based opinion fusion technique described in Section 3. Both the single-cue and the multiple-cue runs were executed using the One-vs-All SVM multiclass extension and saving a file containing for each test image the values of the distances from the separating hyperplanes. We considered these files related to the rich and poor test sets produced by the classification with the best parameters found in the previous phase. The first two higher margins for every test images were subtracted and the difference compared to the threshold τ varying in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. The best threshold was considered that producing the lowest expected score, using the procedure described above.
To evaluate the effect of introducing virtual examples in the classes belonging to the poor set we divided it in two parts: • poor one: images belonging to classes with only one element. A total of 53 images from 53 classes; • poor more: images belonging to classes with more than one element. A total of 76 images from 29 classes.
We only considered the second group and created 6 poor more traini/poor more testi splits of 29/47 images, where the train sets were defined extracting one image from each of the 29 classes. We also introduced virtual examples as described in Section 4 such that each poor more train set was enriched with 29*17=493 images. Then we combined these sets joining rich traini and poor more trainj to build the training set and testing separately on rich testi and poor more testj with i = 1, 2, 3, 4, 5 and j = 1, 2, 3, 4, 5, 6. We executed experiments with this setup and the best kernel parameters obtained form the previous single and multiple-cue experiments. The described procedure, for each i, j couple produced again two classification outputs. The error scores were normalized and combined as described above. We also repeated this group of experiments without introducing the virtual examples and the score resulted lower of about 4 points on average showing that the addition of virtual elements is useful for the classification task.
Finally we applied the confidence based decision fusion on the output of the just presented experiments with the virtual examples in the training set. In this way we obtained our lowest expected score both for the single-cue and the multiple-cue approaches. Note that, independently of the selected feature or combination of features, applying together our two proposed methods always improves the score.
All the parameters of the validation phase were then used to run our submission experiments
on the 1000 unlabelled images of the challenge test set using all the 12076 images of the original
dataset as training. We submitted 9 runs. One of them (idiap-MCK pix sift) consisted simply in
repeating our 2007 winner run, that is combining modSIFT and pixel features through MCK using
exactly the same parameters of last year [
Considering that our validation results did not show great differences between the low-level and the mid-level integration scheme we decided to use just the low-level cue-integration scheme for sake of simplicity. We submitted only one MCK run using both the confidence based opinion fusion and the virtual examples. Hence the remaining runs consisted in: • using the two new cues separately (idiap-SIFTnew, idiap-LBP); • applying cue-integration (idiap-LOW lbp siftnew); • combining cue-integration with the confidence based opinion fusion (idiap-LOW 2MARG); • combining cue-integration with the introduction of virtual examples in the training set (idiap
LOW MULT); • combining cue-integration with the confidence based opinion fusion and the introduction of virtual examples in the training set (idiap-LOW MULT 2MARG, idiap-MCK MULT 2MARG).
The ranking, name and score of our submitted runs together with the score gain respect to the best run of other participants are listed in Table 1. 6
This paper presents a combination of three different strategies to face the medical image annotation in a highly unbalanced database with great inter-vs-intra class variability. The first consists in combining cues through two different SVM approaches corresponding to a low-level and a mid-level integration scheme. The second allows to estimate the confidence of the classifier decision and, on this basis, to assign to a test image the class label corresponding to the hard decision of the classifier, or to a combination of the labels related to the first two produced opinions. The third consists in enlarging the training set through virtual examples defined as modified copies of the images in the less populated classes. The method obtained combining the low-level cue-integration scheme together with the confidence based opinion fusion and the introduction of virtual examples obtained a score of 74.92 ranking first among all submissions.
This work can be extended in many ways. First, it could be interesting to understand if the low-level cue-integration scheme results still better then the mid-level one when the number of combined cues grows. We could for example analyze what happens adding to the two presented local features a global one. Second, we would like to integrate the confidence estimation and the cue integration in a unique strategy. The classifier should measure its own level of confidence and, in case of uncertainty, to seek for extra information considering multiple cues, so to increase its own knowledge only when necessary. Third, here we have introduced virtual examples modifying the original images through translation, rotation and brightness changes. The results prove the effectiveness of this strategy, but we wonder if it is possible to avoid this passage. A solution could be to design different features which are able to capture the information coming from all the modified copies. This would make the classification accurate even working with few images. Finally, in this work we used the hierarchical structure of the data only to put in the image label some “don’t know” terms. Moreover our preliminary results using the axis-wise classification did not produce good results. We want to study more deeply the hierarchical structure to understand if it is possible to exploit it to produce better classification performance. Future work will explore these directions.
This work is part of the EMMA project and was supported by the Hasler Foundation and by the Blanceflor Boncompagni Ludovisi Foundation (www.haslerstiftung.ch, www.blanceflor.se). The support is gratefully acknowledged.