Introduction

In 2005, the medical image annotation task was introduced in the ImageCLEF1 challenge. Its main contribution was to provide a resource for benchmarking content-based image classification systems focusing on medical images. Hospitals collect hundreds of imaging data every day, and automatic image annotation can be an important step when searching for images in huge databases [8,4]. Automatic techniques able to identify acquisition modality, body orientation, body region, and biological system examined could be used for multilingual image annotations as well as for DICOM header corrections in medical image acquisition routine.

Over the last four years, the medical annotation task evolved in terms of number of images, classes, and classes' framework provided. It was born as a 60 plain class problem [2], grew up to a 120 class problem [7], and became a complex hierarchical class task in 2007 [6]. In 2008, class imbalance was added to foster the use of prior knowledge encoded into the hierarchy of classes [1]. This year we celebrate the fifth medical image annotation task anniversary and we decided to organize its conclusive round as a survey on the last years experience. The idea is to compare the scalability of different image classification techniques as the number of classes grows, their hierarchical structure increase, and badly populated classes appear.

Database and Task Description

As in the past challenge editions, the annotation task was defined on the basis of the IRMA project2 . This year, a database of 12,677 fully classified radiographs, taken randomly from medical routine, was made available as training set. Images are labelled according to four classification label sets considering: For the first two label settings, images are associated to simple raw numbers while in the last two label settings, images are identified by their complete IRMA code (see Section 3). The 1-57 labels used for the first group definition are derived through a high level identification of images in IRMA code terms. Considering a more detailed image annotation and the introduction of some new classes, we pass to 116 and then to 193 classes. The "clutter" class for a specific setting contains all the images belonging to new classes, or images described with a higher level of detail in the final 2008 setting. Participants to the medical annotation task were asked to classify the test images according to all the four label settings. Each group is allowed to submit different runs, but each of them should be based only on one algorithm which should be optimized to face the four different classification problems. The aim is to understand how each algorithm answers to the increasing number of classes and to the unbalancing. The classification results are considered per year and the error score summed to have a final unique way to rank the performance of the submitted runs.

•

IRMA Code

Standardized nomenclature for medical imaging are generally roughly structured, ambiguous, and often use optional tags. Concerning the needs for content-based image retrieval (CBIR) and annotation in the medical field, a detailed unambiguous coding scheme is required. Valid relations between code and sub-code elements could be "is-a" and "part-of", defining a strict hierarchical order. Causality is also important for grouping of processing strategies. Therefore, a monohierarchical scheme is required, where each sub-code element is connected to only one code element. Since categorization of medical images must cover all aspects influencing the image content and structure, a multi-axial scheme is needed [5].

The IRMA code strictly relies on these rules. It is composed from four axes having three to four positions, each in {0, . . . , 9, a, . . . , z}, where "0" denotes "unspecified " to determine the end of a path along an axis:

• the technical code (T) describes the image modality;

• the directional code (D) models body orientations;

• the anatomical code (A) refers to the body region examined;

• the biological code (B) describes the biological system examined. This results in a string of 13 characters (IRMA: TTTT-DDD-AAA-BBB). A small exemplary excerpt from the anatomy axis of the IRMA code is given in Table 1. The IRMA code can be easily extended by introducing characters in a certain code position, e.g., if new image modalities are introduced. Based on the hierarchy, the more code positions differ from "0", the more detailed is the description.

Error Evaluation

We now define the error score for the medical image annotation challenge. On the basis of the image labeling, we defined two different evaluation strategies.

2005 and 2006. For these two years, the error is evaluated just on the capability of the algorithm to make the correct decision. There is also the possibility to say "don't know", which is encoded by "*". An example is given in Table 2.

2007 and 2008. For these two years, the error is evaluated on the basis of the hierarchical IRMA code.

Let an image be coded by the technical, directional, anatomical and biological axes. These axes are independent and therefore, the errors for each axis simply are summed up: • let lI 1 = l1 , l2 , . . . , li , . . . , lI be the classified code (for one axis) of an image;

• let l I 1 = l 1 ,

where l i is specified precisely for every position, and in li is allowed to say "don't know", which is encoded by "*". Note that I (the depth of the tree to which the classification is specified) may be different for different axes and different images.

Given an incorrect classification at position li , we consider all succeeding decisions to be wrong and given a not specified position, we consider all succeeding decisions to be not specified. Furthermore, we do not count any error if the correct code is unspecified and the predicted code is a wildcard. In that case, we do consider all remaining positions to be not specified.

We want to penalize wrong decisions that are easy (fewer possible choices at that node) over wrong decisions that are difficult (many possible choices at that node), we can say, a decision at position l i is correct by chance with a probability of 1 bi if b i is the number of possible labels for position i. This assumes equal priors for each class at each position.

Furthermore, we want to penalize wrong decisions at an early stage in the code (higher up in the hierarchy) over wrong decisions at a later stage in the code (lower down on the hierarchy) (i.e. l i is more important than l i+1 ).

Putting together:

I i=1 1 b i (a) 1 i (b) δ(l i , li ) (c)(1)

with

δ(l i , li ) =    0 if l j = lj ∀j ≤ i 0.5 if l j = * ∃j ≤ i 1 if l j = lj ∃j ≤ i(2)

where the parts of the equation:

(a) accounts for difficulty of the decision at position i (branching factor);

(b) accounts for the level in the hierarchy (position in the string);

(c) correct/not specified/wrong, respectively.

In addition, for every axis, the maximal possible error is calculated and the errors are normalized such that a completely wrong decision (i.e. all positions for that axis wrong) gets an error count of 0.25 and a completely correctly predicted axis has an error of 0.00. Thus, an image where all positions in all axes are wrong has an error count of 1.00, and an image where all positions in all axes are correct has an error count of 0.00. Finally setting a wildcard "*" instead of a "0" is not considered a mistake (see Table 3).

Clutter in 2005, 2006 and 2007. For these three years, we introduced a class called "clutter" (C). Even if in the test set there are images belonging to this class, their classification do NOT influence the error score for the challenge (see Table 4). An example of the released database complete labeling is given in Table 5: Examples of the four years labels settings.

Participation

In 2009, in total seven groups from five nations of two continents participated in the medical annotation task, and 19 runs were submitted in total. In the following, we briefly describe the methods applied by the participating groups.

TAUbiomed. The Medical Image Processing Lab from Tel Aviv University in Israel submitted one run using a multiple-resolution patch-based bag-of-visual words approach. Classification is performed through support vector machines. The code hierarchy is completely neglected and no wildcards "*" were used.

Idiap. The Idiap Research Institute from Switzerland submitted four runs reproposing the same strategies used in 2008. They consisted of different classification schemes for support vector machines and coupling two different image descriptors.

FEITIJS. The Faculty of Electrical Engineering and Information Technologies from the University of Skopje in Macedonia submitted one run. It is based on global and local image descriptors, which are classified using bagging and random forest.

VPA. The Computer Vision and Pattern Analysis Laboratory from Sabanci University in Turkey submitted five runs. They used local binary patterns as features and support vector machine as classifier. They adopted a hierarchical approach considering, when applicable, the four IRMA code axes separately.

medGIFT. The medGIFT group from University Hospitals of Geneva in Switzerland submitted three runs using different descriptors and voting schemes in the medGIFT image retrieval system.

DEU. The Dokuz Eylul University in Turkey participated submitting four runs. Different global and local features are extracted from images and classification is performed with a k-Nearest Neighbor algorithm.

IRMA. As in the previous years, the Image Retrieval in Medical Application (IRMA) group at RWTH Aachen University, Germany, provided a baseline run. It was defined using Tamura Texture Measures, Cross Correlation Features, and the Image Distortion Model. The parametrization was unchanged over all the years to provide a general reference. Therefore, the IRMA code hierarchy is disregarded.

Results and Discussion

The results of the challenge evaluation are given in Table 6, sorted by error score sum over the four year label setting. Considering the error score per-year, the group ranking does not change except for the first and second rank positions between the Idiap and TAU group in 2006, respectively. In general, analyzing the results it can be seen that the top-performing runs do not consider the hierarchical structure of the given task (2007 and 2008 labels), but rather use each individual code as one class and train a plain classifier. To assess the semantics captured in the code hierarchy, local rather global image features are required to narrow the semantic gap [3]. In addition, the local features should be associated to segmented image objects rather than squared patches.

Comparing the 2005 and 2006 results, we see that there is a general decrease in the error score. A possible explanation is that in 2005 the 57 classes are wide, each one containing different sub-levels in terms of IRMA codes. This make them difficult to be modeled by a classifier in the training phase. On the other hand, comparing the 2007 and 2008 results there is a general increase in the error score. This effect was expected: here new classes with the same level of detail respect to the IRMA code are added passing from 2007 to 2008. Moreover, some of the new classes are poorly populated in the training set.

As a final remark, we notice that methods using patch-based local image descriptors and discriminative SVM classification methods outperform the other approaches.

Conclusions

We have presented the ImageCLEF 2009 medical image annotation task. This is its conclusive round and we organized it as a survey on the last four years experience. We want to compare the scalability of different image classification techniques as the number of classes grows, their hierarchical structure increase, and badly populated classes appear. A plain classification scheme using support vector machine and local descriptors outperformed the other methods. The obtained scores range from 852.8, over 1994.84, to 3979.8 for best, baseline and worst respectively.