One of the proposed tasks of the ImageCLEF 2005 campaign has been an Automatic Annotation Task. The objective is to provide the classification of a given set of 1,000 previously unseen medical (radiological) images according to 57 predefined categories covering different medical pathologies. 9,000 classified training images are given which can be used in any way to train a classifier. The Automatic Annotation task uses no textual information, but image-content information only. This paper describes our participation in the automatic annotation task of ImageCLEF 2005.
participate is to promote and encourage multidisciplinary participation in all aspects of information retrieval, no matter if it is text or content based. 2
Automatic image classification or image annotation is an important step when searching for images from a database, as a way to limit the number of results or filter them to increase precision or as a starting point for a guided search.
In the specific context of medical images, the automatic image annotation may be used as part of a diagnosis
support system [
Based on the IRMA (Image Retrieval in Medical Applications) project [
The aim is to find out how well current techniques can identify image modality, body orientation, body region, and biological system examined based on the images. The results of the classification step can be used for multilingual image annotations as well as for DICOM (Digital Imaging and Communications in Medicine) header corrections.
The images are annotated with complete IRMA code, a multi-axial code for image annotation. The IRMA code is currently available in English and German. It is planned to use the results of such automatic image annotation tasks for further, textual image retrieval tasks in the future. However, to simplify the task, only 57 simple class numbers are provided for ImageCLEF 2005. The meaning of each class and the number of images belonging to it is shown in Table 1.
Description cranium, musculosceletal system cranium, facial cranium, musculosceletal system spine, cervical spine, musculosceletal system spine, thoracic spine, musculosceletal system spine, lumbar spine, musculosceletal system arm, hand, musculosceletal system arm, radio carpal joint, musculosceletal system arm, handforearm, musculosceletal system arm, elbow, musculosceletal system arm, upper arm, musculosceletal system arm, shoulder, musculosceletal system chest chest, bones, musculosceletal system abdomen, gastrointestinal system abdomen, uropoietic system abdomen, upper abdomen, gastrointestinal system pelvis, musculosceletal system leg, foot, musculosceletal system leg, ankle joint, musculosceletal system leg, lower leg, musculosceletal system leg, knee, musculosceletal system leg, upper leg, musculosceletal system leg, hip, musculosceletal system cranium, facial cranium, musculosceletal system cranium, neuro cranium, musculosceletal system spine, cervical spine, musculosceletal system The distribution of images is not homogeneous among all classes, with a clear deviation to class 12 (chest) with more than 28% of the training images. This may be cause for concern when building the classifiers and should be taken into account. 3
This task is clearly aimed at image analysis research groups and the areas of expertise of the MIRACLE group don’t include image analysis research. However, as our group did have a strong expertise in automatic learning algorithms applied to different projects mainly in the fields of data, text and web mining, we decided to make the effort and participate in this task to promote and encourage multidisciplinary participation in all aspects of information retrieval, no matter if it is text or content based.
To isolate from the content-based retrieval part of the process, we resorted to GIFT (GNU Image Finding Tool)
[
Our approach is based on the multidisciplinary combination of the usage of GIFT to perform content-based searches and the application of learning techniques over the retrieval results to build a classifier. Our system is divided in two parts: the content-based retrieval component (mainly GIFT) and the learning component, which makes calls to the retrieval component when necessary and uses the results to build the classifier. We think that this is a naive approach in the sense that we had to completely trust the results from the retrieval engine without no possibility or knowledge to change its behaviour. The only margin for improvement was on the learning component of the system, which in fact relied on the retrieval component.
We finally submitted two different runs to be evaluated by the task coordinators.
Unzipping the database with the 9,000 training images provided by the task coordinators results in a structure of 57 directories (Train01 to Train57), which allows to easily know the class of each image simply by parsing the path of the file. GIFT was then used to index the whole set of images, down-scaled to 32x32 pixels. The retrieval component takes two parameters as inputs: a query image and a number of results. It internally makes calls to GIFT with the image as a query, gets the images that are more similar to the query image, and finally returns the given number of top results, each with the filename of the image and its relevance. Although different search algorithms can be integrated as plug-ins in GIFT, only the provided separate normalisation algorithm has been used in our experiments.
This run selects the classification by using a decision table majority classifier [
After several tests using 10-fold cross-validation with the training images, the best results were obtained when assuming N=20 (taking the top 20 results to compute the aggregated class confidence) and using a factor of 1/n (n being the number of result, from 1 to 20) as the weighting function.
This run is based on the previous experiment and applies a K-Nearest-Neighbour classifier [
After several tests using 10-fold cross-validation with the training images, the best results were obtained when assuming K=8 (8 nearest examples) and enabling attribute normalization and no distance weighting. Although we were aware of the non homogeneous training examples among different classes, we didn’t take this fact into account due to lack of time to carry out the experiments. 4
As in the training set, some sampling bias can be observed also in the test set, and, furthermore, differences in the relative distribution between them. We think that this differences may affect the building of learning models and the also the evaluation of the different groups, and it has to be taken into account for next years. The results of the classifiers are shown in Table 2, ranked by error rate (note that each 0.1% corresponds to 1 misclassification).
Run mira20relp57.txt (1) mira20relp58IB8.txt (2) Euclidean distance, 32x32 images, 1-Nearest-Neighbour (3) Error rate 21.4 22.3 36.8 (1) mira20relp57.txt is the Decision table classifier (2) mira20relp58IB8.txt is the Nearest neighbour classifier (3) According to the track organizers, for a 1-Nearest-Neighbour classifier comparing the images downscaled to 32x32 pixels using Euclidean distance, the error rate is 36.8% (which means 368 images were misclassified).
As shown in Table 2, the best result was obtained with the decision table classifier. This error rate greatly improves the baseline of a 1-nearest-neighbour classifier.
The differences between our two runs haven’t still been analysed in detail at this moment, but a possible explanation for the performance loss with the nearest neighbour classifier may be imputed to model overtraining (when selecting the value of the parameters) or to the non homogeneous distribution of the training examples. Figure 2 shows the distribution of error rate for each class for the decision table classifier. The full data is shown in Table 5 (see appendix). Note the lower error rate for the classes with many training examples (such as 12 and 34). This may also lead to think that the model is overfitting the training set. Apart from MIRACLE, other 12 groups participated in this year's evaluation, with more than one submission. Next table shows the results for each group’s best submission. Our best submission misclassifies only 88 images more than the best submission, from RWTH Aachen Computer Science. These results are very satisfactory for us, considering the simple techniques which have been employed in our experiment and also that we are not a group with expertise in image processing as the others. 5
Our naive approach to this task has proved to be valuable and the results are good enough to be comparable to more sophisticated techniques used by other groups. The combination of a “black-box” search using a publicly accessible content-based retrieval engine with a simple classification algorithm based on a decision table with weighted relevance aggregation has turned to provide similar results to other “more complex” algorithms such as nearest-neighbour or not much worse than boosting. This simplicity may be a good starting point for the implementation of a real system.
Regarding the classifier, we think that there is still space for improvement with a more careful training of the model, probably having a better selection of the training set, or introducing extra parameters to model the sampling biases. We will study if another combination of parameter values would have led to better results, controlling the model overtraining.
In addition, there are other techniques which we also want to test, such as decision trees, neural networks or Bayesian algorithms. Another possibility which couldn’t be tested due to lack of time is to apply clustering to filter results before the learning algorithm, which could discriminate noisy classes and thus increase precision. This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01. Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, Ana González-Ledesma, José Mª Guirao-Miras, Sara Lana-Serrano, Paloma MartínezFernández, Ángel Martínez-González, Antonio Moreno-Sandoval and César de Pablo Sánchez.
33.3% 16.7% 44.4% 62.5% 0.0% 0.0% 100.0% 33.3% 100.0% 0.0% 0.0% The last column shows the number of different classes related to the test class, which can be used to study the noise level for the class or its discrimination ability.