This paper describes the participation of MIRACLE research consortium at the ImageCLEF Medical Image Annotation task of ImageCLEF 2007. Our areas of expertise do not include image analysis, thus we approach this task as a machine-learning problem, regardless of the domain. FIRE is used as a black-box algorithm to extract different groups of image features that are later used for training different classifiers in order to predict the IRMA code. Three types of classifiers are built. The first type is a single classifier that predicts the complete IRMA code. The second type is a two level classifier composed of four classifiers that individually predict each axis of the IRMA code. The third type is similar to the second one but predicts a combined pair of axes. The main idea behind the definition of our experiments is to evaluate whether an axis-by-axis prediction is better than a prediction by pairs of axes or the complete code, or vice versa. We submitted 30 experiments to be evaluated and results are disappointing compared to other groups. However, the main conclusion that can be drawn from the experiments is that, irrespective of the selected image features, the axis-by-axis prediction achieves more accurate results not only than the prediction of a combined pair of axes but also, in turn, than the prediction of the complete IRMA code. In addition, data normalization seems to improve the predictions and vector-based features are preferred over histogram-based ones.
FIRE (Flexible Image Retrieval Engine) [2] [3] is a freely available content-based information retrieval system developed under the GNU General Public License that allows to perform query by example on images, using an image as the starting point for the search process and relying entirely on the image contents. FIRE offers a wide repertory of available features and distance functions. Specifically, the distribution package includes a set of scripts that extracts different types of features from the images, including color/gray histograms, invariant features histograms, Gabor features, global texture descriptor, Tamura features, etc.
Our approach to the task is to build different classifiers that use image features to predict the IRMA code. For that purpose, all images in the training, development and testing dataset have been processed with FIRE. The extracted features have been arranged in three groups, as shown in Table 1, to build the training data matrixes for the classifiers.
Histogram
Vector
Complete
(
Different strategies have been evaluated, using several multiclassifiers built up with a set of specialized individual classifiers:
IRMA Code Classifier: single classifier that uses the image features to predict the complete IRMA code (4 axes: Technical, Direction, Anatomical and Biological).
IRMA Code Axis Classifier: a two level classifier that is composed of four different classifiers that individually predict the value of each axis of the IRMA code; the prediction is the concatenation of partial solutions.
IRMA Code Combined Axis Classifier: similar to the axis classifier, this one predicts the axes grouped in pairs.
These classifiers are all based on the K-Nearest-Neighbour algorithm [8], with K=10, to predict the output class. The main idea behind the definition of the experiments is to evaluate whether an axis-by-axis prediction is better than a prediction by pairs of axes or the complete code, or vice versa. In addition, the effect of applying the data normalization will be also analyzed.
Finally we submitted 30 experiments to be evaluated, described in Table 2.
Run Identifier MiracleA MiracleAA MiracleAATABD MiracleAATBDA MiracleAATDAB MiracleH MiracleHA MiracleHATABD MiracleHATBDA MiracleHATDAB MiracleV MiracleVA
Combined axis: T+A and B+D Combined axis: T+B and D+A Combined axis: T+D and A+B Complete code Axis-by-axis Combined axis: T+A and B+D Combined axis: T+B and D+A Combined axis: T+D and A+B Complete code Axis-by-axis Combined axis: T+A and B+D Combined axis: T+B and D+A Combined axis: T+D and A+B Complete code Axis-by-axis Combined axis: T+A and B+D Combined axis: T+B and D+A Combined axis: T+D and A+B (1) IRMA code axes are: Technical (T), Direction (D), Anatomical (A) and Biological (B).
(
Results are shown in Table 3. The “Error count” column contains the experiment score as computed by the task organizers [1]. This score is defined to penalize wrong decisions that are easy to take (i.e., there are few possible choices at that node) over wrong decisions difficult to take (i.e., there are many possible choices at that node). Furthermore, it also penalizes wrong decisions at an early stage in the code (higher up in the IRMA code hierarchy) over wrong decisions at a later stage (lower down in the hierarchy). The “Well-Classified” column shows the actual number of images with correct predicted codes.
In addition, data normalization seems to improve the predictions and vector-based features are preferred over histogram-based ones.
Comparing to other groups, our results were considerably worse. The best experiment reached a score of 26.84, 17% of our own best error count. MIRACLE ranked 9th out of 10 participants in the task.
The main conclusion that can be drawn from the evaluation is that, irrespective of the selected image features, the best experiments are those that predict the IRMA code from the individual partial predictions of the 1-axis classifiers. Moreover, the predictions of combined pairs of axes are better than the predictions of the complete IRMA code. By extension, it could be concluded that the finer granularity of the classifier, the more accurate 800 600 400 200 0 634 581 701 655 predictions are achieved. In the extreme case, the prediction may be built up from 13 classifiers, one per each character of the IRMA code. This issue will be further investigated and some experiments are already planned. One of the toughest challenges to face when designing a classifier is the selection of the vector of features that best captures the different aspects that allow to distinguish one class from the others. Obviously, this requires an expert knowledge of the problem to be solved, which we currently lack. We are convinced that one of the weaknesses of our system is the feature selection. Therefore more effort will be invested in improving this topic for future participations.
This work has been partially supported by the Spanish R&D National Plan, by means of the project RIMMEL
(Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the
Madrid’s R&D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of
Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267.
[1] Deselaers, Thomas; Kalpathy-Cramer, Jayashree; Müller, Henning; Deserno, Thomas. Hierarchical
classification for ImageCLEF 2007 Medical Image Annotation. On line
http://www-i6.informatik.rwthaachen.de/~deselaers/imageclef07/hierarchical.pdf [Visited 10/08/2007].
line
http://www-i6.informatik.rwth-aachen.de/
[4] Goodrum, A.A. Image Information Retrieval: An Overview of Current Research. Informing Science, Vol
3(
Müller, Henning; Deselaers, Thomas; Kim, Eugene; Kalpathy-Cramer, Jayashree; Deserno, Thomas; Clough, Paul; Hersh, William. Overview of the ImageCLEFmed 2007 Medical Retrieval and Annotation Tasks. Working Notes of the 2007 CLEF Workshop, Budapest, Hungary, September 2007. [7] Villena-Román, J.; González-Cristóbal, J.C.; Goñi-Menoyo, J.M.; and Martínez Fernández, J.L.
MIRACLE’s Naive Approach to Medical Images Annotation. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005.
Witten, Ian H.; Frank, Eibe. Data Mining: Practical machine learning tools and techniques, 2nd Edition, Morgan Kaufmann, San Francisco, 2005.
The following figures compare the predictions of the complete IRMA code versus partial predictions of combined pairs of axes. Only normalized datasets are shown because they lead to better results. 499 426
DA
234