This paper describes the medical image retrieval and medical image annotation tasks of ImageCLEF 2007. Separate sections describe each of the two tasks, with the participation and an evaluation of major findings from the results of each given. A total of 13 groups participated in the medical retrieval task and 10 in the medical annotation task. The medical retrieval task added two news data sets for a total of over 66'000 images. Tasks were derived from a log file of the Pubmed biomedical literature search system, creating realistic information needs with a clear user model in mind. The medical annotation task was in 2007 organised in a new format as a hierarchical classification had to be performed and classification could be stopped at any confidence level. This required algorithms to change significantly and to integrate a confidence level into their decisions to be able to judge where to stop classification to avoid making mistakes in the hierarchy. Scoring took into account errors and unclassified parts.
ImageCLEF1 [
The enthusiastic participation in CLEF and particularly for ImageCLEF has shown the need for benchmarks and their usefulness to the research community. Again in 2007, a total of 48 groups registered for ImageCLEF to get access to the data sets and tasks. Among these, 13 participated in the medical retrieval task and 10 in the medical automatic annotation task.
Other important benchmarks in the field of visual information retrieval include TRECVID3
on the evaluation of video retrieval systems [
This article focuses on the two medical tasks of ImageCLEF 2007, whereas two other papers
[
The medical image retrieval task has been run for four consecutive years. In 2007, two new databases were added for a total of more than 66’000 images in the collection. For the generation of realistic topics or information needs, log files of the medical literature search system Pubmed were used. 2.1
Again and as in previous years, the medical retrieval task showed to be popular among many research groups registering for CLEF. In total 31 groups from all continents and 25 countries registered. A total of 13 groups submitted 149 runs that were used for the pooling required for the relevance judgments. 2.2
In 2007, the same four datasets were used as in 2005 and 2006 and two new datasets were added.
The Casimage8 dataset was made available to participants [
In 2007, we added two new datasets. The first was the myPACS 13 dataset of 15’140 images and 3’577 cases, all in English and containing mainly radiology images. The second was the Clinical Outcomes Research Initiative (CORI 14) Endoscopic image database contains 1’496 images with an English annotation per image and not per case. This database extends the spectrum of the total dataset as so far there were only few endoscopic images in the dataset. An overview of the datasets can be seen in Table 1
As such, we were able to use more than 66’000 images, with annotations in three different languages. Through an agreement with the copyright holders, we were able to distribute these images to the participating research groups. The myPACS database required an additional copyright agreement making the process slightly more complex than in previous years. 2.3
In 2007, 31 groups from all 6 continents and 25 countries registered for the ImageCLEFmed retrieval task, underlining the strong interest in this evaluation campaign. As in previous years, only about half of the registered groups finally submitted results, often blaming a lack of time for this. The feedback of these groups remains positive as they say to use the data for their research as a very useful resource.
The following groups finally also submitted results for the medical image retrieval task: • CINDI group, Concordia University, Montreal, Canada; • Dokuz Eylul University, Izmir, Turkey; • IPAL/CNRS joint lab, Singapore, Singapore; • IRIT–Toulouse, Toulouse, France; • MedGIFT group, University and Hospitals of Geneva, Switzerland; • Microsoft Research Asia, Beijing, China; • MIRACLE, Spanish University Consortium, Madrid, Spain; 13http://www.mypacs.net/ 14http://www.cori.org
Ultraschallbild mit rechteckigem Sensor.
Ultrason avec capteur rectangulaire. • MRIM–LIG, Grenoble, France; • OHSU, Oregon Health & Science University, Portland, OR, USA; • RWTH Aachen Pattern Recognition group. Aachen, Germany; • SINAI group, University of Jaen Intelligent Systems, Jaen, Spain; • State University New York (SUNY) at Buffalo, NY, USA; • UNAL group, Universidad Nacional Colombia, Bogot`a, Colombia; In total, 149 runs were submitted, with the maximum being 36 of a single group and the minimum a single run per group. Several runs had incorrect formats. These runs were corrected by the organisers whenever possible but a few runs were finally omitted from the pooling process and the final evaluation because trec eval could not parse the results even after our modifications. All groups have the possibility to describe further runs in their working notes papers after the format corrections as the qrels files were made available to all. 2.4
Query topics for 2007 were generated based on a log file of Pubmed15. The log file of 24 hours contained a total of 77’895 queries. In general, the search terms were fairly vague and did not contain many image–related topics, so we filtered out words such as image, video, and terms relating to modalities such as x–ray, CT, MRI, endoscopy etc. We also aimed for the resulting terms to cover at least two or more of the axes: modality, anatomic region, pathology, and visual observation (e.g., enlarged heart).
A total of 50 candidate topics were taken from these and sometimes an additional axis such as modality was added. From these topics we checked whether at least a few relevant images are in the database and once this was finished, 30 topics were selected.
All topics were categorised with respect to the retrieval approach expected to perform best: visual topics, textual (semantic) topics and mixed topics. This was performed by an experienced image retrieval system developer. For each of the three retrieval approach groups, ten topics were selected for a total of 30 query topics that were distributed among the participants. Each topic consisted of the query itself in three languages (English, German, French) and 2–3 example images for the visual part of the topic. Topic images were searched for on the Internet and were not part of the database. This made visual retrieval significantly harder as most images were taken with different collections compared to those in the database and had changes in the grey level or colour values.
Figure 1 shows a visual topic, Figure 2 a topic that should be retrieved well with a mixed approach and Figure 3 a topics with very different images in the results sets that should be well-suited for textual retrieval, only.
Relevance judgments in ImageCLEFmed were performed by physicians and other studennts in the OHSU biomedical informatics graduate program. All were paid an hourly rate for their work. The pools for relevance judging were created by selecting the top ranking images from all submitted runs. The actual number selected from each run has varied by year. In 2007, it was 35 images per run, with the goal of having pools of about 800-1200 images in size for judging. The average pool size in 2007 was 890 images. Judges were instructed to rate images in the pools are definitely relevant (DR), partially relevant (PR), or not relevant (NR). Judges were instructed to use the partially relevant desingation only in case they could not determine whether the image in question was relevant.
One of the problems was that all judges were English speakers but that the collection had a fairly large number of French and German documents. If the judgment required reading the text, judges had more difficulty ascertaining relevance. This could create a bias towards relevance for documents with English annotation. 2.6
This section quickly summarises the main techniques used by the participants for retrieval and the sort of runs that they submitted. We had for the first time several problems with the submissions although we sent out a script to check runs for correctness before submission. In 2006, this script was part of the submission web site, but performance problems had us change this setup. The unit for retrieval and relevance was the image and not the case but several groups submitted case IDs that we had to replace with the first image of the case. Other problems include the change of upper/lower case for the image IDs and the change of the database names that also changed the image IDs. Some groups reused the 2006 datasets that were corrected before 2007 and also ended up with invalid IDs.
The CINDI group submitted a total of 4 valid runs, two feedback runs and two automatic runs, each time one with mixed media and a purely visual run. Text retrieval uses a simple tf/idf weighting model and uses English, only. For visual retrieval a fusion model of a variety of features and image representations is used. The mixed media run simply combine the two outcomes in a linear fashion.
Dokuz Eylul University submitted 5 runs, 4 visual and one textual run. The text runs is a simple bag of words approach and for visual retrieval several strategies were used containing color layout, color structure, dominant color and an edge histogram. Each run contained only one single technique. 2.6.1 2.6.2
DEU 2.6.3
IPAL 2.6.7
LIG 2.6.8
OHSU 2.6.9
RWTH IPAL submitted 6 runs, all of them text retrieval runs. After having had the best performance for two years, the results are now only in the middle of the performance scale. 2.6.4
IRIT The IRIT group submitted a single valid run, which was a text retrieval run. 2.6.5
MedGIFT The MedGIFT group submitted a total of 13 runs. For visual retrieval the GIFT (GNU Image Finding Tool) was used to create a sort of baseline run, as this system had been used in the same configuration since the beginning of ImageCLEF. Multilingual text retrieval was performed with EasyIR and a mapping of the text in the three languages towards MeSH (Medical Subject Headings) to search in semantic terms and avoid language problems. 2.6.6
MIRACLE MIRACLE submitted 36 runs in total and thus most runs of all groups. The text retrieval runs were among the best, whereas visual retrieval was in the midfield. The combined runs were worse than text alone and also only in the midfield.
MRIM–LIG submitted 6 runs, all of them textual runs. Besides the best textual results, this was also the best overall result in 2007.
The OHSU group submitted 10 textual and mixed runs, using Fire as a visual system. Their mixed runs had good performance as well as the best early precision.
The Human language technology and pattern recognition group from the RWTH Aachen University in Aachen, Germany submitted 10 runs using the FIRE image retrieval system. The runs are based on a wide variety of 8 visual descriptors including image thumbnails, patch histograms, and different texture features. For the runs using textual information, a text retrieval system is used in the same way as in the last years. The weights for the features are trained with the maximum entropy training method using the qrels of the 2005 and 2006 queries.
The SINAI group submitted 30 runs in total, all of them textual or mixed. For text retrieval, the terms of the query are mapped onto MeSH, and then, the query is expanded with these MeSH terms.
SUNY submitted 7 runs, all of which are mixed runs using Fire as visual system. One of the runs is among the best mixed runs.
The UNAL group submitted 8 runs, all of which are visual. The runs use a single visual feature, only and range towards the lower end of the performance spectrum.
The combination of runs from RWTH, OHSU, MedGIFT resulted in 13 submissions, all of which were automatic and all used visual and textual information. The combinations were linear and surprisingly the results are significantly worse than the results of single techniques of the participants. For the first time in 2007, the best overall system used only text for the retrieval. Up until now the best systems always used a mix of visual and textual information. Nothing can really be said on the outcome of manual and relevance feedback submissions as there were too few submitted runs.
It became clear that most research groups participating had a single specialty, usually either visual or textual retrieval. By supplying visual and textual results as example, we gave groups the possibility to work on multi-modal retrieval as well. 2.7.1
Automatic Retrieval As always, the vast majority of results were automatic and without any interaction. There were 146 runs in this category, with 27 visual runs, 80 mixed runs and 39 textual submissions, making automatic mixed media runs the most popular category. The results shown in the following tables are averaged over all 30 topics, thus hiding much information about which technique performed well for what kind of tasks.
Visual Retrieval Purely visual retrieval was performed in 27 runs and by six groups. Results from GIFT and FIRE (Flexible Image Retrieval Engine) were made available for research groups not having access to a visual retrieval engine themselves.
To make the tables shorter and to not bias results shown towards groups with many submissions, only the best two and the worst two runs of every group are shown in the results tables of each category. Table 2 shows the results for the visual runs. Most runs had an extremely low MAP (<3% MAP), which had been the case during the previous years as well. The overall results were lower than in preceding years, indiacting that tasks might have become harder. On the other hand, two runs had good results and rivaled, at least for early precision, the best textual results. These two runs actually used data from 2005 and 2006 that was somewhat similar to the tasks in 2007 to train the system for optimal feature selection. This showed that an optimised feature weighting may result in a large improvement! textual retrieval A total of 39 submissions were purely textual and came from nine research groups.
Table 3 shows the best and worst two results of every group for purely textual retrieval. The best overall runs were from LIG and were purely textual, which happened for the first time in ImageCLEF. (LIG participated in ImageCLEF this year for the first time. Early precision (P10) was only slightly better than the best purely visual runs and the best mixed runs had a very high early precision whereas the highest P10 was actually a purely textual system where the MAP was situated significantly lower. (Despite its name, MAP is more of a recall-oriented measure.) mixed retrieval Mixed automatic retrieval had the highest number of submissions of all categories. There were 80 runs submitted by 8 participating groups.
Table 4 summarises the best two and the worst two mixed runs of every group. For some groups the results for mixed runs were better than the best text runs but for others this was not the case. This underlines the fact that combinations between visual and textual features have to be done with care. Another interesting fact is that some systems with only a mediocre MAP performed extremely well with respect to early precision. 2.8
Only three runs in 2007 were in the manual or interactive sections, making any real comparison impossible. Table 5 lists these runs and their performance
Although information retrieval with relevance feedback or manual query modifications are seen as a very important area to improve retrieval performance, research groups in ImageCLEF 2007 did not make use of these categories. Visual retrieval without learning had very low results for MAP and even for early precision (although with a smaller difference from text retrieval). Visual topics still perform well using visual techniques. Extensive learning of feature selection and weighting can have enormous gain in performance as shown by the FIRE runs.
Purely textual runs had the best overall results for the first time and text retrieval was shown to work well for most topics. Mixed–media runs were the most popular category and are often better in performance than text or visual features alone. Still, in many cases the mixed media runs did not perform as well as text alone, showing that care needs to be taken to combine media.
Interactive and manual queries were almost absent from the evaluation and this remains an important problem. ImageCLEFmed has to put these domains more into the focus of the researchers although this requires more resources to perform the evaluation. System–oriented evaluation is an important part but only interactive retrieval can show how well a system can really help the users.
With respect to performance measures, there was less correlation between the measures than in previous years. The runs with the beast early precision (P10) were not close in MAP to the best overall systems. This needs to be investigated as MAP is indeed a good indicator for overall system performance but early precision might be much more what real users are looking for. 3
Over the last two years, automatic medical image annotation has been evolved from a simple
classification task with about 60 classes to a task with about 120 classes. From the very start
however, it was clear that the number of classes cannot be scaled indefinitely, and that the number
of classes that are desirable to be recognised in medical applications is far to big to assemble
sufficient training data to create suitable classifiers. To address this issue, a hierarchical class
structure such as the IRMA code [
The classes in the last years were based on the IRMA code where created by grouping similar codes in one class. This year, the task has changed and the objective is to predict complete IRMA codes instead of simple classes.
This year’s medical automatic annotation task builds on top of last year: 1,000 new images were collected and are used as test data, the training and the test data of last year was used as training and development data respectively. 3.1
The complete database consists of 12’000 fully classified medical radiographs taken randomly from medical routine at the RWTH Aachen University Hospital. 10’000 of these were release together with their classification as training data, another 1’000 were also published with their classification as validation data to allow for tuning classifiers in a standardised manner. One thousand additional images were released at a later date without classification as test data. These 1’000 images had to be classified using the 11’000 images (10’000 training + 1’000 validation) as training data.
Each of the 12’000 images is annotated with its complete IRMA code (see Sec. 3.1.1). In total, 116 different IRMA codes occur in the database, the codes are not uniformly distributed, but some codes have a significant larger share among the data than others. The least frequent codes however, are represented at least 10 times in the training data to allow for learning suitable models.
Example images from the database together with textual labels and their complete code are given in Figure 4.
1121-120-200-700 T: x-ray, plain radiography, analog, overview image D: coronal, anteroposterior (AP, coronal), unspecified A: cranium, unspecified, unspecified B: musculosceletal system, unspecified, unspecified
T: x-ray, plain radiography, analog, overview image D: coronal, anteroposterior (AP, coronal), unspecified A: spine, cervical spine, unspecified
B: musculosceletal system, unspecified, unspecified
1121-127-700-500
TD:: xco-rraoyn,apl,laainnterraodpioogstrearpihoyr,(aAnPa,locgo,roonvaerl)v,ieswupiimneage
A: abdomen, unspecified, unspecified
B: uropoietic system, unspecified, unspecified
1123-211-500-000
T: x-ray, plain radiography, analog, high beam energy
D: sagittal, lateral, right-left, inspiration
A: chest, unspecified, unspecified
B: unspecified, unspecified, unspecified
3.1.1 IRMA Code
Existing medical terminologies such as the MeSH thesaurus are poly-hierarchical, i.e., a code
entity can be reached over several paths. However, in the field of content-based image retrieval,
we frequently find class-subclass relations. The mono-hierarchical multi-axial IRMA code strictly
relies on such part-of hierarchies and, therefore, avoids ambiguities in textual classification [
This results in a string of 13 characters (IRMA: TTTT – DDD – AAA – BBB). For instance, the body region (anatomy, three code positions) is defined as follows: AAA 000 not further specified ... 400 upper extrimity (arm) 410 upper extrimity (arm); hand 411 upper extrimity (arm); hand; finger 412 upper extrimity (arm); hand; middle hand 413 upper extrimity (arm); hand; carpal bones 420 upper extrimity (arm); radio carpal joint 430 upper extrimity (arm); forearm 431 upper extrimity (arm); forearm; distal forearm 432 upper extrimity (arm); forearm; proximal forearm 440 upper extrimity (arm); ellbow ...
The IRMA code can be easily extended by introducing characters in a certain code position, e.g., if new imaging modalities are introduced. Based on the hierarchy, the more code position differ from ”‘0”’, the more detailed is the description. 3.1.2
Hierarchical Classification To define a evaluation scheme for hierarchical classification, we can consider the 4 axes to be independent, such that we can consider the axes independently and just sum up the errors for each axis independently.
Hierarchical classification is a well-known topic in different field. For example the classification
of documents often is done using a ontology based class hierarchy [
Our evaluation scheme is described in the following, where we only consider one axis. The same scheme is applied to each axis individually.
Let l1I = l1, l2, . . . , li, . . . , lI be the correct code (for one axis) of an image, i.e. if a classifier predicts this code for an image, the classification is perfect. Further, let lˆ1I = lˆ1, lˆ2, . . . , lˆi, . . . , lˆI be the predicted code (for one axis) of an image.
The correct code is specified completely: li is specified for each position. The classifiers however, are allowed to specify codes only up to a certain level, and predict “don’t know ” (encoded by *) for the remaining levels of this axis.
Given an incorrect classification at position lˆi we consider all succeeding decisions to be wrong and given a not specified position, we consider all succeeding decisions to be not specified.
We want to penalise wrong decisions that are easy (fewer possible choices at that node) over wrong decisions that are difficult (many possible choices at that node), we can say, a decision at position li is correct by chance with a probability of b1i if bi is the number of possible labels for position i. This assumes equal priors for each class at each position.
Furthermore, we want to penalise wrong decisions at an early stage in the code (higher up in the hierarchy) over wrong decisions at a later stage in the code (lower down on the hierarchy) (i.e. li is more important than li+1).
Assembling the ideas from above in a straight forward way leads to the following equation: with where the parts of the equation account for
XI 1 1 δ(li, lˆi) i=1 |{bzi} |{iz} | ({cz) }
(a) (b) 0 if lj = lˆj ∀j ≤ i δ(li, lˆi) = 0.5 if lj = * ∃j ≤ i 1
if lj 6= lˆj ∃j ≤ i (a) accounts for difficulty of the decision at position i (branching factor) (b) accounts for the level in the hierarchy (position in the string) (c) correct/not specified/wrong, respectively.
In addition, for every code, the maximal possible error is calculated and the errors are normed such that a completely wrong decision (i.e. all positions wrong) gets an error count of 1.0 and a completely correctly classified image has an error of 0.0.
Table 7 shows examples for a correct code with different predicted codes. Predicting the completely correct code leads to an error measure of 0.0, predicting all positions incorrectly leads to an error measure of 1.0. The examples demonstrate that a classification error in a position at the back of the code results in a lower error measure than a position in one of the first positions. The last column of the table show the effect of the branching factor. In this column we assumed the branching factor of the code is 2 in each node of the hierarchy. It can be observed that the errors for the later positions have more weight compared to the real errors in the real hierarchy. 3.2
In the medical automatic annotation task, 29 groups registered of which 10 groups participated, submitting a total of 68 runs. The group with the highest number of submissions had 30 runs in total.
In the following, groups are listed alphabetically and their methods are described shortly. 3.2.1
BIOMOD: University of Liege, Belgium
The Bioinformatics and Modelling group from the University Liege16 in Belgium submitted four
runs. The approach is based on an object recognition framework using extremely randomised trees
and randomly extracted sub-windows [
BLOOM: IDIAP, Switzerland
The Blanceflor-om2-toMed group from IDIAP in Martigny, Switzerland submitted 7 runs. All
runs use support vector machines (either in one-against-one or one-against-the-rest manner).
Features used are downscaled versions of the images, SIFT features extracted from sub-images, and
combinations of these [
16http://www.montefiore.ulg.ac.be/services/stochastic/biomod 3.2.3
Geneva: medGIFT Group, Switzerland
The medGIFT group17 from Geneva, Switzerland submitted 3 runs, each of the runs uses the
GIFT image retrieval system. The runs differ in the way, the IRMA-codes of the top-ranked
images are combined [
CYU: Information Management AI lab, Taiwan The Information Management AI lab from the Ching Yun University of Jung-Li, Taiwan submitted one run using a nearest neighbour classifier using different global and local image features which are particularly robust with respect to lighting changes. 3.2.5
MIRACLE: Madrid, Spain The Miracle group from Madrid Spain18 submitted 30 runs. The classification was done using a 10-nearest neighbour classifier and the features used are gray-value histograms, Tamura texture features, global texture features, and Gabor features, which were extracted using FIRE. The runs differ which features were used, how the prediction was done (predicting the full code, axis-wise prediction, different subsets of axes jointly), and whether the features were normalised or not. 3.2.6
Oregon Health State University, Portland, OR, USA The Department of Medical Informatics and Clinical Epidemiology19 of the Oregon Health and Science University in Portland, Oregon submitted two runs using neural networks and GIST descriptors. One of the runs uses a support vector machine as a second level classifier to help discriminating the two most difficult classes. 3.2.7
RWTHi6: RWTH Aachen University, Aachen, Germany
The Human Language Technology and Pattern Recognition group20 of the RWTH Aachen
University in Aachen, Germany submitted 6 runs, all are based on sparse histograms of image patches
which were obtained by extracting patches at each position in the image. The histograms have
65536 or 4096 bins [
IRMA: RWTH Aachen University, Medical Informatics, Aachen, Germany The IRMA group from the RWTH Aachen University Hospital21, in Aachen Germany submitted three baseline runs using weighted combinations of nearest neighbour classifiers using texture histograms, image cross correlations, and the image deformation model. The parameters used are exactly the same as used in previous years. The runs differ in the way in which the codes of the five nearest neighbours are used to assemble the final predicted code. 3.2.9
UFR: University of Freiburg, Computer Science Dep., Freiburg, Germany
The Pattern Recognition and Image Processing group from the University Freiburg22, Germany,
submitted four runs using relational features calculated around interest points which are later
combined to form cluster cooccurrence matrices [
UNIBAS: University of Basel, Switzerland
The Databases and Information Systems group from the University Basel23, Switzerland submitted
14 runs using a pseudo two-dimensional hidden Markov model to model image deformation in the
images which were scaled down keeping the aspect ratio such that the longer side has a length of
32 pixels [
The results of the evaluation are given in Table 7. For each run, the run-id, the score as described above and additionally, the error rate, which was used in the last years to evaluate the submissions to this task are given.
The method which had the best result last year is now at rank 8, which gives an impression how much improvement in this field was achieved over the last year.
Looking at the results for individual images, we noted, that only one image was classified correctly by all submitted runs (top left image in Fig. 4). No image was misclassified by all runs. 3.4
Analysing the results, it can be observed that the top-performing runs do not consider the hierarchical structure of the given task, but rather use each individual code as one class and train a 116 classes classifier. This approach seems to work better given the currently limited amount of codes, but obviously would not scale up infinitely and would probably lead to a very high demand for appropriate training data if a much larger amount of classes is to be distinguished. The best run using the code is on rank 6, builds on top of the other runs from the same group and uses the hierarchy only in a second stage to combine the four runs.
Furthermore, it can be seen that a method that is applied once accounting for the hierarchy/axis structure of the code and once using the straight forward classification into 116 classes approach, the one which does not know about the hierarchy clearly outperforms the other one (runs on ranks 11 and 13/7 and 14,16).
Another clear observation is that methods using local image descriptors outperform methods using global image descriptors. In particular, the top 16 runs are all using either local image features alone or local image features in combination with a global descriptor.
It is also observed that images where a large amount of training data is available are more far more likely to be classified correctly.
Considering the ranking wrt. to the applied hierarchical measure and the ranking wrt. to the error rate it can clearly be seen that there are hardly any differences. Most of the differences are clearly due to use of the code (mostly inserting of wildcard characters) which can lead to an improvement for the hierarchical evaluation scheme, but will always lead to a deterioration wrt. to the error rate. 3.5
The success of the medical automatic annotation task could be continued, the number of participants is pretty constant, but a clear performance improvement of the best method could be observed. Although only few groups actively tried to exploit the hierarchical class structure many of the participants told us that they consider this an important research topic and that a further investigation is desired. 0.5 0 10
100 code frequency 1000
Our goal for future tasks is to motivate more groups to participate and to increase the database size such that it is necessary to use the hierarchical class structure actively. 4
The two medical tasks of ImageCLEF again attracted a very large number of registrations and participation. This underlines the importance of such evaluation campaigns giving researchers the opportunity to evaluate their systems without the tedious task of creating databases and topics. In domains such as medical retrieval this is particularly important as data access if often difficult.
In the medical retrieval task, visual retrieval without any learning only obtained good results for a small subset of topics. With learning this can change strongly and deliver even for purely visual retrieval fairly good results. Mixed–media retrieval was the most popular category and results were often better for mixed–media than textual runs of the same groups. This shows that mixed–media retrieval requires much work and more needs to be learned on such combinations. Interactive retrieval and manual query modification were only used in 3 out of the 149 submitted runs. This shows that research groups prefer submitting automatic runs , although interactive retrieval is important and still must be addressed by researchers.
For the annotation task, it was observed that techniques that rely heavily on recent developments in machine learning and build on modern image descriptors clearly outperform other methods. The class hierarchy that was provided could only lead to improvements for a few groups. Overall, the runs that use the class hierarchy perform worse than those which consider every unique code as a unique class which gives the impression that for the current number of 116 unique codes the training data is sufficient to train a joint classifier. As opposed to the retrieval task, none of the groups used any interaction although this might allow for a big performance gain.
We would like to thank the CLEF campaign for supporting the ImageCLEF initiative. We would like to thank all the organizations who provided images and annotations for this year’s task, including myPACS.net (Rex Jakobovits) and the OHSU CORI project (Judith Logan).
This work was partially funded by the DFG (Deutsche Forschungsgemeinschaft) under contracts Ne-572/6 and Le-1108/4, the Swiss National Science Foundation (FNS) under contract 205321-109304/1, the American National Science Foundation (NSF) with grant ITR–0325160, and the EU Sixth Framework Program with the SemanticMining project (IST NoE 507505) and the MUSCLE NoE.