The Visual Concept Detection Task (VCDT) of ImageCLEF 2008 is described. A database of 2,827 images were manually annotated with 17 concepts. Of these, 1,827 were used for training and 1,000 for testing the automated assignment of categories. In total 11 groups participated and submitted 53 runs. The runs were evaluated using ROC curves, from which the Area Under the Curve (AUC) and Equal Error Rate (EER) were calculated. For each concept, the best runs obtained an AUC of 80% or above.
Searching for images is, despite intensive research on alternative methods in the last 20 years, still a task which is mainly done based on textual information. For a long time, searching for images based on text was the most feasible method because on the one hand, the number of images to be searched was rather restricted, and on the other hand, only few people needed to access huge repositories of images. Both of these conditions have changed. The number of available images is growing more rapidly than ever due to the falling prices of high-end imaging equipment for professional use and of digital cameras for consumer use. Publicly available image databases such as Google picassa and Flickr have become major sites of interest on the Internet.
Nevertheless, accessing images is still a tedious task because sites such as Flickr do not allow images to be accessed based on their content but only based on the annotations that users create. These annotations are commonly disorganised, not very precise, and multilingual. Access problems can be addressed by improving the textual access methods, but none of these improvements can ever be perfect as long as the users do not annotate their images perfectly, which is very unlikely. Therefore, content-based methods have to be employed to improve access methods to digitally stored images.
A problem with content-based methods is that they are often costly and cannot be applied in real-time. An intermediate step is to automatically create textual labels based on the images' content. To make these labels as useful as possible, frequently occurring visual concepts should be annotated in a standard manner.
In the visual concept detection task (VCDT) of ImageCLEF 2008, the aim was to apply labels of frequent categories in the photo retrieval task to the images and evaluate how well automated visual concept annotation algorithms function. Additionally, participants of the VCDT could create annotations for all images used in the photo retrieval task, which were provided to the participants of this task. Person
Animal
Water (i.e. river, lake, etc)
Sky
Sunny Partly Cloudy Overcast
Day Night
Road or pathway
Buildings
Beach Mountains
Vegetation
Tree
In the following, we describe the visual concept detection task of ImageCLEF 2008, the database used, the methods of the participating groups, and the results. 2
Database and Task Description
As database for the ImageCLEF 2008 visual concept detection task, a total of 2,827 images were
used. These are taken from the same pool of images used to create the IAPR-TC12 database [
The visual concepts were chosen based on concepts used in work on visual concept annotation. They were organised in a hierarchy, shown in Figure 1.
As for the object detection task in 2007 [
Of the 2,827 manually annotated images, 1,827 were distributed with annotations to the participants as training data. The remaining 1,000 images were provided without labels as test data. The participants' task was to apply labels to these 1,000 images.
Table 1 gives an overview of the frequency of the 17 visual concepts in the training data and in the test data and Figure 2 gives an example image for each of the categories.
For each run, results for each concept were evaluated by plotting ROC curves. The results for each concept were summarised by two values: the area under the ROC curve (AUC) and the Equal Error Rate (EER). The latter is the error rate at which the false positive rate is equal to the false negative rate. Furthermore, for each run, the average AUC and average EER over all concepts were calculated.
indoor night tree sky
Results from the Evaluation In total 11 groups participated and submitted 53 runs. Below, we brie y describe the methods employed by each group: CEA-LIST. The Lab of applied research on software-intensive technologies of the CEA, France submitted 3 runs using di erent image features accounting for color and spatial layout with nearest neighbour and support vector machine classi ers.
HJ FA. The Microsoft Key Laboratory of Multimedia Computing and Communication of the University of Science and Technology, China submitted one run using color and SIFT descriptors which are combined and classi ed using a nearest neighbour classi er. IPAL I2R. The IPAL French-Singaporean Joint Lab of the Institute for Infocomm Research in
Singapore submitted 8 runs using a variety of di erent image descriptors.
LSIS. The Laboratory of Information Science and Systems, France submitted 7 runs using a structural feature combined with several other features using multi-layer perceptrons. MMIS. The Multimedia and Information Systems Group of the Open University, UK submitted 4 runs using CIELAB and Tamura features and combinations of these.
Makerere. The Faculty of Computing and Information Technology, Makerere University, Uganda submitted one run using luminance, dominant colors, and di erent texture and shape features which are classi ed using a nearest neighbour classi er.
RWTH. The Human Language Technology and Pattern Recognition Group from RWTH Aachen University, Germany submitted one run using a patch-based bag-of-visual words approach using a log-linear classi er.
TIA. The Group for Machine Learning for Image Processing and Information Retrieval from the National Institute of Astrophysics, Optics and Electronics, Mexico submitted 7 runs using global and local features with support vector machines and random forest classi ers. UPMC. The University Pierre et Marie Curie in Paris, France submitted 5 runs using fuzzy decision forests.
XRCE. The Textual and Visual Pattern Analysis group from the Xerox Research Center Europe in France submitted two runs using multi-scale, regular grid, patch-based image features and a Fisher-Kernel Vector classi er. budapest. The Datamining and Websearch Research Group, Hungarian Academy of Sciences, Hungary submitted 13 runs using a wide variety of di erent features, classi ers, and combinations.
The average EER and average AUC for each submitted run are given in Table 2. From this table, it can be seen that the best overall runs were submitted by XRCE.
Table 3 shows a breakdown of the results per concept. For each concept, the best and worst EER and AUC are shown, along with the average EER and AUC over all runs submitted. The best results were obtained for all concepts by XRCE, with budapest doing equally well on the night concept. The AUC per concept for all the best runs is 80.0% or above. Among the best results, the concepts having the highest scores are indoor and night . The concept with the worst score among the best results is road or pathway, most likely due to the high variability in the appearance of this concept. The concept with the highest average score, in other words, the concept that was detected best in most runs is sky. Again, the concept with the worst average score is road or pathway.
Only one group participating in the photo retrieval task of ImageCLEF made use of the concept annotation created by participants of the VCDT. The group from Universit Pierre et Marie Curie in Paris, France used the annotation in 12 of their 19 runs. However, the runs of that group were not ranked very highly and thus it is hard to judge how big the impact was. Their best two runs used the annotation and have a P (20) value of slightly over 0.26 while their third best run, which did not use these data has a P (20) value of 0.25. 4
This paper summarises the ImageCLEF 2008 Visual Concept Detection Task. The aim was to automatically annotate images with concepts, with a list of 17 hierarchically organised concepts provided. The results demonstrate that this task can be solved reasonably well, with the best run having an average AUC over all concepts of 90.66%. Six further runs obtained AUCs between 80% and 90%. When evaluating the runs on a per concept basis, the best run also obtained an AUC of 80% or above for every concept. Concepts for which automatic detection was particularly successful are: indoor/outdoor , night , and sky. The worst results were obtained for the concept road or pathway.
group CEA_LIST CEA_LIST CEA_LIST HJ_FA IPAL_I2R IPAL_I2R IPAL_I2R IPAL_I2R IPAL_I2R IPAL_I2R IPAL_I2R IPAL_I2R LSIS LSIS LSIS LSIS LSIS LSIS LSIS MMIS MMIS MMIS MMIS Makerere RWTH TIA TIA TIA TIA TIA TIA TIA UPMC UPMC UPMC UPMC UPMC UPMC XRCE XRCE budapest budapest budapest budapest budapest budapest budapest budapest budapest budapest budapest budapest budapest run # concept EER group average
worst 97.4 96.6 89.7 85.7 97.4 84.6 80.0 89.9 88.3 93.8 86.8 89.7 95.7 96.4 92.1 93.7 85.7
XRCE XRCE XRCE XRCE XRCE/budapest XRCE XRCE XRCE XRCE XRCE XRCE XRCE XRCE XRCE XRCE XRCE XRCE