The ImageCLEF 2013 Scalable Concept Image Annotation Subtask was the second edition of a challenge aimed at developing more scalable image annotation systems. Unlike traditional image annotation challenges, which rely on a set of manually annotated images as training data for each concept, the participants were only allowed to use automatically gathered web data instead. The main objective of the challenge was to focus not only on the image annotation algorithms developed by the participants, where given an input image and a set of concepts they were asked to decide which of them were present in the image and which ones were not, but also on the scalability of their systems, such that the concepts to detect were not exactly the same between the development and test sets. The participants were provided with web data consisting of 250,000 images, which included textual features obtained from the web pages on which the images appeared, as well as various visual features extracted from the images themselves. To evaluate the performance of the submitted systems a development set was provided containing 1,000 images that were manually annotated for 95 concepts and a test set containing 2,000 images that were annotated for 116 concepts. In total 13 teams participated, submitting a total of 58 runs, most of which significantly outperformed the baseline system for both the development and test sets, including for the test concepts not present in the development set and thus clearly demonstrating potential for scalability.
Automatic concept detection within images is a challenging and as of yet
unsolved research problem. Over the past decades impressive improvements have
been achieved, albeit admittedly not yet successfully solving the problem. Yet,
these improvements have been typically obtained on datasets for which all
images have been manually, and thus reliably, labeled. For instance, it has become
common in past image annotation benchmark campaigns [
(a) Images from a search query of \rainbow".
(b) Images from a search query of \sun".
tiple annotators label a large collection of images. Nonetheless, crowdsourcing is
expensive and di cult to scale to a very large amount of concepts. The image
annotation datasets furthermore usually include exactly the same concepts in
the training and test sets, which may mean that the evaluated visual concept
detection algorithms are not necessarily able to cope with detecting additional
concepts beyond what they were trained on. To address these shortcomings a
novel image annotation task [
The second edition of the scalable image annotation task is what is
presented in this overview paper, which is one of several ImageCLEF benchmark
campaigns [
Image concept detection generally has relied on training data that has been manually, and thus reliably, annotated, which is an expensive and laborious endeavor that cannot easily scale. To address this issue, the ImageCLEF 2013 scalable annotation subtask concentrated exclusively on developing annotation systems that rely only on automatically obtained data. A very large amount of images can be easily gathered from the web, and furthermore, from the webpages that contain the images, text associated with them can be obtained. However, the degree of relationship between the surrounding text and the image varies greatly. Moreover, the webpages can be of any language or even a mixture of languages, and they tend to have many writing mistakes. Overall the data can be considered to be very noisy.
To illustrate the objective of the evaluation, consider for example that someone searches for the word \rainbow" in a popular image search engine. It would be expected that many results be of landscapes in which in the sky a rainbow is visible. However, other types of images will also appear, see Figure 1a. The images will be related to the query in di erent senses, and there might even be images that do not have any apparent relationship. In the example of Figure 1a, one image is a text page of a poem about a rainbow, and another is a photograph of an old cave painting of a rainbow serpent. See Figure 1b for a similar example on the query \sun". As can be observed, the data is noisy, although it does have the advantage that this data can also handle the possible di erent senses that a word can have, or the di erent types of images that exist, such as natural photographs, paintings and computer-generated imagery.
In order to handle the web data, there are several resources that could be employed in the development of scalable annotation systems. Many resources can be used to help match general text to given concepts, amongst which some examples are stemmers, word disambiguators, de nition dictionaries, ontologies and encyclopedia articles. There are also tools that can help to deal with noisy text commonly found on webpages, such as language models, stop word lists and spell checkers. And last but not least, language detectors and statistical machine translation systems are able to process webpage data written in various languages.
In summary, the goal of the scalable image annotation subtask was to evaluate di erent strategies to deal with noisy data, so that the unsupervised web data can be reliably used for annotating images for practically any topic. 2.2
The subtask2 consisted of the development of an image annotation system given training data that only included images crawled from the Internet, the corresponding webpages on which they appeared, as well as precomputed visual and 2 Subtask website at http://imageclef.org/2013/photo/annotation textual features. As mentioned in the previous section, the aim of the subtask was for the annotation systems to be able to easily change or scale the list of concepts used for image annotation. Apart from the image and webpage data, the participants were also permitted and encouraged to use any other automatically obtainable resources to help in the processing and usage of the training data. However, the most important rule was that the systems were not permitted to use any kind of data that had been explicitly and manually labeled for the concepts to detect.
For the development of the annotation systems, the participants were provided with the following: { A training dataset of images and corresponding webpages compiled speci cally for the subtask, including precomputed visual and textual features (see Section 2.3). { Source code of a simple baseline annotation system (see Section 2.4). { Tools for computing the appropriate performance measures (see Section 2.5). { A development set of images with ground truth annotations (including precomputed visual features) for estimating the system performance.
After a period of two months, a test set of images was released that did not include any ground truth labels. The participants had to use their developed systems to predict the concepts for each of the input images and submit these results to the subtask organizers. A maximum of 6 submissions (also referred to as runs) were allowed per participating group. Since one of the objectives was that the annotation systems be able to scale or change the list of concepts for annotation, the list of concepts for the test set was not exactly the same as those for the development set. The development set consisted of 1,000 images labeled for 95 concepts, and the test set consisted of 2,000 images labeled for 116 concepts (the same 95 concepts for development and 21 more).
To observe the possible over tting of the development set and the di erence of performance with respect to the test set, the participants were also required to submit the concept predictions of the development set, using exactly the same system and parameters as for the test set.
The concepts to be used for annotation were de ned as one or more WordNet
synsets [
The dataset3 used was mostly the same as the one in ImageCLEF 2012 for
the rst edition of this task [
The 250,000 training set images were exactly the same as the ones for
ImageCLEF 2012. However, some images from the development and test sets had
been changed. To guaranty that the visual features were the same for the new
images, due to changes in software versions, the features were recalculated and
therefore are di erent from those supplied in the previous edition of this
subtask. Also this year the original images and webpages were provided. The most
signi cant change of the dataset with respect to 2012 was the labeling of the
development and test sets, where the images have now been labeled and linked
to concepts in WordNet [
Textual Data: Since the textual data was to be used only during training, it was only provided for the training set. Four sets of data were made available to the participants. The rst one 4 was the list of words used to nd the image when querying the search engines, along with the rank position of the image in the respective query and search engine it was found on. The second set of textual data4 contained the image URLs as referenced in the webpages they appeared in. In many cases the image URLs tend to be formed with words that relate to the content of the image, which is why they can also be useful as textual features. The third set of data were the webpages in which the images appeared, for which the only preprocessing was a conversion to valid XML just to make any subsequent processing simpler. The nal set of data4 were features obtained from the text extracted near the position(s) of the image in each webpage it appeared in.
To extract the text near the image, after conversion to valid XML, the script
and style elements were removed. The extracted text were the webpage title and
all the terms closer than 600 in word distance to the image, not including the
4 This textual data was identical to the 2012 edition [
HTML tags and attributes. Then a weight s(tn) was assigned to each of the words near the image, de ned as s(tn) = P
1 8t2T s(t)
X 8tn;m2T
Fn;m sigm(dn;m) ;
(1)
where tn;m are each of the appearances of the term tn in the document T , Fn;m
is a factor depending on the DOM (e.g. title, alt, etc.) similar to what is done
in the work of La Cascia et al. [
A toolkit was supplied to the participants as a performance reference for the evaluation, as well as to serve as a starting point. This toolkit included software that computed the evaluation measures (see Section 2.5) and the implementations of two baselines. The rst baseline was a simple random, which is important since any system that gets worse performance than random is useless. The other baseline, referred to as Co-occurrence Baseline, was a basic technique that gives better performance than random, although it was simple enough to give the participants a wide margin for improvement. In the latter technique, when given an input image, obtains its nearest K = 32 images from the training set using only the 1,000 bag-of-words C-SIFT visual features and the L1 norm. Then, the textual features corresponding to these K nearest images are used to derive a score for each of the concepts. This is done by using a concept-word co-occurrence matrix estimated from all of the training set textual features. In order to make the vocabulary size more manageable, the textual features are rst processed keeping only English words. Finally, the annotations assigned to the image are always the top 6 ranked concepts. 2.5
Ultimately the goal of an image annotation system is to make decisions about which concepts to assign to given image from a prede ned list of concepts. Thus to measure annotation performance what should be considered is how good are those decisions. On the other hand, in practice many annotations systems are based on estimating a score for each of the concepts and then a second technique uses these scores to nally decide which concepts are chosen. For systems of this type a measure of performance can be based only on the concept scores, which considers all aspects of the system except for the technique used for concept decisions, making it an interesting characteristic to measure.
For this task, two basic performance measures have been used for comparing the results of the di erent submissions. The rst one is the F-measure (F1), which takes into account the nal annotation decisions, and the other is the Average Precision (AP), which considers the concept scores.
The F-measure is de ned as
F1 = 2P R P + R ; where P is the precision and R is the recall. In the context of image annotation, the F1 can be estimated from two di erent perspectives, one being concept-based and the other sample-based. In the former, one F1 is computed for each concept, and in the latter one F1 is computed for each image to annotate. In both cases, the arithmetic mean is used as a global measure of performance, and will be referenced as MF1-concepts and MF1-samples, respectively.
The AP is algebraically de ned as
AP = 1 XjKj k jKj k=1 rank(k) ; where K is the ordered set of the ground truth annotations, being the order induced by the annotation scores, and rank(k) is the order position of the k-th ground truth annotation. The fraction k= rank(k) is actually the precision at the k-th ground truth annotation, and has been written like this to be explicit on (2) (3) the way it is computed. In the cases that there are ties in the scores, a random permutation is applied within the ties. The AP can also be estimated for both the concept-based and sample-based perspectives, however, the concept-based AP is not a suitable measure of annotation performance (it is more adequate for a retrieval scenario), so only the sample-based AP has been considered in this evaluation. As a global measure of performance, also the arithmetic mean is used, which will be referred to as MAP-samples.
A bit of care must be taken when comparing systems using the MAP-samples measure. What the MAP-samples turns out saying is that if for a given image the scores are used to sort the concepts, how good would it rank the true concepts for the image. Depending on the system, its scores could or could not be optimal for ranking the concepts. Thus a system with a relatively low MAP-samples, could still have a good annotation performance if the method used to select the concepts is adequate for its concept scores. Because of this, as well as the fact that there can be systems that do not rely on scores, it was optional for the participants of the task to provide scores. 3 3.1
The participation was excellent, especially considering that this was the second edition of the task and last year there was only one participant. In total 13 groups took part, submitting 58 runs overall. The following teams participated: { CEA LIST: The team from the Vision & Content Engineering group of CEA LIST (Gif-sur-Yvettes, France) was represented by Herve Le Borgne, Adrian Popescu and Amel Znaidia. { INAOE: The team from the Instituto Nacional de Astrof sica, Optica y
Electronica (Puebla, Mexico) was represented by Hugo Jair Escalante. { KDEVIR: The team from the Computer Science and Engineering department of the Toyohashi University of Technology (Aichi, Japan), was represented by Ismat Ara Reshma, Md Zia Ullah and Masaki Aono. { LMCHFUT: The team from Hefei University of Technology (Hefei, China) was represented by Yan Zigeng. { MICC: The team from the Media Integration and Communication Center of the Universita degli Studi di Firenze (Florence, Italy) was represented by Tiberio Uricchio, Marco Bertini, Lamberto Ballan and Alberto Del Bimbo. { MIL: The team from the Machine Intelligence Lab of the University of Tokyo (Tokyo, Japan) was represented by Masatoshi Hidaka, Naoyuki Gunji and Tatsuya Harada. { RUC: The team from the School of Information of the Renmin University of China (Beijing, China) was represented by Xirong Li, Shuai Liao, Binbin Liu, Gang Yang, Qin Jin, Jieping Xu and Xiaoyong Du. { SZTAKI: The team from the Datamining and Search Research Group of the Hungarian Academy of Sciences (Budapest, Hungary) was represented by Balint Daroczy. { THSSMPAM: The team from Beijing, China was represented by Jile Zhou. { TPT: The team of CNRS TELECOM ParisTech (Paris, France) was represented by Hichem Sahbi. { UNED&UV: The team from the Universidad Nacional de Educacion a Distancia (Madrid, Spain) and the Universitat de Valencia was represented by Xaro Benavent, Angel Castellanos Gonzales, Esther de Ves, D. HernandezAranda, Ruben Granados and Ana Garcia-Serrano. { UNIMORE: The team from the University of Modena and Reggio Emilia (Modena, Italy) was represented by Costantino Grana, Giuseppe Serra, Marco Manfredi, Rita Cucchiara, Riccardo Martoglia and Federica Mandreoli. { URJC&UNED: The team of the Universidad Rey Juan Carlos (Mostoles, Spain) and the Universidad Nacional de Educacion a Distancia (Madrid, Spain) was represented by Jesus Sanchez-Oro, Soto Montalvo, Antonio Montemayor, Juan Pantrigo, Abraham Duarte, V ctor Fresno and Raquel Mart nez. In Table 1 we provide a comparison of a the key details of the best submission of each group. For a more in depth look of the annotation systems of each team, please refer to their corresponding paper listed in the table. Note that there were four groups that did not submit a working notes paper describing their system, so for those submissions less information could be listed. 3.2
Table 2 presents the performance measures (mentioned in 2.5) for the baseline techniques and all of the submitted runs by the participants.The last column of the table corresponds to the MF1-concepts measure which was only computed for the 21 concepts that did not appear in the development set. The systems are ordered by performance, beginning at the top with the best performing one. This order of the systems has been derived by considering for the test set the average rank when comparing all of the systems, using the MF1-samples, the MF1-concepts and the MF1-concepts unseen in dev. measures, while breaking ties by the average of the same three performance measures.
For an easier comparison and a more intuitive visualization, the same results of Table 2 are presented as graphs in Figure 2 (only for the test set). These graphs include for each result the 95% con dence intervals. These intervals have been estimated by Wilson's method, employing the standard deviation for the individual measures (for the samples or concepts, and for the average precisions (AP) or F-measures (F1), depending on the case).
Finally, in Figure 3 there is for each of the 116 test set concepts, a boxplot (or also known as box-and-whisker plot) for the F1-measures when combining all runs. In order to t all of the concepts in the same graph, for multiple outliers with the same value, only one is shown. The concepts have been sorted by the median performance of all submissions, which in a way orders them by di culty. a Concept scores not provided, only annotation decisions. test 50 45 40 se 35 l Fig. 2: Graphs showing the test set performance measures (in %) for all the submissions. The error bars correspond to the 95% con dence intervals computed using Wilson's method.
Due to the considerable participation in this evaluation very interesting results have been obtained. As can be observed in Table 2 and Figure 2, most of the submitted runs signi cantly outperformed the baseline system for both the development and test sets. When analyzing the sample based performances, very large di erences can be observed amongst the systems. For both MAP-samples and MF1-samples the improvement has been from below 10% to over 40%. Moreover, the con dence intervals are relatively narrow, making the improvements quite signi cant. An interesting detail to note is that for MAP-samples there are several top performing systems, however, when comparing to the respective MF1-samples measures, three of the TPT submissions clearly outperform the rest. The key di erence between these is the method for deciding which concepts are selected for a given image. This leads to believe that that many of the systems could improve greatly by changing that last step of their systems. As a side note, many of the participants chose to use the same scheme as the baseline system for selecting the concepts, the top N and xed for all images. The number of concepts per image is expected to be variable, thus making this strategy less than optimal. Future work should be addressed in this direction.
The MF1-concepts results in Figure 2, in contrast to the sample based performances, present much wider con dence intervals. This is due to two reasons, there are fewer concepts than sample images and the performance for di erent concepts varies greatly (see Figure 3). This e ect is even greater for the MF1-concepts unseen, since these were only 21. Nevertheless, for MF1-concepts unseen, the top performing systems are statistically signi cantly better than the baselines and some of the lower performance systems. Moreover, in Figure 3 it can be observed that the unseen concepts do not tend to perform worse. The di culty of each particular concept a ects more the performance than the fact that these have not been seen during development, or from another perspective the systems have been able to generalize rather well to the new concepts. Thus, this demonstrates potential for scalability of the systems. It would be desired for future benchmarking campaigns of this type to have more labeled data available for the evaluation, or nd an alternative more automatic analysis, to be able to compare better the systems in this scalability performance aspect.
In contrast to usual image annotation evaluations with labeled training data, this challenge required work in more fronts, such as handling the noisy data, textual processing and multilabel annotations. This has given considerable freedom to the participants to concentrate their e orts in di erent aspects. Several teams extracted their own visual features, for which they did observe improvements with respect to the features provided by the organizers. On the other hand, for the textual processing, several di erent approaches were tried by the participants. Some of these teams (namely MIL, UNIMORE, CEA LIST, and URJC&UNED) reported in their working notes papers and/or as observed in the results in this paper that as more information and additional resources are used (e.g. synonyms, plus hyponyms, etc.) the performance of the systems improved. Curiously, the best performing system, TPT, only used the provided visual features and did a very simple expansion of the concepts. Overall it seems that several of the proposed ideas by the participants are complementary, and thus considerable improvements could be expected in future works. 4
This paper presented an overview of the ImageCLEF 2013 Scalable Concept Image Annotation Subtask, the second edition of a challenge aimed at developing more scalable image annotation systems. The goal was to develop annotation systems that for training only rely on unsupervised web data and other automatically obtainable resources, thus making it easy to add or change the concepts for annotation.
Considering that it is a relatively new challenge, the participation was excellent, 13 teams submitted in total 58 system runs. The performance of the submitted systems was considerably superior to the provided baselines, improving from below 10% to over 40% for both MAP-samples and MF1-samples measures. With respect to the performance of the systems when analyzed per concept, it was observed that the concepts vary greatly in di culty. An important result was that for the concepts that were not seen during the development, the improvement was also signi cant, thus showing that the systems are capable of successfully using the noisy web data and generalizing well to new concepts. This clearly demonstrates potential for scalability of the systems. Finally, the participating teams presented several interesting approaches to address the proposed challenge, concentrating their e orts in di erent aspects of the problem. Many of these approaches are complementary, thus considerable improvements could be expected in future works.
Due to the success of this year's campaign and the very interesting results obtained, it would be important to continue organizing future editions. To be able to derive better conclusions about the performance generalization to unseen concepts, it would be desirable to have more labeled data available and/or nd an alternative more automatic analysis which can help in giving more insight in this respect. Also, related challenges could be organized, for instance it could be assumed that for some concepts there is labeled data available, and nd out how to take advantage of both the supervised and unsupervised data.
The authors are very grateful of the support of the CLEF campaign for the ImageCLEF initiative. The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under the tranScriptorium project (#600707), the LiMoSINe project (#288024), and from the Spanish MEC under the STraDA project (TIN2012-37475-C02-01). unpaved rodent monument cloudless closeup male female spectacles indoor overcast smoke daytime shadow poster countryside embroidery elder teenager bird drum outdoor park cityscape hat airplane spider highway rain garden child sport dog table re ection phone cat sculpture arthropod coast soil book train guitar footwear diagram sign bridge baby sh rainbow reptile car harbor motorcycle truck river submarine nighttime painting toy helicopter portrait castle wagon beach food chair violin horse instrument drink bottle church butter y lake re silhouette tra c bicycle sun logo snow sand protest moon ower furniture grass forest sea vehicle newspaper aerial boat desert tricycle nebula cloud pool road building galaxy mountain bus water sky cartoon tree fog person plant underwater space lightning sunrise/sunset rework F in n io en t e a s t t o o n n n a s
e t n ep o c e n h o t c
e r r e a p
t e n h o t f f o ed t r e s n
i t s s e t t p e e h c t n r o o C f
. ) e rs c e n k a s i m h r
o w f - r d e n p a - n x a o i b ed s a m n e
h w t o n y k b o d ls re (a e
d s r t o o l
e p r x a o
s B t
o : l