The ImageCLEF 2015 Scalable Image Annotation, Localization and Sentence Generation task was the fourth edition of a challenge aimed at developing more scalable image annotation systems. In particular this year the focus of the three subtasks available to participants had the goal to develop techniques to allow computers to reliably describe images, localize the di erent concepts depicted in the images and generate a description of the scene. All three tasks use a single mixed modality data source of 500,000 web page items which included raw images, textual features obtained from the web pages on which the images appeared, as well as various visual features extracted from the images themselves. Unlike previous years the test set was also the training set and in this edition of the task hand-labelled data has been allowed. The images were obtained from the Web by querying popular image search engines. The development and subtasks 1 and 2 test sets were both taken from the \training set" and had 1,979 and 3,070 samples, and the subtask 3 track had 500 and 450 samples. The 251 concepts this year were chosen to be visual objects that are localizable and that are useful for generating textual descriptions of visual content of images and were mined from the texts of our large database of image-webpage pairs. This year 14 groups participated in the task, submitting a total of 122 runs across the 3 subtasks and 11 of the participants also submitted working notes papers. This result is very positive, in fact if compared to the 11 participants and 58 submitted runs of the last year it is possible to see how the interest in this topic is still very high.
Every day, users struggle with the ever-increasing quantity of data available to
them. Trying to nd \that" photo they took on holiday last year, the image
on Google of their favourite actress or band, or the images of the news article
someone mentioned at work. There is a large number of images that can be
cheaply found and gathered from the Internet. However, more valuable is mixed
modality data, for example, web pages containing both images and text. A large
amount of information about the image is present on these web pages and
viceversa. However, the relationship between the surrounding text and images varies
greatly, with much of the text being redundant and/or unrelated. Moreover,
images and the webpages on which they appear can be easily obtained for
virtually any topic using a web crawler. In existing work such noisy data has indeed
proven useful, e.g. [
The Scalable Image Annotation, Localization and Sentence Generation task
is a continuation of the general image annotation and retrieval task that has been
part of ImageCLEF since its very rst edition in 2003. In the early years the
focus was on retrieving relevant images from a web collection given (multilingual)
queries, while from 2006 onwards annotation tasks were also held, initially aimed
at object detection, but more recently also covering semantic concepts. In its
current form, the 2015 Scalable Concept Image Annotation task is its fourth
edition, having been organized in 2012 [
This paper presents the overview of the fourth edition of the Scalable
Concept Image Annotation task [
Image concept annotation, localization and natural sentence generation generally has relied on training data that has been manually, and thus reliably annotated, an expensive and laborious endeavour that cannot easily scale, particularly as the number of concepts grow. However, images for any topic can be cheaply gathered from the web, along with associated text from the webpages that contain the images. The degree of relationship between these web images and the surrounding text varies greatly, i.e., the data are very noisy, but overall these data contain useful information that can be exploited to develop annotation systems. Motivated by this need for exploiting this useful data, the ImageCLEF 2015 Scalable Concept annotation, localization and sentence generation task aims to develop techniques to allow computers to reliably describe images, localize the di erent concepts depicted in the images and generate a description of the scene. Figure 1 shows examples of typical images found by querying search engines. As can 1 http://www.clef-initiative.eu (a) Images from a search query of \rainbow".
(b) Images from a search query of \sun". be seen, the data obtained are useful and furthermore a wider variety of images is expected, not only photographs but also drawings and computer generated graphics. This diversity has the advantage that this data can also handle the possible di erent senses that a word can have, or the di erent types of images that exist. Likewise, there are other resources available that can help to determine the relationships between text and semantic concepts, such as dictionaries or ontologies. There are also tools that can help to deal with noisy text commonly found on webpages, such as language models, stop word lists and spell checkers. The goal of this task was to evaluate di erent strategies to deal with the noisy data so that it can be reliably used for annotating, localizing, and generating natural sentences from practically any topic. 1. Subtask 1: The image annotation task continues in the same line of past years. The objective required the participants to develop a system that receives as input an image and produces as output a prediction of which concepts are present in that image, selected from a prede ned list of concepts and starting this year, where they are located within the image. 2. Subtask 2 (Noisy Track ): In light of recent interest in annotating images beyond just concept labels, this subtask required the participants to describe images with a textual description of the visual content depicted in the image.
It is thought of as an extension of subtask 1. This track was geared towards 2 Challenge website at http://imageclef.org/2015/annotation participants interested in developing systems that generated textual descriptions directly from images, e.g. by using visual detectors to identify concepts and generating textual descriptions from the detected concepts. This had a large overlap with subtask 1. 3. Subtask 3 (Clean track ): Aimed primarily at those interested only in the Natural Language Generation aspects of the subtask, therefore a gold standard input (bounding boxes labelled with concepts) was provided to develop systems that generate sentence, natural language based descriptions based on these gold standard annotations as input.
As common training set the participants were provided with 500,000 images crawled from the Internet, the corresponding webpages on which they appeared, as well as precomputed visual and textual features. Apart from the image and webpage data, the participants were also permitted and encouraged to use similar datasets and any other automatically obtainable resources to help in the processing and usage of the training data. In contrast to previous years, in this edition of the task hand labelled data has been allowed. Thus, the available trained ImageNet CNNs could be used, and the participants were encouraged to use also other resources such as ontologies, word disambiguators, language models, language detectors, spell checkers, and automatic translation systems. Unlike previous years, the test set was also the training set.
For the development of the annotation systems, the participants were provided with the following: { A training dataset of images and corresponding webpages compiled speci cally for the three subtasks, including precomputed visual and textual features (see Section 2.3). { A development set of images with ground truth labelled bounding box annotations and precomputed visual features for estimating the system performance. { A development set of images with at least ve textual descriptions per image for Subtask 2 and Subtask 3. { A subset of the development set for Subtask 3 with gold standard inputs (bounding boxes labelled with concepts) and correspondence annotation between bounding box inputs and terms in textual descriptions.
This year the training and the test images are all contained within the 500,000 images released at the beginning of the competition. At test time, it was expected that participants provided a classi cation for all images. After a period of two months, the development set, which included ground truth localized annotations, was released and about two months were given for participants to work on the development data. A maximum of 10 submissions per subtask (also referred to as runs) was allowed per participating group.
The 251 concepts this year were chosen to be visual objects that are
localizable and that are useful for generating textual descriptions of the visual content
of images. They include animate objects such as people, dogs and cats,
inanimate objects such as houses, cars and balls, and scenes such as city, sea and
mountains. The concepts were mined from the texts of our database of 31
million image-webpage pairs [
The dataset3 used was very similar to the one of the rst three editions of the
task [
The development and test sets were both taken from the \training set". A set of 5,520 images was selected for this purpose using a CNN trained to identify images suitable for sentence generation. The images were then annotated via crowd-sourcing in three stages: (i) image level annotation for the 251 concepts; (ii) bounding box annotation; (iii) textual description annotation. For the textual descriptions, basic spell correction was performed manually by the organisers using Aspell4. Both American and British English spelling variants (color vs. colour ) were retained to re ect the challenge of real-world English spelling variants. A subset of these samples was then selected for subtask 3 and further annotated by the organisers with correspondence annotations between bounding box instances and terms in textual descriptions.
The development set contained 2,000 samples, out of which 500 samples were further annotated and used as the development set for the subtask 3. Note that only 1,979 samples from the development set contain at least one bounding box annotation. The number of textual descriptions for the development set ranged from 5 to 51 per image (with a mean of 9.5 and a median of 8 descriptions). The test set for subtasks 1 and 2 contains 3,070 samples, while the test set for subtask 3 comprises 450 samples which are disjoint from the test set of subtasks 1 and 2. 3 Dataset available at http://risenet.prhlt.upv.es/webupv-datasets 4 http://aspell.net/ Textual Data: Four sets of data were made available to the participants. The rst one was the list of words used to nd the image when querying the search engines, along with the rank position of the image in the respective query and search engine it was found on. The second set of textual data contained the image URLs as referenced in the webpages they appeared in. In many cases, the image URLs tend to be formed with words that relate to the content of the image, which is why they can also be useful as textual features. The third set of data was the webpages in which the images appeared, for which the only preprocessing was a conversion to valid XML just to make any subsequent processing simpler. The nal set of data were features obtained from the text extracted near the position(s) of the image in each webpage it appeared in.
To extract the text near the image, after conversion to valid XML, the script and style elements were removed. The extracted texts were the webpage title, and all the terms closer than 600 in word distance to the image, not including the HTML tags and attributes. Then a weight s(tn) was assigned to each of the words near the image, de ned as s(tn) = P
1 8t2T s(t)
X 8tn;m2T
Fn;m sigm(dn;m) ;
(1)
where tn;m are each of the appearances of the term tn in the document T , Fn;m
is a factor depending on the DOM (e.g. title, alt, etc.) similar to what is done
in the work of La Cascia et al. [
Subtask 1 Ultimately the goal of an image annotation system is to make decisions about which concepts to assign and localize to a given image from a prede ned list of concepts. Thus to measure annotation performance, what should be considered is how good and accurate are those decisions the precision of a system. Ideally a recall measure would also be used to penalize a system that has additional false positive output. However given di culties and unreliability of with the hand labeling of the concepts for the test images it wasn't possible to guarantee all concepts were labeled, however, it was assumed that the labels present were accurate and of a high quality.
The annotation and localization of Subtask 1 were evaluated using the
PASCAL VOC [
BBfg [ BBgt (2) where BB is a rectangle bounding box, f g is a foreground proposed annotation label, gt is the ground truth label of the concept. It calculates the area of intersection between the foreground in the proposed output localization and the ground-truth bounding box localization, divided by the area of their union. IoU is superior to a more naive measure of the percentage of correctly labelled pixels as IoU is automatically normalized by the size of the object and penalizes segmentation's that include the background. This means that small changes in the percentage of correctly labelled pixels can correspond to large di erences in IoU, and as the data-set has a wide variation in object size, the performance increases from our approach are more reliably measured. The evaluation of the ground truth and proposed output overlap was recorded from 0% to 90%. At 0%, this is equivalent to an image level annotation output, and 50% is the standard PASCAL VOC style metric used. The localized IoU is then used to compute the mean average precision (MAP) of each concept independently. This is then reported both per concept and averaged over all concepts.
Subtask 2 Subtask 2 was evaluated using the METEOR evaluation metric [
Subtask 3 Subtask 3 was also evaluated using the METEOR evaluation metric (see above). In addition, we have pioneered a ne-grained metric to evaluate the content selection capabilities of the sentence generation system. The content selection metric is the F1 score averaged across all 450 test images, where each F1 score is computed from the precision and recall averaged over all gold standard descriptions for the image. Intuitively, this measure evaluates how well the sentence generation system selects the correct concepts to be described against gold standard image descriptions. Formally, let I = fI1; I2; :::IN g be the set of test images. Let GIi = fGI1i ; GI2i ; :::; GIMi g be the set of gold standard descriptions for image Ii, where each GImi represents the set of unique bounding box instances referenced in gold standard description m of image Ii. Let SIi be the set of unique bounding box instances referenced by the participant's generated sentence for image Ii. The precision P Ii for test image Ii is computed as: where jGImi \ SIi j is the number of unique bounding box instances referenced in both the gold standard description and the generated sentence, and M is the number of gold standard descriptions for image Ii.
Similarly, the recall RIi for test image Ii is computed as:
P Ii =
M m jSIi j RIi =
M m jGImi j F Ii = 2
P Ii P Ii + RIi
RIi (3) (4) (5) The content selection score for image Ii, F Ii , is computed as the harmonic mean of P Ii and RIi : The nal P , R and F scores are computed as the mean P , R and F scores across all test images.
The advantage of the macro-averaging process in equations (3) and (4) is that it implicitly captures the relative importance of the bounding box instances based on how frequently to which they are referred across the gold standard descriptions. 3 3.1
The participation was excellent, with a greater number of teams including a number of new groups. In total 14 groups took part in the task and submitted overall 122 system runs. The number of runs is nearly double the previous year. Among the 14 participating groups, 11 of them submitted a corresponding paper describing their system, thus for these there were speci c details available. The following 14 teams submitted a working paper: Subtask 1 was well received despite the additional requirement of labelling and localizing all 500,000 images. All submissions were able to provide results on all 500,000 images, indicating that all groups have developed systems that are scalable enough to annotate large amounts of images. The nal results are presented in Table 1 in terms of mean average precision (MAP) over all images of all concepts, with both 0% overlap (i.e. no localization) and 50% overlap. It can Group
SMIVA IVANLPR
RUC CEA Kdevir
ISIA CNRS-TPT IRIP-iCC
UAIC MLVISP6 REGIM
Lip6 be seen that three groups have achieved over 0.50 MAP across the evaluation set with 50% overlap with the ground-truth. This seems an excellent result given the challenging nature of the images used and the wide range of concepts provided. The graph in Figure 2 shows the performance of each submission for an increasing amount of overlap of the ground truth labels.
The results from the groups seem encouraging and it would seem that the use of CNNs has allowed for large improvements in performance. Of the top 4 groups all use CNNs in their pipeline for the feature description.
SMIVA used a deep learning framework with additional annotated data, while IVANLPR implemented a two-stage process, initially classifying at an image level with an SVM classi er, and then applying deep learning feature classi cation to provide localization. While RUC trained a per concept, an ensemble of linear SVMs trained by Negative Bootstrap using CNN features as image representation. Concept localization was achieved by classifying object proposals generated by Selective Search. The approach by CEA LIST could be thought of as the baseline, they just use the CNN learnt features in a small grid based approach for localization.
Examples of the most and least successful localized concepts are shown in tables 2 and 3 respectively, together with the number of labelled occurrences of these concepts in the test data. Discussion for subtask 1 As can be observed in Table 1, the performance of many submissions was high this year, even given the additional constraint of localization. In fact, the 4 teams managed to achieve over 0.5 MAP, with 50% overlap with the ground truth. This perhaps indicates that in conjunction with the improvements from the CNN's real progress is starting to be made in the image annotation.
Figure 2 shows the change in performance as the requirements for intersection with the ground truth labels increases. All the approaches show a steady drop o in performance which is encouraging, illustrating that the approaches don't fail to detect a number of concepts correctly even with a high degree of accuracy. Even 90% overlap with the groundtruth the MAP for SMIVA was 0.35, which is impressive. Table 2 shows the most correctly localized concepts and also the number of occurrences of the concept. As it is important to remember that due to imperfect annotation no recall level is calculated. This is likely to be why the concept bee is so high. However there is encouraging performance for mountain, statue, bench and suit. These are all quite varied concepts, in term of scale and percentage of the image the concept will cover. However examining Table 3 shows a number of concepts that should be detected and are not such as leaf and wheel. However, many in that table are quite small concepts and, therefore, harder to localize and intersect with the labelled ground truth. This could be an area to direct the challenge objectives in future years.
From a computer vision perspective, we would argue that the ImageCLEF
challenge has two key di erences in its dataset construction to that of the other
popular data sets ImageNet [
For subtask 2, participants were asked to generate sentence-level textual descriptions for all 500,000 training images. The systems were evaluated on a subset of 3,070 instances. Four teams participated in this pilot subtask. Table 4 shows the METEOR scores for subtask 3, for all submitted runs by all four participants. Three teams achieved METEOR scores of over 0.10. RUC achieved the highest METEOR score, followed by ISIA, MindLab, and UAIC. We observed a large variety of approaches used by participants to tackle this subtask. RUC used the state of the art deep learning based CNN-LSTM caption generation system, MindLab employed a joint image-text retrieval approach, and UAIC a template-based approach.
As a comparison, we estimated a human upper-bound for this subtask by evaluating one description against the other descriptions for the same image and repeating the process for all descriptions. The METEOR score for the human upper-bound is estimated to be 0.3385 (Table 4). Therefore, there is clear scope for future improvement and work to improve image description generation systems. 3.4
For subtask 3, participants were provided with gold standard labelled bounding box inputs for 450 test images (released one week before the submission deadline), and were asked to generate textual descriptions for each image based on the gold standard input bounding boxes. To enable evaluation using the content selection metric (Section 2.4), participants were also asked to indicate within the textual descriptions the bounding box(es) to which the relevant term(s) correspond.
Two teams participated in this subtask (both of whom also participated in subtask 2). Table 5 shows the content section and METEOR scores for the subtask, again for all submitted runs by the two participants. RUC performed marginally better than UAIC in terms of the F and METEOR scores. Interestingly, it can be observed that RUC's sentence generation system has higher precision P , while UAIC achieved higher recall R in general than RUC. This is possibly due to RUC's use of a deep learning based sentence generator coupled with re-ranking based on the gold standard input which yielded higher precision, while UAIC's template-based generator selected more bounding boxes to be described resulting in a higher recall. Note that the METEOR scores are generally higher in subtask 3 compared to subtask 2 as participants are provided with gold standard input concepts, as well as the subtask having a smaller test set of 450 samples.
As a baseline, we generated textual descriptions per image by selecting at most three bounding boxes from the gold standard at random (the average number of unique instance mentions per description in the development set is 2.89). These concepts terms were then connected with random words or phrases selected randomly from a prede ned list of prepositions and conjunctions followed by an optional article the. Like subtask 2, we also computed a human upperbound. The results for these are shown in Table 5. As observed, all participants performed signi cantly better than the random baseline. Compared to the human upper-bound, again much work can still be done. An interesting note is that RUC achieved a high precision P almost on par with the human upper-bound, at the expense of a lower R. RUC ISIA MindLab UAIC Human 1 2 3 4 5 6 Median Mean F
Mean P
Mean R RUC UAIC Baseline Human There are two major limitations that we have identi ed with the challenge this year. Very few of the groups used the provided data set and features, we found this surprising, considering the state of the art CNN features and many others were included. However, this is likely to be due to the complexity and challenge of the 500,000 web page based images. Given they were collected from the Internet with little, a large number of the images are poor representations of the concept. In fact a number of participants annotated a large amount of their own more perfect training data, as their learning process assumes perfect or near perfect training examples, it will fail. As the number of classes increases and become more varied annotating all perfect data will become more di cult.
Another shortcoming of the overall challenge is the di culty of ensuring the ground truth has 100% of concepts labelled, thus allowing a recall measure to be used. This is especially problematic as the concepts selected include ne-grained categories such as eyes and hands that are generally small but occur frequently in the dataset. In addition, it was di cult for annotators to reach a consensus in annotating bounding boxes for less well-de ned categories such as trees and eld. Given the current crowd-source based hand-labelling of the ground truth, the concepts have missed annotations. Thus, in this edition a recall measure is not evaluated for subtask 1. 4
This paper presented an overview of the ImageCLEF 2015 Scalable Concept Image Annotation task, the fourth edition of a challenge aimed at developing more scalable image annotation systems. The focus of the three subtasks available to participants had the goal to develop techniques to allow computers to reliably annotate images, localize the di erent concepts depicted in the images and generate a description of the scene.
The participation increased this year compared to last year with 14 teams
submitting in total 122 system runs. The performance of the submitted systems
was somewhat superior to last year's results for sub task 1. Especially
considering the requirement to label all 500,000 images in the training/test set. This
was in part probably due to the increased CNN usage as the feature
representation. The clear winner of this year's subtask 1 evaluation was the SMIVA [
The results of the task have been very interesting and show that useful annotation systems can be built using noisy web crawled data. Since the problem requires to cover many fronts, there is still a lot of work that can be done, so it would be interesting to continue this line of research. Papers on this topic should be published, demonstration systems based on these ideas be built and more evaluation of this sort be organized. Also, it remains to see how this can be used to complement systems that are based on clean hand labeled data and nd ways to take advantage of both the supervised and unsupervised data.
The Scalable Image Annotation, Localization and Sentence Generation task was coorganized by the ViSen consortium under the EU CHIST-ERA D2K Programme, supported by EPSRC Grants EP/K01904X/1 and EP/K019082/1, and by French ANR Grant ANR-12-CHRI-0002-04. This work was also supported by the European Science Foundation (ESF) through the research networking programme Evaluating Information Access Systems (ELIAS).
hog hole hook horse hospital house jacket jean key keyboard kitchen knife ladder lake leaf leg letter library lighter lion lotion magazine male child man mask mat mattress microphone milk mirror monkey motorcycle mountain mouse mouth mushroom neck necklace necktie nest newspaper nose nut office onion orange oven painting pan park pen pencil piano picture pillow planet pool pot potato prison pumpkin rabbit rack radio ramp ribbon