ImageCLEF's Wikipedia Retrieval task provides a testbed for the system-oriented evaluation of multimedia information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. This paper presents an overview of the resources, topics, and assessments of the Wikipedia Retrieval task at ImageCLEF 2010, summarizes the retrieval approaches employed by the participating groups, and provides an analysis of the main evaluation results.
The Wikipedia Retrieval task is an ad-hoc image retrieval task. The evaluation scenario is thereby similar to the classic TREC ad-hoc retrieval task: simulation of the situation in which a system knows the set of documents to be searched, but cannot anticipate the particular topic that will be investigated (i.e. topics are not known to the system in advance). Given a multimedia query that consists of a title and one or more example images describing a user’s multimedia information need, the aim is to find as many relevant images as possible from a Wikipedia image collection.
The Wikipedia Retrieval task differs from other benchmarks in multimedia information retrieval, like TRECVID, in the sense that the textual modality in the Wikipedia image collection contains less noise than the speech transcripts in TRECVID. Similarly to past years, participants are encouraged to develop approaches that combine the relevance of different media types into a single ranked list of results. A number of resources that support participants towards this research direction were provided this year.
The paper is organized as follows. First, we introduce the task’s resources: the Wikipedia image collection and additional resources, the topics, and the assessments (Sections 2–4). Section 5 presents the approaches employed by the participating groups and Section 6 summarizes their main results. Section 7 concludes the paper.
The ImageCLEF 2010 Wikipedia collection consists of 237,434 Wikipedia
images, their user-provided annotations, the Wikipedia articles that contain these
images, and low-level visual features of these images. The collection was built
to cover similar topics in English, German and French and it is based on the
September 2009 Wikipedia dumps. Images are annotated in none, one or
several languages and, wherever possible, the annotation language is given in the
metadata file. The articles in which these images appear were extracted from
the Wikipedia dumps and are provided as such. Image features were extracted
using MM, CEA LISTs image indexing tool [
The main difference between the ImageCLEF 2010 Wikipedia collection and
the INEX MM Wikipedia collection [
The collection contains 237,434 images and the associated annotations are distributed as follows: – English only: 70,127 – German only: 50,291 – French only: 28,461 – English and German: 26,880 – English and French: 20,747 – German and French: 9,646 – English, German and French: 22,899 – Language undetermined: 8,144 – No textual annotation: 239 This distribution shows that the annotations in the ImageCLEF 2010 Wikipedia collection are heterogeneous, with nearly 10% of the collection with annotations in all three languages, 24% of the images with annotations in two languages out of three, 62% of the images with annotations in only one language, and the rest of the images with annotations for which language was not identified or no annotation exists. This distribution of annotations aims to encourage the investigation of multilingual approaches since they are likely to work better than monolingual approaches. 2.1
Metadata are provided as a single metadata.zip archive which is split into 26 directories (from 1 to 26): “metadata/1” contains XML files from 0.xml to 9999.xml, “metadata/2” contains images from 10000.xml to 19999.xml, etc. Note that each directory may contain the metadata for less than 10,000 images, since some of the initial images were removed during the collection construction so as to eliminate duplicates and to ensure, to the extent possible, that the provided images are copyright free and valid. Textual annotations were extracted from the Wikimedia Commons files that describe the images and from the article(s) that contain the images. Annotations are grouped by language when it was possible to identify their language.
The main components of the .xml files (for an example see Figure 1) are: – <image> - contains the unique ID of the image and a link to the image file. – <name> - the name of the image as found in the Wikimedia Commons repository. No processing has been applied to this text. – <text xml:lang=”LANGUAGE”> - where LANGUAGE if one of {en, de, fr}. These are textual annotations for which the language was identified. • <description> was extracted from the Wikimedia Commons page of the image whenever the language of this text was explicitly marked in a normalized manner. • <comment> was extracted from the Wikimedia Commons page of the image whenever the language of this text was marked in a normalized manner. <comment> is a substring of the raw comment described below. • <caption> is the text that accompanies the image in Wikipedia articles which are provided in the TEXT part of the collection (see Section 2.3) and are linked in the <caption> element. Sometimes Wikipedia articles contain images without captions. In such cases the <caption> element links to the article but is empty. Note that one image can appear in more than one article; in that case, all captions or links to the articles are provided. – <comment> - raw annotation as found on the Wikimedia Commons page of the image. No processing was applied to this text and the comment can be in one or more languages (not necessarily one of {en, de, fr}). – <license> - licensing information extracted from the Wikimedia Commons page of the image.
Any or all of the textual annotations of the images can be missing and some of them can be redundant.
The images are provided in the “images” directory and, due to their total size, are provided in 26 separate archives. They are organized in the same way as the associated metadata, namely in 26 directories and have the same IDs as the XML files (images/1/7924.png corresponds to metadata/1/7924.xml). Images are linked in the <image> element of the XML files using “file”. 2.3 Text Wikipedia article texts are provided in the “text” directory and are separated in language subdirectories (“text/en”, “text/de”, “text/fr”). Articles were renamed in order to have unique IDs in the collection and their IDs start at 300000 for English, 400000 for German and 500000 for French. They are grouped in 5 subdirectories for each language, with each such subdirectory containing 10,000 elements or less (for instance, “text/en/1/” includes articles from 300000 to 309999). Articles in which images appear are referenced in the <caption> element of the XML files contained in the “metadata” directory.
Image features were also provided to support the participants in their investigation of multimodal approaches.
Image features are provided in the “features” directory in binary files (one
file per feature). Features were computed using the MM, CEA LIST’s image
indexing tool [
Along with the feature files a Perl script was provided to show how to exploit features in order to compute image similarities.
The additional resources are beneficial to researchers who wish to exploit visual evidence without performing image analysis. Of course, participants could also extract their own image features. 3
3.1
The topics are descriptions of multimedia information needs that contain textual and visual hints.
These multimedia queries consist of a textual part, the query title, and a visual part, one or several example images. <title> query by keywords <image> query by image content (one or several) <narrative> description of query in which the definitive definition of relevance and irrelevance are given <title> The topic <title> simulates a user who does not have (or want to use) example images or other visual constraints. The query expressed in the topic <title> is therefore a text-only query. This profile is likely to fit most users searching digital libraries or the Internet.
Upon discovering that a text-only query does not produce many relevant hits, a user might decide to add visual hints and formulate a multimedia query. <image> The visual hints are example images, which express the narrative of the topic. <narrative> A clear and precise description of the information need is required in order to unambiguously determine whether or not a given document fulfils the given information need. In a test collection this description is known as the narrative. It is the only true and accurate interpretation of a user’s needs. Precise recording of the narrative is important for scientific repeatability - there must exist, somewhere, a definitive description of what is and is not relevant to the user.
Textual terms and visual examples can be used in any combination in order to produce results. It is up to the systems how to use, combine or ignore this information; the relevance of a result does not directly depend on these constraints, but it is decided by manual assessments based on the <narrative>.
The topics in the ImageCLEF 2010 Wikipedia Retrieval task (see Table 1), created by the organizers of the task, aim to cover diverse information needs and to have a variable degree of difficulty. They were chosen from an initial pool of 137 candidate topics that were derived from a search log file and from the topics of the 2008 and 2009 WikipediaMM tasks. Candidate topics were run through the Cross Modal Search Engine 4 (CMSE - developed by the University of Geneva) in order to get an indication of the number of relevant images in top results for baseline image only, text only and multimodal approaches. The final pool contains around 1/3 of topics that return good results for each type of retrieval.
The topics range from simple, and thus relatively easy (e.g., “postage stamp”), to semantic, and hence highly difficult (e.g., “white house with garden”), with the latter forming the bulk of the topics. Semantic topics typically have a complex set of constraints, need world knowledge, and/or contain ambiguous terms, so they are expected to be challenging for current state-of-the-art retrieval algorithms. We encouraged the participants to use multimodal approaches since they are more appropriate for dealing with semantic information needs.
Image examples were selected from Flickr, after checking that they are uploaded under the Creative Commons license. Each topic has one or several image examples, chosen so as to illustrate the visual diversity of the topic. Query image examples and their low-level features are also associated to the collection in order to ensure repeatability of the experiments. On average, the 70 topics contain 1.68 images and 2.7 words. The Wikipedia Retrieval task is an image retrieval task, where an image with its metadata is either relevant or not (binary relevance). We adopted TREC-style pooling of the retrieved images with a pool depth of 100, resulting in pool sizes of between 1421 and 3850 images with a mean of 2659 and median of 2531. The evaluation was performed by three participant groups and by the organizers within a period of 4 weeks after the submission of runs. The assessors used a modified version of the web-based interface that was used last year and which has also been previously employed in the INEX Multimedia and TREC Enterprise tracks.
A total of 13 groups submitted 127 runs. The participation was significantly higher than last year both in terms of number of participants (13 vs. 8) and of submitted runs (127 vs. 57). Although the highest number of groups are located in European countries, the geographic spread of participants has increased this year, with North-American and Asian groups being better represented.
Table 2 gives an overview of the types of the submitted runs. This year more multimodal (text/visual) than text-only runs were submitted. Table 3 presents the combinations of annotation and query languages used by participants in their textual and multimodal runs. A majority of submitted runs are multilingual in at least one of the two aspects. Many teams used both multilingual queries and multilingual annotations in order to maximize retrieval performance and the best results presented in the next section (see Tables 4 and 5) validate this approach. Although runs that implicate English only queries are by far more frequent than runs implicating German and French only, some participants also submitted the latter type of runs. A short description of the participants’ approaches follows.
CHESHIRE (8 runs) [
DAEDALUS (6 runs) [
I2RCVIU (6 runs) [
NUS (14 runs) They submitted both text and multimodal runs. For textual runs, they mapped image metadata to Wikipedia concepts, which were subsequently used during retrieval. For mixed runs, they also identify concepts from images.
RGU (8 runs) [
SZTAKI (5 runs) [
TELECOM (16 runs) [
UAIC (2 runs) From the image’s textual metadata, they generated image keywords which were filtered using a comparison to visually similar images. During retrieval, the topics were matched against the previously generated keywords. Visual similarity was equally used in order to rank retrieved images.
UNED (20 runs) [
UNT (3 runs) [
XRCE (16 runs) [
and the best performing run for each team, respectively, ranked by Mean Average Precision (MAP). Compared to 2009, when the best submitted runs were textual, this year multimodal runs submitted by XRCE were ranked best with a MAP of 0.2765. The best textual run, also submitted by XRCE was ranked 12th and had a MAP of 0.2361. The results in Table 5 show that results for individual teams are more nuanced, with five teams having multimodal runs as their best submission. The complete list of results can be found at the ImageCLEF website 5. 5 http://www.imageclef.org/2010/wikiMM-results Here, we analyze the evaluation results using only the top 90% of the runs to exclude noisy and buggy results. Because there was only one visual only run among the top 90%, it was also discarded. Table 6 shows the average performance and standard deviation with respect to each modality. On average, the textual runs have a slightly better performance than multimodal ones with respect to all examined evaluation metrics (MAP, Precision at 20, and precision after R (= number of relevant) documents retrieved). To analyze the average difficulty of the topics, we classify the topics based on the AP values per topic averaged over all runs as follows: easy: M AP > 0.3 medium: 0.2 < M AP <= 0.3 hard: 0.1 < M AP <= 0.2 very hard: M AP < 0.1.
We also analyzed the performance of runs that use only text (TXT) versus runs that use both text and visual resources (TXTIMG). Figure 2 shows the average performance on each topic for all, text-only and text-visual runs. The text-based runs outperform the text-visual ones in 37 out of the 70, are outperformed by mixed runs in 31 cases and have the same performances in 2 cases. This indicates that less than half of the topics benefit from a multi modal approach.
The “visuality” of topics can be deduced from the performance of text-only and text-visual approaches that were presented in the last section. We consider that, if for a topic the text-visual approaches improve significantly the MAP over all runs (i.e., by diff(M AP ) >= 0.01), then we could consider that to be a visual topic. In the same way, we can define topics as textual, if the text-only approaches improve significantly the MAP over all runs of a topic. Based on this analysis, 26 of the topics can be characterized as textual and 24 as visual. The remaining 20 topics, where no clear improvements are observed, are considered to be neutral.
Table 8 presents the topics in each group, as well as some statistics on the topic, their relevant documents, and their distribution over the classes that indicate their difficulty. There are small differences between the average number of words and example images for textual, neutral and visual topics. An important difference is observed for the number of relevant documents/topic, with a significantly higher number of such documents for visual topics compared to textual topics. This distribution of the number of relevant images indicates that a larger number of positive examples per query are needed for the visual features to be effective. Interestingly, the average mean average precision is distributed inversely, with a significantly higher average score for textual queries compared to visual ones (0.219 vs. 0.131). The distribution of the textual, visual and neutral topics over the classes expressing their difficulty shows that the visual and neutral topics are more likely to fall into the hard/very hard class than the textual ones.
A closer look at the topics themselves indicates that textual ones often include a named entity (“ISS”, “Ferrari”, “Christmas”, “Jintao Hu” etc.). Visual cues, which could be useful for topics that have a well defined semantic interpretation or a coherent visual aspect, do not help in these cases.
Finally, we analyze the effect of the application of query expansion (QE) and relevance feedback (FB) techniques. Similarly to the analysis in the previous section, we consider the techniques to be useful for a topic, if they improved significantly the MAP over all runs. Table 9 presents the best performing topics for these techniques and some statistics. Query expansion is useful for 34 topics and relevance feedback for 15. As with visual topics, query expansion seems to be useful for queries which have a lot of associated relevant documents and for queries that are either hard or very hard. 7
For the first time this year, a multimodal approach performed best in the Wikipedia Retrieval task. It is encouraging to see more than half of the submitted runs were multimodal. This is possibly a consequence of the fact that visual descriptors were provided with the collection. A novelty this year was that participants were able to submit multilingual runs. Although a majority of runs focused either on a combination of topic languages or on English queries only, several groups submitted runs for German and French queries only. 8
Adrian Popescu was supported by the French ANR (Agence Nationale de la Recherche) vie the Georama project (ANR-08-CORD-009). Theodora Tsikrika was supported by the European Union via the European Commission project VITALAS (contract no. 045389). Jana Kludas was funded by the Swiss National Fund (SNF). The authors would also like to thank all the groups participating in the relevance assessment process.
The authors would like to thank Herve´ le Borgne and Pierre-Alain Moe¨llic (CEA LIST) for providing the visual features for the collection. dia Search with Noisy Modalities: Fusion and Multistage Retrieval In Working notes