The wikipediaMM task provides a testbed for the system-oriented evaluation of ad-hoc retrieval from a collection of Wikipedia images. It became a part of the ImageCLEF evaluation campaign in 2008 with the aim of investigating the use of visual and textual sources in combination to improve the retrieval performance. This paper presents an overview over the wikipediaMM 2008 task's resources, topics, assessments, participants' approaches, and main results.
The wikipediaMM task provides a testbed for the system-oriented evaluation of multimedia
information retrieval from a collection of Wikipedia (http://www.wikipedia.org/) images which has
been previously used in the INEX 2006-2007 Multimedia track [
It is an ad-hoc image retrieval task with an evaluation scenario similar to the classic TREC ad-hoc retrieval task and the ImageCLEFphoto task: simulation of the situation in which a system knows the set of documents to be searched, but cannot anticipate the particular topic that will be investigated (i.e. topics are not known to the system in advance). The goal of the simulation is: given a textual query (and/or sample images and/or concepts) describing a user’s multimedia information need to find as many relevant images as possible from the (INEX MM) wikipedia image collection. A multimedia retrieval approach in that case should be able to combine the relevance of different media types into a single ranking that is presented to the user. In this first year of the task, the focus in on monolingual retrieval.
The paper is organized as follows. First, we introduce the task resources (the image collection and other available resources), the topics, and assessments (Sections 2–4). Section 5 presents the approaches employed by the different participants, while Section 6 summarizes their main results. Section 7 concludes the paper and provides an outlook on next year’s task. 2
The resources used for the wikipediaMM task are based on Wikipedia data. The following resources
were available to the participants:
(INEX MM) wikipedia image collection: The collection consists of approximately 150,000
JPEG and PNG images provided by Wikipedia’s users. Each image is associated with
user generated alphanumeric, unstructured metadata in English. These metadata usually
contain a brief caption or description of the image, the Wikipedia user who uploaded the
image, and the copyright information. These descriptions are highly heterogeneous and of
varying length. Further information about the image collection can be found in [
These resources were also provided in the INEX 2006-2007 Multimedia track. The latter two resources are beneficial to researchers who wish to exploit visual evidence without performing image analysis. Of course, participants could also extract their own image features.
The topics for the ImageCLEF 2008 wikipediaMM task include (i) topics previously used in INEX 2006-2007 Multimedia track, and (ii) topics created by this year’s task participants. They are descriptions of multimedia information needs that may contain not only textual, but also visual and conceptual evidence. 3.1
Topic Format The wikipediaMM topics are multimedia queries that can consist of a textual, visual, and conceptual part, with the latter two parts being optional. <title> query by keywords <concept> query by concept (optional) <image> query by image content (optional) <narrative> description of query in which the definitive definition of relevance and irrelevance are given 3.1.1
<title> The topic <title> simulates a user who does not have (or does not want to use) example images or other visual information. The query expressed in the topic <title> is therefore a text-only query. This profile is likely to fit most users searching digital libraries.
Upon discovering that their <title> query returned many irrelevant hits, users might decide to add visual information and formulate a multimedia query. 3.1.2
<concept> 3.1.3
<image> 3.1.4
<narrative> This field is directly related to the concepts for which classification results are provided as an additional source of information (see 2), i.e., they are restricted to the 101 MediaMill concepts. The second type of visual evidence are example images, which can be taken from outside or inside Wikipedia and can be of any common format.
A clear and precise description of the information need is required in order to unambiguously determine whether or not a given image fulfils the given need. In a test collection this description is known as the narrative. It is the only true and accurate interpretation of a user’s needs. Precise recording of the narrative is important for scientific repeatability - there must exist a definitive description of what is and is not relevant to the user. To aid this, the <narrative> should explain not only what information is being sought, but also the context and motivation of the information need, i.e., why the information is being sought and what work-task it might help to solve.
The three different types of information sources (textual terms, visual examples and visual concepts) can be used in any combination. For each field more than one entry can be specified. It is up to the systems how to use, combine or ignore this information; the relevance of a result item does not directly depend on them, but it is decided by manual assessments based on the <narrative>. 53 military aircraft 55 animated cartoon 57 people riding bicycles 59 dancing couple 61 Singapore 63 star galaxy 65 famous buildings of
Paris 67 bees with flowers 69 mountains with snow
under sky 71 children riding bicycles 73 bridges at night 75 manga female character
Y
Y
Y
N
N
Y
N
N
N
N
Y
N
The topics in the wikipediaMM task have been mainly developed by the participants. Altogether,
12 of the participating groups submitted 70 candidate topics. The 35 topics used in INEX
20062007 Multimedia were also added to the candidate topic pool. The task organizers judged all
topics in the pool towards their “visuality” as proposed in [
The candidate topics were classified by the organisers; for the old INEX topics, the results of the INEX participants’ runs were also used to aid this classification. The final topic set is listed in Table 1 and consists of 75 topics: 5 visual (topic ids: 1-5), 35 textual (topic ids: 6-40), and 35 semantic (topic ids: 41-75). Table 2 shows some statistics on the topics. Not all topics contain visual/multimedia information (i.e., image examples or visual concepts); this corresponds well with realistic scenarios, since users who express multimedia information needs do not necessarily want to employ visual evidence. The wikipediaMM task is an image retrieval task, where an image with its metadata is either relevant or not (binary relevance). We adopted TREC-style pooling of the retrieved images with a pool depth of 100, resulting in pools of between 753 and 1850 images with a mean and median both around 1290. The evaluation was performed by the participants of the task within a period of 4 weeks after the submission of runs. The 13 groups that participated in the evaluation process used a web-based interface previously employed in the INEX Multimedia and TREC Enterprise tracks. Each participating group was assigned 4-5 topics and an effort was made to ensure that most of the topics were assigned to their creators. This was achieved in 76% of the assignments for the topics created by this year’s participants. 5
A total of 12 groups submitted 77 runs. Table 3 gives an overview of the resources used by the submitted runs. Most of the runs are textual only approaches, but compared to the INEX Multimedia track, there is a rise in fusion approaches that combine text and images, text and concepts, and all three modalities.
Below we briefly describe the appproaches investigated by the groups that participated in the ImageCLEF 2008 wikipediaMM task: Digital Media Institute, Peking University, China (7 runs). They investigated: (i) a text-based approach with query expansion, where the expansion terms are automatically selected from a knowledge base that is (semi-)automatically constructed from Wikipedia, (ii) a content-based visual approach, where they first train 1-vs-all classifiers for all queries by using the training images obtained by Yahoo! search, and then treat the retrieval task as a visual concept detection in the given Wikipedia image set, and (iii) a cross-media approach that combines and re-ranks the text- and content-based retrieval results.
CEA LIST, France (2 runs). Their approach is based on a query reformulation using concepts that are semantically related to those in the initial query. For each interesting entity in the query, they used Wikipedia and WordNet to extract related concepts, which were further ranked based on their perceived usefulness. A small list of visual concepts was used to rerank the answers to queries that included them, implicitly or explicitly. They submitted two automatic runs, one based on query reformulation only and one combining query reformulation and visual concepts.
NLP and Information Systems, University of Alicante, Spain (24 runs). They employed their textual passage-based retrieval system as their baseline approach, which was enhanced by a module that decomposes the (compound) file names in camel case notation into single terms, and by a module that performs geographical query expansion. They also investigated Probabilistic Relevance Feedback and Local Context Analysis techniques.
Data Mining and Web Search, Hungarian Academy of Sciences (8 runs). They used their own retrieval system to experiment with a text-based approach, that uses BM25 and query expansion based on blind relevance feedback, and its combination with a segmentbased visual approach.
Database Architectures and Information Access, CWI, Netherlands (2 runs). They used a language modelling approach based on purely textual evidence and also incorporated a length prior to bias retrieval towards images with longer descriptions than the ones retrieved by the language model.
Laboratoire Hubert Curien, Universit´e Jean Monnet, Saint-Etienne, France (6 runs).
They used a vector space model to compute similarities between vectors of both textual and visual terms; the textual part is a term vector of the terms’ BM25 weights and the visual part a 6-dimensional vector of clusters of colour features. The applied both manual and blind relevance feedback to a text-based run in order to expand the query with visual terms. Dept. of Computer Science and Media, Chemnitz University of Technology (4 runs).
They used their Xtrieval framework based on both textual and visual features, and also made use of the provided resources (concepts and features). They experimented with text-based retrieval, its combination with a visual approach, the combination of all three modalities, and thesaurus-based query expansion. They also investigated the efficiency of the employed approaches.
Multimedia and Information Systems, Imperial College, UK (6 runs). They examined textual features, visual features, and their combination. Their text-based approach was combined with evidence derived from a geographic co-occurrence model mined from Wikipedia, which aimed at disambiguating geographic references either in a context-dependent or a context-independent manner. Their visual-based approach, employing Gabor texture features and the Cityblock distance as a similarity measure, was combined with the text-based approach and with blind relevance feedback using a convex combination of ranks. SIG-IRIT, Toulouse, France (4 runs). They explored the use of images’ names as evidence in text-based image retrieval. They used them in isolation, by computing a similarity score between the query and the name of images using the vector space model, and in combination with textual evidence, either by fusing the ranking of a text-based approach (based on the XFIRM retrieval system) with the ranking produced by the name-based technique, or by using the text-based approach with an increase in the weight of terms in the image name. Computer Vision and Multimedia, Universit´e de Gen`eve, Switzerland (2 runs). Our approach is based the preference ranking option of the SVM light library developed by the Cornell University. The first run, which is our baseline, uses only text features that were extracted form the Wikipedia collection and the queries. The second run applies a feature selection to the high dimensional textual feature vector based on the features relevance to each query. The results were disappointing, which is due to that our system is still under development.
UPMC/LIP6 - Computer Science Lab, Paris, France (7 runs). They investigated (i) text-based retrieval, using a tf.idf approach, a language modelling framework and their combination based on the ranks of retrieved images, and (ii) the combination of textual and visual evidence, by reranking the text-based results using visual similarity scores (Euclidean distance and a manifold-based technique, both based on HSV features).
LSIS, UMR CNRS & Universit´e Sud Toulon-Var, France (5 runs). They applied the same techniques they used for the Visual Concept Detection task at ImageCLEF 2008, by relating each of the wikipediaMM topics to one or more visual concepts from that task. They also fused these visual-based rankings with the results of a text-based approach. 6
Table 4 shows the top 30 runs ranked by their Mean Average Precision (MAP); the complete result list of results can be found at: http://www.imageclef.org/2008/wikimm-results.
We first analyse the performance per modality by comparing the average performance over runs that employ the same type of resources. For every analysis, we take into account only the top 75% of all runs so as to exclude noise by removing erroneous and buggy runs. Table 5 shows the average performance and standard deviation with respect to modality.
According to the MAP measure, the best performing runs fuse text and concepts, followed by the runs that fuse text and images, and the text-only baseline. The latter two perform almost identically. A similar result is obtained with the precision after R (= number of relevant) documents are retrieved (R-prec.). Again, the TxtCon runs outperform all others. But contrary to the MAP, here all runs that use multi-modal fusion achieve better results than the text-only baseline, even though TxtImgCon does this only by a small margin.
An actual surprise comes for the precision at 20 documents, where the visual-only based runs (only 2 in the top 75%) outperform by large the fusion approaches TxtCon and TxtImgCon, and the text-only baseline. So, good content-based retrieval of images can return highly relevant documents in the top ranks. The problem seems to be the fusion with the textual and conceptual evidence, since TxtImg and TxtImgCon fusion approaches cannot outperform the other approaches. But in general it can be said, that this year the fusion approaches perform surprisingly well, which is mostly due to the success of incorporating concepts.
Group 1 upeking 2 cea 3 ualicante 4 cea 5 ualicante 6 ualicante 7 ualicante 8 ualicante 9 ualicante 10 sztaki 11 sztaki 12 ualicante 13 cwi 14 sztaki 15 sztaki 16 ualicante 17 sztaki 18 sztaki 19 cwi 20 sztaki 21 sztaki 22 curien 23 ualicante 24 ualicante 25 ualicante 26 ualicante 27 chemnitz 28 ualicante 29 ualicante 30 chemnitz
We further analyse the performance per topic. Figure 2 shows for each topic the average MAP over all runs in the top 75%, over only the text-based ones among them, and over only the visualbased ones among them. The text-based runs outperform the visual-based ones in about half (54%) the topics. Table 7 presents the top 10 topics, where the visual runs outperform the textual, and vice versa. In the textual top 10, there are a lot of topics with proper nouns. Additionally, about half of them, do not have an example image in the topic (see Table 1), which explains the good performance of text-based approaches. The topics in the top 10 of the visual-based runs contain more general concepts. These could also be topics that are not well annotated in the collection. Oddly, the visual top 10 also includes topics without image examples, which suggests that some visual feedback or query expansion has been used by some of the participants.
Figure 3 compares the text baseline with the 3 fusion approaches: (1) text/concept, (2) text/visual and (3) text/concept/visual. The fusion of text and concepts outperforms the textonly baseline for 62% of the topics, the fusion of text and images for about half the topics (49%), and the fusion of all 3 modalities for 44% of the topics. We again create a list of the top 10 topics for which each fusion approach outperforms the text-only baseline. The lists for the fusion of text/visual and text/concept/visual have many entries in common. All 3 lists contain topics with proper nouns, as well as topics with general concepts. The clear distinction that was visible before between the visual and textual runs is now blurred.
So, in summary fusion approaches that use several modalities are catching up with the text-based approaches. Especially the performance of the text/concept fusion approaches is impressively good. The visual hints help mainly for topics that incorporate an image example, but can also improve the overall performance.
Our debut in ImageCLEF 2008 attracted much interest from groups researching multimedia retrieval and significantly more participants than the INEX 2006-2007 Multimedia track. With the help of our participants, we both developed and assessed a diverse set of 75 multimedia topics. The results show that the dominance of text-based image retrieval is coming to an end; multi-modal fusion approaches help to improve the retrieval performance in this domain.
Our main focus for next year remains the same: researching the combination of evidence from different modalities in a “standard” ad-hoc image retrieval task. Possible new directions for 2009 include the addition of multilinguality in form of multi-lingual topics (and if possible annotations), and access to the context of the images, i.e., the Wikipedia web pages that contain them. We also aim at providing new sets of classification scores and low-level features, so that participants can concentrate their research effort on information fusion. 8
Theodora Tsikrika was supported by the European Union via the European Commission project VITALAS (contract no. 045389). Jana Kludas was funded by the European project MultiMATCH (EU-IST-STREP#033104). The authors would also like to thank Thomas Deselaers for invaluable technical support and all the groups participating in the relevance assessment process.