As digital information is increasingly becoming multimodal, the days of single-language text-only retrieval are numbered. Take as an example Wikipedia where a single topic may be covered in several languages and include non-textual media such as image, audio, and video. Moreover, non-textual media may be annotated with text in several languages in a variety of metadata fields such as object caption, description, comment, and filename. Current search engines usually focus on limited numbers of modalities at a time, e.g. English text queries on English text or maybe on textual annotations of other media as well, not making use of all information available. Final rankings are usually results of fusion of individual modalities, a task which is tricky at best especially when noisy modalities are involved.
In this paper we present the experiments performed by Democritus University of Thrace (DUTH), Greece, in the context of our participation to the ImageCLEF 2011 Wikipedia Retrieval task.1 The ImageCLEF 2011 Wikipedia collection is the same as in 2010. It has image as its primary medium, consisting of 237; 434 items, associated with noisy and incomplete user-supplied textual annotations and the Wikipedia articles containing the images. Associated annotations are written in any combination of English, German, French, or any other unidentified language. This year there are 50 new test topics, each one consisting of a textual and a visual part: three title fields (one per language—English, German, French), and 4 or 5 example images. The exact details of the setting of the task, e.g., research objectives, collection etc., are provided at the task’s webpage.
We kept building upon and improving the experimental multimodal search engine
we introduced last year, www.mmretrieval.net (Fig.1). The engine allows
multiple image and multilingual queries in a single search and makes use of the total
available information in a multimodal collection. All modalities are indexed separately and
searched in parallel, and results can be fused with different methods. The engine
demonstrates the feasibility of the proposed architecture and methods, and furthermore enables
a visual inspection of the results beyond the standard TREC-style evaluation. Using the
engine, we experimented with different score normalization and combination methods
for fusing results. We eliminated the least effective methods based on our last year’s
participation to ImageCLEF [
The rest of the paper is organized as follows. In Section 2 we describe the MMretrieval engine, give the details on how the Wikipedia collection is indexed and a brief overview of the search methods that the engine provides. In Section 3 we describe in more detail the fusion methods we experimented with and justify their use. A comparative evaluation of the methods is provided in Section 4; we used the 2010 topics for tuning. Experiments with the 2011 topics are summarized in Section 5. Conclusions are drawn in Section 6.
www.MMRetrieval.net: A Multimodal Search Engine
During last year’s ImageCLEF Wikipedia Retrieval, we introduced an experimental
search engine for multilingual and multimedia information, employing a holistic web
interface and enabling the use of highly distributed indices [
To index images, we employ the family of descriptors known as Compact Composite
Descriptors (CCDs). CCDs consist of more than one visual features in a compact vector,
and each descriptor is intended for a specific type of image. We index with two
descriptors from the family, which we consider them as capturing orthogonal information
content, i.e., the Joint Composite Descriptor (JCD) [
The collection of images at hand, i.e. the ImageCLEF 2010/2011 Wikipedia collection, comes with XML metadata consisting of a description, a comment, and multiple captions, per language (English, German, and French). Each caption is linked to the wikipedia article where the image appears in. Additionally, a raw comment is supplied which may contain some of the per-language comments and any other comment in an unidentified language. Any of the above fields may be empty or noisy. Furthermore, a name field is supplied per image containing its filename. We do not use the supplied <license> field.
For text indexing and retrieval, we employ the Lemur Toolkit V4.11 and Indri V2.11 with the tf.idf retrieval model.2 In order to have clean global (DF) and local statistics (TF, document length), we split the metadata and articles per language and index them separately. Thus, we have 4 indices: one per language which includes metadata and articles together but allows limiting searches in either of them, plus one for the unidentified language metadata including the name field (which can be in any language). For English text, we enable Krovetz stemming; no stemming is done for other languages in the current version of the system. We also Krovetz-stem the unidentified language metadata, assuming that most of it is probably English. 2.2
The web application is developed in the C#/.NET Framework 4.0 and requires a fairly modern browser as the underlying technologies which are employed for the interface are HTML, CSS and JavaScript (AJAX). Fig.2 illustrates an overview of the architecture. The user provides image and text queries through the web interface which are dispatched in parallel to the associated databases. Retrieval results are obtained from each of the databases, fused into a single listing, and presented to the user.
Users can supply no, single, or multiple query images in a single search, resulting in 2 i active image modalities, where i is the number of query images. Similarly, users can supply no text query or queries in any combination of the 3 languages, resulting in 3 l active text modalities, where l is the number query languages. Each supplied query results to 3 modalities: it is run against the corresponding language metadata, articles, as well as, the unidentified language metadata. The current alpha version assumes that the user provides multilingual queries for a single search, while operationally query translation may be done automatically.
The results from each modality are fused by one of the supported methods. Fusion consists of two components: score normalization and combination. In CombSUM, the user may select a weigh factor W 2 [0; 100] which determines the percentage contribution of the image modalities against the textual ones.
For efficiency reasons, only the top-2500 results are retrieved from each modality. If a modality returns less than 2500 items, all non-returned items are assigned zero scores for the modality. When a modality returns 2500 items, all non-occurring items in the top-2500 are assigned half the score of the 2500th item. 3
Let i = 1; 2; : : : be the index running over example images, and j running over the visual descriptors (only two in our setup), i.e. j 2 f1; 2g. Let DESCji be the score of a collection image against the ith example image for the jth descriptor. 3.1
The parameter w controls the relative contribution of the two media; for w = 1 retrieval is based only on text while for w = 0 is based only on image.
Image Modalities Assuming that the descriptors capture orthogonal information, we add their scores per example image. Then, to take into account all example images, the natural combination is to assign to each collection image the maximum similarity seen from its comparisons to all example images; this can be interpreted as looking for images similar to any of the example images. Summarizing, the score s for a collection image against the topic is defined as:
0 s = max @X DESCjiA
i j
Let l 2 f1; 2; 3g be the index running over provided natural languages (or example text queries, i.e. three in our setup), and m 2 f1; 2; 3g running over the textual data streams per language (we consider three: metadata, articles, and undefined language metadata). Let TEXTml be the score of a collection item against the text query in the lth language for the mth text stream.
Fusion consists of two successive steps: score normalization and score combination. 1 ! (1) (2) (3) (4) Text Modalities Assuming that the text streams capture orthogonal information, we add their scores per language. Then, to take into account all the languages, the natural combination is to assign to each collection item the maximum similarity seen from its comparisons to all text queries; this can be interpreted as looking for items in any of the languages. Summarizing, the score s for a collection image against the topic is defined as: s = max l
X TEXTml m Combining Media Incorporating text, again as an orthogonal modality, we add its contribution. Summarizing, the score s for a collection image against the topic is defined as: s = (1 0
1 w) miax @ 1j Xj DESCjiA + w mlax
Query Difficulty Inverse document frequency (IDF) is a widely used and robust term
weighting function capturing term specificity [
QSk = log
Yk N ! i=1 dfi = Xk log N i=1 dfi (5) where dfi is the document frequency (DF), i.e. the number of collection documents in which the term i occurs.
In the Query Difficulty (QD) normalization, we divide all scores per modality by QS, using the df statistics corresponding to the modality. This will promote the scores of ‘easy’ modalities and demote the scores of ‘difficult’ modalities for the query.
For image modalities, we do a similar normalization as defined in the above equation, except that the k terms are replaced by each descriptor’s bins.
Tables 5, 6, 7, and 8 summarize the MinMax+CompDUTH results. Per-modalitytype is the weakest MinMax normalization, followed by per-query-language. Best early precision is achieved by per-modality (best P10) at w = 0:5 and per-index-language (best P20) at w = 0:6. Per-modality at w = 0:7 achieves the best MAP, while perindex-language achieves the best bpref at w = 0:7. Although per-index-language has lower MAP than per-modality, its MAP comparable to per-modality; moreover, perindex-language achieves a higher bpref which signals that we may be retrieving unjudged relevant items. All in all, we conclude that per-index-language is the strongest MinMax normalization. 4.3
Overall, best early precision is achieved by per-index-language MinMax with CompSUM at w = 0:7, and all other measures are optimized by per-modality MinMax with CompSUM at w = 0:8. However, since the 2011 topic set consists of 4 or 5 example images per topic, CompDUTH may show larger effectiveness differences than these on the 2010 topic set; consequently, we will retain CompDUTH runs with 2011 topic set, using per-index-language MinMax and w = 0:6; 0:7; 0:8. All these will result to 5 runs in total. 4.4
QD Normalization
Tables 9 and 10 summarize the QD normalization results with both combination methods. In early precision, the QD normalization works much better with CompSUM than with CompDUTH. The best CompSUM results are achieved for w = 0:4; this run has also the best P10 we have reported so far. In all other measures, although CompSUM is slightly better than CompDUTH, their effectiveness is comparable.
In comparison to the MinMax normalizations, the QD normalization achieves the best initial precision results (when CompSUM is used for combination), and comparable effectiveness to the best MinMax normalization in all other measures.
In summary, we will retain QD+CompSUM at w = 0:4 and QD+CompDUTH at w = 0:3 and 0:5; thus, we will have 3 QD runs in total. 4.5
While we have experimented with radically different normalization and combination methods, our results have not shown a large variance. This suggests that we are ‘pushing’ at the effectiveness ceiling of the 2010 dataset. It is worth noting that most of the runs reported so far have a better MAP and bpref than last year’s best automatic run submitted to ImageCLEF, and a slightly lower but comparable initial precision.3 Nevertheless, a visual inspection of our results reveals that with CompDUTH we are retrieving un-judged items which are sometimes relevant, a fact that most of the times does not seem to get picked up by bpref. 5
3 Last year’s best MAP, P10, P20, and bpref were 0.2765, 0.6114, 0.5407, and 0.3137,
respectively; they were all achieved by the XRCE group [
We reported our experiences and research conducted in the context of our participation to the controlled experiment of the ImageCLEF 2010 Wikipedia Retrieval task. As second-time participants, we improved upon and extended our experimental search engine, http://www.mmretrieval.net, which combines multilingual and multiimage search via a holistic web interface and employs highly distributed indices. Modalities are search in parallel, and results can be fused via several methods.
All in all, we are modestly satisfied with our results. Although our best MAP run ranked our system as the second-best among the other participants’ systems (excluding all relevance feedback and query expansions runs), we believe that the content-based image retrieval part of the problem has a large room for improvement. A promising direction may be using new image modalities such as those based on the bag-of-visualwords paradigm and other similar approaches. Furthermore, we consider score normalization and combination important problems; while effective methods exist in traditional text retrieval, those problems are not trivial in multimedia setups.