This paper reports our multimedia information retrieval experiments carried out for the ImageCLEF track (ImageCLEFwiki). The task is to answer to user information needs, i.e. queries which may be composed of several modalities (text, image, concept) with ranked lists of relevant documents. The purpose of our experiments is twofold: firstly, our overall aim is to develop a multimedia document model combining text and/or image modalities. Secondly, we aim to compare results of our model using a multimedia query with a text only model.
The capacity of data storage increases constantly making possible the collection of large amount of information of all kinds, as texts, images, videos or combinations of them. In order to retrieve documents in such amount of data, information retrieval techniques tailored for the data types are required.
The ImageCLEF collection consists of multimedia documents made up of text and images. In this paper, we present our participation to the ImageCLEFwiki task. Our research goals are twofold: First, we aim to propose a multimedia document model combining text and image modalities adapted for multimedia retrieval. Second, we want to study the performance of our model compared to a text retrieval approach. In order to benefit from our long time experience with textual model, we develop a vector-based model composed of textual and visual terms. The textual terms correspond to words of the text. The visual terms are obtained through a bag of words approach. Local colour descriptors are extracted from images and quantized by k-means leading to an image vocabulary.
After presenting our model, we will describe the runs we submitted. Then, we will comment on the results we obtained and conclude. 2
ImageCLEFwiki is a multimedia collection where documents are composed of text and image. User needs are represented by queries (”topics”), which are also multimedia (text, image and concept). Hence a multimedia document model is necessary to handle such a collection. We focus our work on combining textual and visual information without using the concept field of the topics. Before explaining our model, we will describe the collection data. 2.1
The ImageCLEF Wikipedia collection is composed of 151’519 multimedia XML documents and 75 multimedia topics. The documents are made up of an image and a short text. Images are in a common format (jpeg and png) and their sizes are heterogeneous. They depict either drawings, paintings or screenshots. They are in colour or in black and white.
The textual part of a document is unstructured and consists of a description of the image, information about the Wikipedia user who has uploaded the image, or the copyright of the image. The average number of words per document is about 33 words.
It is also possible to get the whole Wikipedia article from which each image is extracted. The ImageCLEF Wikipedia collection provides 75 topics whose only 28 contains an image as example of the expected answer. For the text part, these queries were composed of about 2 or 3 words. 2.2
One of the most known document model in textual information retrieval is the vector space
model introduced by Salton and al. [
(k1 + 1) ∗ ni,j ni,j + k1 ∗ (1 − b + b ∗ d|advi|g ) where k1 = 1.2 and b = 0.75 are two constants empirically defined in the Okapi formula, ni,j is the occurrence of the term tj in the document di, |di| is the size of the document di and davg is the average size of all documents in the corpus.
The idfj (inverse document frequency) measures the discriminatory power of a term tj and is
defined as [
The weight wi,j is then obtained by multiplying tfi,j and idfj. This weight is higher when the term tj is frequent in the document di but rare in the others. In our case, the size of our vocabulary T is 217’323 after applying a Porter stemming. The indexing has been performed with the Lemur software1. 2.3
In order to combine the visual information with the textual one, the images are also represented by a vector of visual words. It is therefore necessary to create a visual vocabulary V {v1, ..., vj, ..., v|T |}. Our method consists in partitioning all images into 16x16 grids, a minimum of 8x8 pixels being required for each cell. It leads to about 256 cells per image, or about 38 million over all images.
For each cell, we compute a feature vector that contains the colour properties of the region. The vector is a 6 dimensional vector. The 6 dimensions correspond to the mean and the standard deviation for R+G+B , R+G+B and R+G+B where R, G and B are the red, green and blue comR G
3∗255 ponents of the cell.
We apply a k-means algorithm over 4 millions of cells randomly selected within the 38 millions of cells to obtain 2’000 visual terms, which correspond to our visual vocabulary V . 2’000 has been chosen arbitrarily while 4 millions corresponded to the maximum number of cells we could compute due to the complexity of the k-means. Each visual term represents a cluster of feature vectors.
Then, each new image can be represented using a vector of visual terms. It is decomposed into a 16x16 grid and the local features are computed. Each cell is then assigned to the closest visual term from our visual vocabolary V , using the euclidean distance. 3
Using the model described in the last section, we present our approach for multimedia documents retrieval from multimedia queries. Then we describe the runs we submitted to ImageCLEFwiki in order to evaluate our model. 3.1
As mentionned before, the ImageCLEFwiki topics are composed of text, image and concept
modalities. However, our model is designed to take only into account text and image modalities. Our
retrieval approach consists in computing a similarity score between each document di and the
query qk using the Okapi method [
X tfi,j ∗ idfj ∗ qtwk,j uj∈qk qtwk,j = (k3 + 1) ∗ nk,j k3 + nk,j where k3 = 7 is a constant defined in the okapi formula and where nk,j represents the occurrence of the term uj in the query qk. This score is higher when the term uj is frequent in the document di but rare in the others and is weighted by the occurrence of uj in the query.
Let us insist on the fact that the term uj can be either a textual term tj or a visual term vj and that queries can be composed of textual terms only, visual terms only or both textual and visual terms which allows to perform text only queries, image only queries or multimedia queries.
The textual terms used for queries are those provided with topics. When visual terms are used, they are extracted either from the topic images or from the collection images. This will be detailed in the following section. 3.2
We have submitted 6 runs to ImageCLEFwiki 2008, labelled from run 01 to run 06. Part of them are automatic (auto), others required manual selection of some relevant documents (man). Thanks to these 6 runs, we aim to study several aspects of our model, as the choice of the visual words and local features, the combination of textual and visual words for a query and the performance improvements obtained when adding visual information to a pure textual model. All these runs are summed up in table 1.
run name LaHC run01 LaHC run02 LaHC run03
LaHC run06 first query (R1) t t t t t t run type auto auto man auto man man auto • auto: automatic run; man: selection of relevant images by user • v10: automatic selection of the first 10 results from R1 • v100: manual selection of the relevant documents in the first 100 results from R1 • iq: query image • vt: v10 or v100 • vq: visual words extracted from the query image results
R1 R2 R2 R2
R2 R1∩ R2
We define a baseline that corresponds to a pure text model. It uses only textual terms for the query and scoring of documents. This run is run 01 and its results are noted R1. We did not use neither feedback nor query expansion for this automatic run.
All other runs exploit both textual and visual information of documents. They consist in two successive queries. The first one corresponds to the baseline (R1), while the second one is a visual or textual and visual query, either automatic or manual (R2). R2 is automatic when visual terms are extracted automatically from some images: we chose to select all the visual words of the top 10 retrieved documents issued from the baseline (v10), assuming these results as relevant. R2 is manual when the user is asked to select relevant documents among the first 100 results of R1(v100). There is no limit in the number of selected documents.
Run 02 and run 06 are automatic and only visual runs. For run 02, all the results of the second query are given as answer (R2) while for run 06 an intersection is performed between the results of the first and second query (R1∩ R2). This intersection is interesting as it emphasises the gain of the visual information use. Indeed, if the intersection is not null, it means that the visual query leads us to find documents which were not in the baseline results.
Run 03, run 04 and run 05 are manual runs. Run 03 is visual, while run 04 and run 05 are multimedia. For run 03, all the visual words of the selected images are used to perform the second query. For run 04 and run 05 we did a query expansion in order to analyse the combination of textual and visual information (t+ v100). We kept the textual words of the initial query and added some visual words. For run 05, these words come from the manual selected images. The run 04 proceeds as run 05, unless a query image is provided for the considered topic. In that case, the second query is composed of the visual words extracted from the query image. Thanks to these two runs, we can study the influence of the number of relevant images used for queries. 4
These results provide a basis to evaluate our approach on different aspects: the choice of the visual words, the way to combine textual and visual words and the performance improvements obtained when adding visual information to a pure textual model.
Concerning the choice of the visual words, results from run 02 and run 03 show us that they are meaningful in some particular cases. Indeed, using only the visual words, we are able to retrieve relevant documents which were not used for the query. For example, in run 03, we retrieved 42 relevant documents for the query ”blue flowers” whereas we had just selected 9 images. However there are some topics for which the results are bad. Only a third of the topics leads to more relevant documents than the number used for the query. This is due to the visualness of the query. It is obvious that for a query like ”blue flower”, the visual information is more useful than for a query like ”peace anti-war protest”.
Moreover, we observe from the results of run 04 and run 05 that only one image is insufficient to obtain good results. For topics which provide an image query, the results are always worth than the obtained results with several selected images. This can be explained by the fact that image query were not enough representative. Furthermore it is obvious that one image can not be enough expressive compared to several relevant images. The conclusion about the combination of textual and visual words is surprising. The comparison between run 03 and run 05 tells us that adding directly the text to the same visual words leads to worth results. The explanation is not obvious, but it seems to be that the proportion between textual and visual parts is unbalanced. Where we only have 2 or 3 textual words, we have several thousand of visual words.
Results also show that visual information bring complementary relevant documents that were not found with the text query. These initial results are promising and encourage the development of our multimedia model. The comparison between run 01, run 02 and run 06 inform us that 214 new relevant documents were retrieved with the visual query only over all topics. To take an example, 32 documents were found with run 01 and 30 with run 02. The intersection of both results coming from run 06 gives 13 shared documents. Thus, 17 relevant documents were retrieved with the visual information only. 5
We proposed a vector based model for multimedia documents. Thanks to the ImageCLEFwiki collection, we were able to test our model as this collection provide visual and textual information. We obtained encouraging results with visual words only based on coloured features. However, we did not take into account the specificity of the collection. For example, a lot of information as the copyright of the image is useless information for a retrieval task. Moreover, image names could be studied in order to improve search. For example, TheWhiteHouse.jpg could be replace by The White House.
For future work, as our local image features are basic, other features such as texture and edge information would surely lead to performance improvements. We also plan to automatically select the number of visual words using machine learning approaches. Finally, we aim to combine more efficiently textual and visual information. 6
This work is supported by the LIMA project2 and the Web Intelligence project3 which are 2 Rhoˆne-Alpes region projects. 2LIMA project: http://liris.cnrs.fr/lima/ 3WI project: http://www.web-intelligence-rhone-alpes.org/