This paper reports our multimedia information retrieval experiments carried out for the ImageCLEF track 2009. In 2008, we proposed a multimedia document model defined as a vector of textual and visual terms weighted using a tf.idf approch [5]. For our second participation, our goal was to improve this previous model in the following ways: 1) use of additional information for the textual part (legend and image bounding text extracted from the original documents, 2) use of different image detectors and descriptors, 3) new text / image combination approach. Results allow to evaluate the benefits of these different improvements.
ImageCLEFwiki is a multimedia collection where each document is composed of
text and one image. User needs are represented by queries (”topics”), which are
also multimedia. Therefore, a multimedia document model is necessary to handle
such a collection. In 2008, we proposed a first model that combines text and
image information for multimedia retrieval [
The document model we defined for ImageCLEF 2008 lets us rank documents depending on the query using different methods. Firstly, we explain the key features of our approach to rank documents according to a query using only textual information. Secondly, we describe how we extend the method to handle the visual information. Finally, we present our method for combining textual and visual results. 2.1
As in the vector space model introducted by Salton et al. [
k1ni,j ni,j + k2(1 − b + b d|advi|g ) where ni,j is the occurrence of the term tj in the document di, |di| the size of the document di and davg the average size of all documents in the corpus and k1, k2 and b are three constants.
idfj = log |D| − |{di|uj ∈ di}| + 0.5
|{di|tj ∈ di}| + 0.5 where |D| is the size of the corpus and |{di|tj ∈ di}| the number of documents where the term tj occurs at least one time.
If we consider a query qk as a short document, we can represent it as a vector of weights. A score is then computed between the query qk and a document di as shown in table 1. The main difference between score1 and score2 is the representation of the query. In the first score, the weight of the query is defined by its tf only while in the second score the weight is equal to tf.idf . Note that for tfk,j, b = 0 because |dk| and |davg| are not defined for a query.
Different sources of text are available. The legend provided with images is
often very short and sometimes useless: for example, when the text deals with the
copyright of the image or when it gives details about the user who uploaded the
image. In order to gain information, we aim at using the original text extracted
from the wikipedia documents in which images appear. We consider a text
fragment aroung the image. The size of the window is tuned using wikipediaMM
2008 collection as a training collection. We add this text to the legend of the
image and we index both the added text and the original legend. The indexing
is performed with the Lemur software[
In order to combine the visual information with the textual one, we also
represent images as a vector of weights. Provided we are able to extract visual
words from images, it is possible to use the tf.idf formula in the same way
as in the textual model. It is therefore necessary to create a visual vocabulary
V = {v1, ..., vj , ..., v|V |} as in [
For each of our 3 descriptors, we apply a k-means algorithm [
Then, each image can be represented using a vector of visual terms. Local features are first calculated using one of the 3 descriptors. Then visual terms are determined by seeking, for each feature, the closest visual term (according to the euclidian distance) in the corresponding visual vocabulary.
In the same way as for textual words, the weight of each visual term is computed using a tf.idf approach. 2.3
In order to combine textual and visual results we use two different methods. The first one is a simple intersection between the results obtained with the textual query and with the visual one. The second one corresponds to a linear combination between the textual and the visual scores.
score(qk, di) = αscoreV (qk, di) + (1 − α)scoreT (qk, di)
The α parameter lets us add more or less visual information. We calculate its optimal value using the queries from the ImageCLEFwiki 2008 track as a training set.
Using the model described in the previous section, we present runs submitted to ImageCLEFwiki and the results we obtained. 3.1
run id run score text image combination LaHC 1 LaHC TXT okapi score1 legend - LaHC 2 LaHC TXT tfidf score2 legend - LaHC 3 run inter TXT IMG Meanstd score2 legend meanstd intersection LaHC 4 run inter TXT IMG Sift score2 legend sif t1 intersection LaHC 5 run TXTIMG Meanstd 0.015 score2 legend meanstd α= 0.015 LaHC 6 run TXTIMG Meanstd 0.025 score2 legend meanstd α= 0.025 LaHC 7 run TXTIMG Sift 0.012 score2 legend sif t1 α= 0.012 LaHC 8 run inter TXT IMG Siftdense score2 legend sif t2 intersection LaHC 9 run TXT 100 3 1 5 score2 100 char legend LaHC 10 run TXT 50 3 1 5 score2 50 char legend LaHC 11 run TXTIMG 100 3 1 5 meanstd score2 100 char meanstd α= 0.025 LaHC 12 run TXTIMG 50 3 1 5 meanstd score2 50 char meanstd α= 0.025 LaHC 13 run TXTIMG Siftdense 0.084 score2 legend sif t2 α= 0.084 – meanstd: regular partitioning + color descriptor – sif t1: MSER detector + sift descriptor – sif t2: regular partitioning + sift descriptor – legend: text of the image document – n char: size (n characters) of the text window around the image in the original wikipedia documents
All the runs are entirely automatic and are summarized on table 2. We define a baseline, LaHC 1, that corresponds to a pure text model. It uses only textual terms for the query and scoring of documents. We calculate the score1 for each image using terms of the textual content. The image name or bounding characters are not considered. We do not use neither feedback nor query expansion. Since score1 is applied, the query terms are weighted with their frequency tf .
Using only the text, we perform 3 other runs: the LaHC 2 is the same as the baseline except that the query is represented by its tf.idf rather than its tf . The LaHC 9 and LaHC 10 are two other text only runs that make use of the bounding text around the image in the original wikipedia document. The LaHC 9 adds 100 characters before and after the image while the LaHC 10 adds 50 characters.
All other runs exploit both the textual and the visual information of documents. The LaHC 3, LaHC 4 and LaHC 8 are obtained after an intersection of the text only query results (LaHC 2) and the image query using the meanstd, the sif t1 and the sif t2 descriptors. The other runs are obtained from a linear combination of the textual and the visual scores. LaHC 5, LaHC 6, LaHC 7 and LaHC 13 use the textual scores of LaHC 2 and the visual scores of a visual descriptor (meanstd, sif t1 and sif t2). LaHC 11 and LaHC 12 combine the textual scores of LaHC 9 and LaHC 10 with the visual scores of the meanstd descriptor. 4
All the obtained results are summarized in Table 3. On the whole results, our team ranks 2nd on 8 participants1. As we can see, the best results are obtained when we combine the image bounding text and the meanstd descriptor. We could have obtained better results if we had combined the image bounding text and the sif t2 descriptor. Indeed, comparing results of text-image runs LaHC 6, LaHC 7 and LaHC 13, we can notice that the best visual descriptor is sif t2 , followed by meanstd and sif t1. The last three results obtained after an intersection are the worst results, but if we compare them in term of precision, they are the best ones. Indeed, one document over six is relevant. As a rule, combining the textual information with the visual one always improves the results and return more relevant documents which is very encouraging.
rank run
map num ret num rel ret 1 http://imageclef.org/2009/wikiMM-results
In this article we proposed improvements to our multimedia model we introduced
in [
For future work, we aim to combine the textual information with more than just one visual descriptor. Moreover as the visual information importance depends on the query, we also plan to learn a different α parameter for each query. 6
This work was partly supported by the LIMA project2 and the Web Intelligence project3 which are 2 Rhoˆne-Alpes region projects of ISLE cluster4.