This year, XRCE participated in three main tasks of ImageCLEF 2010. The Visual Concept Detection and Annotation Task is presented in a separate paper. In this working note, we rather focus on our participation in the Wikipedia Retrieval Task and in two sub-tasks of the Medical Retrieval Task (Image Modality Classi cation and Ad-hoc Image Retrieval). We investigated mono-modal (textual and visual) and multi-modal retrieval and classi cation systems. For representing text we used either standard language model or a power law (log-logistic or smoothed power law) distribution-based information retrieval model. For representing images, we used Fisher Vectors improved by power and L2 normalizations and a spatial pyramid representation. With theses representations and simple linear classi ers we achieved excellent image modality classi cation both using mono-modal and combined textual and visual information. Concerning the retrieval performances, text based runs performed very well, but visual-only retrieval performances were in general poor showing that even state-of-the art image representations are insu cient to address these tasks accurately. However, we have shown that despite poor visual retrieval results, multimodal runs that combine both visual and textual retrieval scores, can outperform mono-modal systems as long as the information fusion is done appropriately. As a conclusion we can say that our participation in these tasks was successful, as the proposed systems obtained leading positions both in retrieval and modality classi cation and for each type of run: text, image or mixed.
ImageCLEF [
In the case of the Medical Retrieval Task, our aim was to combine Lexical Textual Entailment inspired by Statistical Translation Models with query expansion mechanisms using external resources (in our case Wikipedia pages).
For representing images, we used the Improved Fisher Vectors (IFV) [18, 17]. Fisher Vectors
have been successfully used in our previous ImageCLEF participations [
As the visual retrieval is an important element for the cross-modal similarities technique we
used with success in the Photo Retrieval Tasks of past ImageCLEF sessions [
The rest of the paper is organized as follows. In Section 2 we describe our text retrieval and query expansion models. In Section 3 we brie y describe the image representation with the improved Fisher Vectors. In section 4 we present and compare di erent runs we submitted and we conclude in Section 5. 2
We start from a traditional bag-of-words representation of pre-processed texts where pre-processing includes tokenization, lemmatization, and standard stopword removal. However, in some cases lemmatization might lead to a loss of information. Therefore before building the bag-of-words representation we concatenated a lemmatized version of the document with the original document.
Two information retrieval models were considered: a standard language model (similar to the
techniques used in our past participation in other tasks of ImageCLEF [
To take into account the fact that one is comparing documents of di erent lengths, most IR models do not rely directly on the raw number of occurrences of words in documents, but rather on normalized versions of it. Language models for example use the relative frequency of words in the document and the collection3:
P (wjd) =
d xw + (1 ld )
Pd xdw) P l d d where xdw is the number of occurrences of word w in document d, ld is the length of d in tokens after lemmatization and C is the corpus. Then we can de ne the similarity between the query q = (q1; :::; ql) and a document d using the cross-entropy function:
CE(qjd) =
X P (qijq) log(X P (qijw)P (wjd)) qi w 3 Here we use Jelinek-Mercer interpolation but we can also use e.g. Dirichlet smoothing instead. (1) (2)
Other classical term normalization schemes include the well known Okapi normalization, as
well as pivoted length normalization [20]. More recently, the concept of the amount of information
brought by a term in a document has been considered in several IR models, inspired by the following
observations made by Harter in [
These models are speci ed in three steps: the Divergence from Randomness (DRF)
normalization of terms frequencies, the choice of a probability distribution to model these frequencies in
the corpus and the mean information as the Relevance Score Vector (RSV). In the case of the
log-logistic model, we have (for further details see [
X w2q\d
q xw RSV (q; d) = log P (T fw > tdw) where xdw and xqw are the numbers of occurrences of word w in document d and query q, N and Nw are the numbers of documents in the corpus and the number of those containing w, avgl and ld are the average document length and the length of the document d.
In the case of the smoothed power-law model we have the same steps but the Relevance Score
Vector in the Ranking Model is replaced by (see details in [
RSV (q; d) =
X w2q\d d tw tdw+1 xqw log( w 1 w w ) (3) (4) 2.2
Berger and La erty [
We refer here to a previous work [
To summarize our approach, documents containing images were extracted from the corpus and
we computed a lexical entailment measure on this collection with the model GI-M3 as in [
In this section we give some further details on the text retrieval processes that were designed or applied to a given task.
Wikipedia The Wikipedia Corpus consisted of images with their captions extracted from Wikipedia in di erent languages namely French, English and/or German. In addition, the participants were provided with the original Wikipedia pages in one or several languages in wikitext format. Similarly the textual part of the query was multilingual.
For each image, in addition to the given metadata (image's captions), we extracted from the Wikipedia pages the paragraph of the Wikipedia page in which the image was mentioned. Thus, two indexes were built for the collection: one for the captions (metadata) of the images and one for the paragraphs. The combination of the retrieval scores based on those two types of text, was done by a simple late fusion (mean average) after having normalized the scores between 0 and 1.
To cope with the multilingual nature of the wikipedia collection, we adopted an early fusion
approach that we previously experimented within CLEF'08 Adhoc [
Furthermore we also experimented with query expansion mechanisms using external data. Without such techniques, we would have vocabulary mismatches between queries and documents. For example, for the topic 16 'dermato broma' we found no document containing \dermato broma" as a word. The Lexical Entailment methods using the articles enabled us to address some of the issues, but the coverage of the retrieval system can be improved by using an external knowledge base. Ideally, the use of a thesaurus such as Mesh and the resources provides by UMLS would have been preferable. However, with the lacks of experience (this was our rst participation in the Medical Task) and time for extracting useful information from these resources, we used our usual tools to improve the coverage for a given query with information extracted from Wikipedia pages.
To do so we proceeded as follows. For each query, a set of related pages in Wikipedia was found in order to cover all query terms: topic-1 | thoracic_aortic_dissection topic-2 | Acute_Myeloid_Leukemia topic-4 | congestive_heart_failure topic-5 | brachial plexus nerve_block We ltered redundant pages in order to have a unique coverage of query terms and to disambiguate some terms by taking the sense which was the most similar to the medical collection. Concerning the coverage, if the query contained a multiword expression related to a Wikipedia page, we used this page and not those related to the words within this multiword expression. For example, in topic 1, we can nd the page \thoracic aortic dissection" in Wikipedia, then we use it and not the page \dissection". The disambiguation was done by computing a text similarity with the medical collection and by keeping the sense which had the higher score.
Then, for each of theses pages, we extracted only the hyperlinked text embedded in the body content. For each query, all those new terms were merged and we kept only the 20 terms that were the most frequent in the collection. This can be viewed as a query expansion mechanism provided by Wikipedia pages. Although, this expansion was expected to be noisy, it had the potential to improve recall. Finally, original textual runs were combined with these expanded query runs in order to maintain a high precision (see details in 4.3). 3
As for the image representation, we used an improved version [18, 17] of the Fisher Vector [16]. The Fisher Vector can be understood as an extension of the bag-of-visual-words (BOV) representation. Instead of characterizing an image with the number of occurrences of each visual word, it characterizes the image with the gradient vector derived from a generative probabilistic model. The gradient of the log-likelihood describes the contribution of the parameters to the generation process.
Assuming that the local descriptors I = fxt; xt 2 RD; t = 1 : : : T g of an image I are
generated independently by Gaussian mixture model (GMM) u (x) = PM
i=1 wiN (xjmui; i), I can be
described by the following gradient vector (see also [
T GI = 1 X
T
t=1
r log u (xt)
(5)
where = fwi; i; i; i = 1 : : : M g are the parameters of the GMM. A natural kernel on these
gradients is the Fisher Kernel [
K(I; J ) = GI 0F 1GJ ;
F = Ex u [r log u (x)r log u (x)0] : (6) where F is the Fisher information matrix. As it is symmetric and positive de nite, F 1 has a Cholesky decomposition F 1 = L0 L and K(I; J ) can be rewritten as a dot-product between normalized vectors G with: GI = L GI . We will refer to GI as the Fisher Vector (FV) of the image I.
In the case of diagonal covariance matrices i (we denote by i2 the corresponding variance vectors), closed form formulas can be derived for GwIid , GIid , GIid , for i = 1 : : : M , d = 1 : : : D (see details in [17]). As we do not consider Gwid (the derivatives according to the weights), GI is the
I concatenation of the derivatives GIid and GIid and is therefore N = 2M D-dimensional.
While this representation was successfully used with the L1 normalization and L1 norm based
similarity between the Fisher Vectors in our previous ImageCLEF participation [
P N (GIn ) = sign(GIn )jGIn j ; n = 1 : : : N: (7) The aim of this normalization is to make the distribution of features in a given dimension n less peaky around zero. Then, these vectors are further L2 normalized to discard image-independent (i.e. background) information (see further details for both normalizations in [17]).
Another improvement made to our Fisher Vector based image representation was the use of
the spatial pyramid representation by Lazebnik et al . to take into account the rough geometry
of a scene [
As low level features we used our usual (see for example [
The Wikipedia Retrieval task consists of multilingual and multimedia retrieval. The collection contains images with their captions extracted from Wikipedia in di erent languages namely French, English and German. In addition, participants were provided with the original Wikipedia pages in wikitext format. The task consisted in retrieving as many relevant images as possible from the aforementioned collection, given a textual query translated in the three di erent languages and one or several query images.
We submitted di erent types of runs: mono-media runs and multimedia (mixed) runs with di erent fusion approaches.
Concerning the pure visual retrieval, we used 6 Improved Fisher Vector (IFV), corresponding to the two di erent low level features (ORH and COL) with the 3 spatial-layout (1x1, 2x2, 1x3) as described in section 3. The 6 IFVs were used independently to rank the Wikipedia images using the dot product as similarity measure and the 6 scores were weighted giving higher weights (0.7) to the IFV based on orientation histograms compared to the IFV based on color (0.3) before averaging them. The pure visual run results are mentioned in line 1 of Table 1. The performances are very poor and this shows that even recent advances in visual similarities [17] are not su cient to address the Wikipedia Retrieval task e ciently.
Pure text based retrieval performs much better. Details on the de nition of the methods and the
underlying retrieval models used are mentioned in section 2. Accordingly, from lines 2 to 4 of Table
1, we report 3 pure text runs which are based on recent models presented in [
Runs reported from lines 5 to 16 are mixed, i.e. they make use of both visual and textual similarities. We grouped the runs according to the text run used in the combination. Consequently, from lines 5 to 10, we report the results of mixed runs that used the text run from line 3 (T2=LGD ALL METANOPRF PARAGPRF20) whereas from line 11 to line 16 we report results of the mixed runs that used the text run from line 4 (T3=LGD ALL META PARAG). The runs with a ? symbol (lines 5 and 12) in Table 1 were not submitted but we added them in order to have a better comparison between the results obtained using both aforementioned text runs.
As explained earlier, T2 gave a better result than T3. Moreover, whatever the combination technique we used, we always observe that the run using the PRF on the paragraphs dominates the other one. Therefore, we will only comment on the results of mixed runs reported from lines 5 to 10 in Table 1. Similar conclusions can be deduced for the runs from line 11 to line 16.
From our past experiences with ImageCLEF [
Consequently, our rst fusion strategy was to start with using the text similarities as a preltering stage before applying the visual similarities. In practice, we rst selected the top 2000 images according to the textual similarities. Then, we re-ranked this pre- ltered top-list using the visual similarities only. This pre- ltering step based on textual similarities allowed us to signi cantly increase the pure visual run from 5.53% (line 1) to 18.05% of MAP (line 5). In particular, it gives better results than both image and text runs in terms of Precision at 20 (P@20). Still, this run entitled SEQ RERANKING-T2 (line 5) does not perform as well as pure textual similarities T2 (line 3) in terms of MAP.
Then, going beyond this simple re-ranking strategy, we investigated the combination of the re-ranked run SEQ RERANKING-T2, with the pure textual run T2. We basically normalized the two runs so that the scores were between 0 and 1 and we applied di erent aggregation functions. The di erent results are shown from line 6 to line 10.
Our rst approach consisted in applying a simple weighted mean average operator. In that case, the submitted runs were given by the following aggregated score: wRsT RV (I) + wT sT R(I) (8) where I is an image of the collection, sT RV is the normalized score distribution based on the visual similarities but after having selected the top 2000 images according to the textual similarities, sT R is the normalized score distribution based on the textual similarities and wR and wT are their respective weights in the late fusion method.
Lines 6, 8 and 10 with runs' titles beginning with FUSION are the results obtained with the following parameters: line 6 with wR = 0:5 and wt = 0:5; line 8 with wR = 0:7 and wt = 0:3; and line 10 with wR = 0:3 and wt = 0:7.
Our second fusion technique uses an aggregation operator that was introduced in [
Our last combination approach follows the idea that image similarities are less reliable then text similarities and when combining both of them one should adapt the combination weights accordingly. In that perspective, we propose to set the combination weights as functions of the visual similarity values. We particularly tested linear functions with wR(sT RV (I)) = sT RV (I) and wT (sT Rv(I)) = 1 sT RV (I). This leads to the following aggregation method: (sT RV (I))2 + (1 sT RV (I))sT R(I) (10)
Line 9 corresponds to the results obtained using the aforementioned aggregation function with = 0:7 (CONF -T2).
From Table 1, we can make the following observations: { Image similarities are not su cient to address the task accurately and text similarities perform much better. { Using as a preliminary stage the text similarities in order to select a rst relevant list of images and re-ranking the latter list on the basis of image similarities only allows us to boost the precision (P@20) but not the MAP. The important conclusion to make here and towards multimedia retrieval, is that visual similarities might be relevant but they need to be ltered. In that case, a text-based pre- ltering step is very important in order to select relevant visual similarities by removing images that are not relevant \semantically" despite of being similar to the image query \visually". Visual 0.0553 0.2686 Textual 0.2361 0.4393 Textual 0.2045 0.4200 Textual 0.1903 0.4000 5 ? SEQ RERANKING-LGD ALL METANOPRF PARAGPRF20 Mixed 0.1805 0.4493 6 FUSION TEXTLGD ALL METANOPRF PARAGPRF20 Mixed 0.2765 0.5193 7 CONFAGmin-TEXTLGD ALL METANOPRF PARAGPRF20 Mixed 0.2681 0.5257 8 FUSION TEXTLGD ALL METANOPRF PARAGPRF20-Rer7 Text3 Mixed 0.2627 0.5407 9 CONF0.7-TEXTLGD ALL METANOPRF PARAGPRF20 Mixed 0.2532 0.4986 10 FUSION TEXTLGD ALL METANOPRF PARAGPRF20-Rer3 Text7 Mixed 0.2493 0.4743 11 SEQ RERANKING-LGD ALL META PARAG Mixed 0.1747 0.4471 12 ? FUSION TEXTLGD ALL META PARAG Mixed 0.2660 0.5193 13 CONFAGmin-TEXTLGD ALL META PARAG Mixed 0.2575 0.5164 14 FUSION TEXTLGD ALL META PARAG-Rer7 Text3 Mixed 0.2527 0.5336 15 CONF0.7-TEXTLGD ALL META PARAG Mixed 0.2424 0.4907 16 FUSION TEXTLGD ALL META PARAG-Rer3 Text7 Mixed 0.2415 0.4664 17 best non-XRCE run Textual 0.2251 0.3871 { Combining textual similarities with text-based pre- ltered visual similarities, dramatically outperforms mono-media runs. Any of the aggregation functions used in fusing the two aforementioned runs allows us to increase both the MAP and the P@20 of the best mono-media runs. It appeared that the best aggregation strategy was the simplest one: the mean average. In that case, it improved the text run performance from MAP=20.45% to 27.65% which corresponds to more than a 35% increase. 4.2
Imaging modality is an important aspect of the image for medical retrieval. Therefore, within the
Medical Retrieval Task in 2010 [
In our experiments we investigated mono-modal and mixed modality based classi cation. Concerning the pure visual-based classi ers, we trained 2 linear classi ers per modality (using the one-versus-all scheme) corresponding to the two di erent low level features (ORH and COL). Linear SVM classi ers with hinge loss using the primal formulation were trained with the Stochastic Gradient Descent (SGD) algorithm5 using the Improved Fisher Vectors (IFV) as described in section 3 and a single spatial layout (1x1). The SVM scores were combined by weighted averaging color and SIFT features for color images. We only used the ORH based IFV for gray-scale images. The weights for each modality were tuned by maximizing classi cation accuracy using a 5-fold cross validation scheme on the training set.
Concerning our text based modality classi cation, we used two di erent representations and hence two classi ers. The rst one was based on a pattern matching (PM) technique where we searched for the modalities in the image captions. The second one was based on a binarized bagof-words representation.
The pattern matching was based on the information describing the modalities in the modalityClassi cationREADME.txt le. Indeed, this le provides for each modality a description that 5 An implementation is available on Leon Bottou's web-page: http://leon.bottou.org/projects/sgd. contains a list of expressions mainly related to image sub-modalities grouped in that category (i.e. the PX modality description \PX: optical imaging including photographs, micrographs..." contains expressions \PX",\optical imaging", \photograph", \micrograph"...). Therefore, for each image, we detected (matched) these expressions in the corresponding captions. However, within the corpus, many documents contain several images with the same caption while the latter do not necessarily have the same modalities. Therefore, we reduced6 the image caption for a targeted image name, to: { sentences related to the image reference7, { sentences containing multiple image references including the targeted one, { and the introductory sentences (all sentences before the rst occurrence of an image reference). Consequently, the outputs of the pattern matching (PM) decision function can be a single modality, a set of potential modalities or an empty set.
The binarized bag-of-words representation consisted in a vector indicating for each word whether it appears in this document or not (in our case image caption). Note that for this representation we did not apply the aforementioned caption reduction but used them as they were provided. The feature vectors of the training set made of 2000 items with their modality labels, were then used to train a classi er per modality (one-versus-all scheme). The aim of this representation was to go beyond pattern matching and learn other words related to di erent modalities. To train the linear classi ers we used the logistic regression classi er (LRC) from the liblinear toolkit8 with Laplace Prior (L1 norm).
To combine the outputs of the two classi ers, we proceeded as follows: { If we have no output from PM, the modality is given by: c = arg max scT (I) c2C (11) where scT (I) is the output of the classi er LRC trained for the modality c. { If PM outputs one or more modalities and c is amongst them, the modality is again c . { Otherwise the modality amongst the PM outputs that has the highest score sc(I) was selected (C in equation 11 is reduced to the PM outputs).
In the case of mixed modalities, we rst combined the LRC scores with the SGD visual scores (both normalized to get values between 0 and 1) before combining with the PM output as described above (scT (I) is replaced by the combined scores in equation 11).
The results of the di erent classi ers are shown in Table 2. As these results show, the pure text modality classi er slightly outperforms the visual only information based classi er and both of them were outperformed by the combination of visual and textual modalities. 6 When it was possible to detect automatically. 7 For example references might be (a) or (a-c) 8 http://www.csie.ntu.edu.tw/~cjlin/liblinear/ 4.3
For this task [
In our experiments we wanted to investigate both mono-modal and mixed-modality retrieval. Visual Only Concerning the pure visual retrieval, in contrast to our Wikipedia runs, we used only the 3 Improved Fisher Vector (IFV), corresponding to the orientation histograms (ORH) with the 3 spatial-layout (1x1, 2x2, 1x3). The 3 IFVs were used independently to rank the Wikipedia images using the dot product as similarity measure and the 3 scores were simply summed9 and then normalized to be between 0 and 1. We denote this score by sV R(I) (visual retrieval score). This score was then further combined with the visual only modality score as follows: { First the query images were classi ed by the SGD classi ers using ck(t) = arg maxc2C sSGD(Ikt ), c where ck(t) is the modality assigned to the image Ikt of the topic t. { If for all images Ikt in a topic (if multiple image query) the same modality c was assigned, c the scores sSGD(I) were added to sV R(I) to rank the images of the dataset. The aim was to boost the scores of the images corresponding to that modality. { In contrast, if we had di culty to retain a given modality (as e.g. for the last topics) we preferred to rank the images based only on the retrieval scores sV R(I).
The performance of this run shown in line 1 of Table 3 is very poor, leading to the conclusion that visual only information is insu cient to address the task.
Text Only Concerning the textual retrieval, we submitted two types of text-only retrieval. For the rst one, we simply ranked the text corresponding to some cross-entropy-based text similarity or Relevance Score Vector between the query text and the image caption. For the second, we aggregated these scores with text-only classi cation scores.
First we describe a few textual retrieval runs (see section 2 for technical details). Several such textual runs can be created depending on: { Di erent information retrieval models: Language Model or log-logistic model (the latter was used if LOGIT or LGD appears in the run names, the former otherwise). { Di erent smoothing parameters: we mostly used the classical Jelinek-Mercer interpolation except for the run DIR TXT (line 2) where the Dirichlet smoothing was used. { Di erent query and document enrichment models: statistical translation model (referred to as
AX) and/or10 using the Wikipedia Corpus (referred to as WIKI). { Di erent statistics: using Chi2 with the log-logistic model is referred as CHI2 LOGIT and using
Chi2 statistics instead of GI-M3 in the statistical translation model is referred as CHI2AX. While only a single pure text retrieval run was submitted (line 2), nevertheless, we describe them here since they were used (referred to as textual retrieval score with sT R(I)) in the combination with the modality classi cation outputs and/or with the visual retrieval scores (see below). To aggregate the textual retrieval scores sT R(I) with the text based modality scores, we rst extracted the modality c from the query text by pattern matching (PM) and selected the corresponding sLRC (I) score11 (see section 4.2). If no modality is found, only the sT R(I) is used to rank. Two such c 9 Note that, due to the linearity property of the dot product, this is equivalent to the dot product of the concatenation of all three IFV vectors. 10 Note that WIKI AX means that we combined the lexical entailment based scores with the scores obtained with Wikipedia based query expansion to obtain sT R(I) for further fusion. 11 This is the output of the LRC image modality classi er trained on the binarized bag-of-word feature vectors for the modality c. runs were submitted (lines 3 and 4 in Table 3). While not directly comparable, we can nevertheless see that these runs led to better performances than DIR TXT suggesting that combining text retrieval with modality classi cation helps. Furthermore, WIKI AX MOD late led to a much better MAP than CHI2AX MOD late, showing the bene t of using external data (Wikipedia) for query expansion.
Mixing Textual and Visual Information Finally, we further used both visual and textual information. We also experimented with further combining these scores with modality classi cation scores as follows: wV sV R(I) + wM scLRC (I) + wT (t)sT R(I): (12) Here sV R(I) and sT R(I) are the normalized visual and textual retrieval scores (see above) and c sLRC (I) is the output of the image modality classi er for the modality c using only textual information (LRC scores). The modality c is extracted from the query text of topic t by pattern c matching (PM) and sLRC (I) is set to zero if no modality was found.
We made the weights of the textual retrieval score wT (t) dependent on the type of topic t. Hence, we used 0.5 for topics with query type V isual, 1 for type M ixed and 1.5 for type Semantic, in order to increase or decrease the importance of the textual score. The weights wV and wM took values 1 or 0. Hence setting wV to 0 lead to pure textual based retrieval runs discussed above (such as WIKI AX MOD late and CHI2AX MOD late depending on the sT R(I) used). If we further set wM to 0, we get runs without using the modality (e.g. DIR TXT). If we set wV = 1 and wM = 0 we get the classical late fusion (e.g. AX LGD IMG late) and setting wV = 1 and wM = 1 we get a late fusion based on textual and visual retrieval scores combined with modality classi er scores (CHI2 LOGIT IMG MOD late, WIKI LGD IMG MOD late).
Finally the runs AX rerank and AX rerank comb consist in ranking images based on sT R(I) and then re-ranking the top N=1000 relevant images using respectively sV R(I) and sV R(I) + wT (t)sT R(I).
The MAP, bPref and P10 results of the runs described above are shown in Table 3. These results show that the poor performance of image retrieval for this task, decreases the performances when the visual modality is not appropriately combined with the other modalities (as in AX rerank or WIKI AX IMG MOD late). We can also note that the run AX rerank comb obtained the best performance on (MAP and bPref) while the run WIKI AX MOD late has the best performing P10 among all runs.
However, it is di cult to make conclusions about these runs that performed better than others from this table. Indeed, we need to evaluate di erent text runs with and without combining them with modality classi er and/or visual retrieval scores. Our intention is to do such analysis as soon as the image relevance scores are made available by the organizers. 5
This year we have participated with success in two new main tasks, namely the Wikipedia Retrieval Task and two sub-tasks of the Medical Retrieval Task (Image Modality Classi cation and Ad-hoc Retrieval). In all cases, we obtained leading positions both in retrieval and modality classi cation, and for each type of run: text-only, visual-only and mixed. We achieved excellent text based retrieval results and despite the fact that pure visual based retrieval led to poor results, when we appropriately combined them with our text ranking we were able to outperform the latter showing that multi-modal based systems can be better than mono-modal ones.
This work was partially supported by the European Project PinView FP7/2007-2013 and the French National Project Fragrances ANR-08-CORD-008. We would like also to acknowledge Craig Saunders for his proofreading and useful comments on the paper. 16. F. Perronnin and C. Dance. Fisher kernels on visual vocabularies for image categorization. In CVPR, 2007. 17. F. Perronnin, J. Sanchez, and Y. Liu. Large-scale image categorization with explicit data embedding.
In CVPR, 2010. 18. Florent Perronnin, Yan Liu, Jorge Sanchez, and Herve Poirier. Large-scale image retrieval with compressed sher vectors. In CVPR, 2010. 19. Adrian Popescu, Theodora Tsikrika, and Jana Kludas. Overview of the wikipedia retrieval task at imageclef 2010. In Working Notes of CLEF 2010, Padova, Italy, 2010. 20. Amit Singhal, Chris Buckley, and Mandar Mitra. Pivoted document length normalization. In SIGIR '96: Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, pages 21{29, New York, NY, USA, 1996. ACM. 21. Yiming Yang and Jan O. Pedersen. A comparative study on feature selection in text categorization. In Proceedings of ICML-97, 14th International Conference on Machine Learning, pages 412{420, 1997.