One challenge for this Wikipedia task is the training of visual models. We propose in this paper to link each topics one or few visual concepts of the Visual Concept Detection (VCDT) CLEFimage08 task, even if three topics do not t VCDT concepts. We use the same models and features than in our VCDT systems. We show that our visual IMG NOFB run is the second best model in this campaign for this run type. So it can be concluded that our VCDT visual concept partly t this task. Moreover we show that even a simple boolean text analysis overcomes the best IMG NO FEEDBACK run, which has 0.0037 MAP, against 0.399 for our TXT NOFB text run. This emphases the fact that visual retrieval for Wiki task is very di cult.
The ImageCLEF's 2008 wikipediaMM task provides a test bed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a larger scale and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs.
The given images are associated with unstructured and noisy textual annotations in English. This is an ad-hoc image retrieval task; the evaluation scenario is thereby similar to the classic TREC ad-hoc retrieval task and the ImageCLEFphoto task: simulation of the situation in which a system knows the set of documents to be searched, but cannot anticipate the particular topic that will be investigated (i.e. topics are not known to the system in advance). The goal of the simulation is: given a textual query (and/or sample images and/or concepts) describing a user's (multimedia) information need, nd as many relevant images as possible from the wikipedia image collection.
The characteristics of the (INEX MM) wikipedia image collection allow for the investigation of the following objectives: how well do the retrieval approaches cope with larger scale image collections? How well do the retrieval approaches cope with noisy and unstructured textual annotations? How well do the content-based retrieval approaches cope with images that cover diverse topics and are of varying quality? How well can systems exploit and combine di erent modalities given a user's multimedia information need? Can they outperform monomodal approaches like query-by-text, query-by-concept or query-by-image? In the context of INEX MM 2006-2007, mainly text-based retrieval approaches have been examined. This task is done to attract more visually-oriented approaches and most importantly, multimodal approaches that investigate the combination of evidence from di erent modalities.
In this paper we present our strategy to retrieve relevant documents using a concept-contentbased retrieval method in reusing the Visual Concept models we built for the CLEF VCDT 2008 task, without exclusively training the svms for this task. Then we test a very simple text analysis for a multimodal fusion. Results are showing that the visual process works badly, and does not contribute to the fusion. 2
In 2008, ImageCLEF wikipediaMM used the image collection created and employed by the INEX
Multimedia (MM) Track (2006-2007). This (INEX MM) wikipedia image collection contains
151,519 .jpeg and .png images that cover diverse topics of interest. Further information about the
image collection can be found in Westerveld and al. [
Images were provided by wikipedia users. Each image is associated with user-generated alphanumeric, unstructured metadata in English. These metadata usually contain a brief caption or description of the image, the Wikipedia user who uploaded the image. These descriptions are highly heterogeneous and of varying length. Figure 1 provides an example image and its associated metadata. 3
A possible source of information to help participants in the retrieval tasks was, For each image,
the classi cation scores for the 101 di erent MediaMill [
An important step in content-based image retrieval (CBIR) system is the extraction of discriminant visual feature that are fast to compute. Information theory and Cognitive sciences can provide some inspiration for developping such feature.
Among the many visual features that have been studied, the distribution of color pixels in an
image is the most common visual feature studied. The standard representation of color for
contentbased indexing in image databases is the color histogram. A di erent color representation is based
on the information theoretic concept of entropy. Such entropic feature can simply equal the entropy
of the pixel distribution of the image, as proposed in [
Let I be an image, or any rectangular subpart of an image.
For each normalized color (L = R + G + B, r = R=L; andg = G=L), we rst calculate two orthogonal pro ls by the projections of the pixels of I. We consider two simple orthogonal projection axes : the horizontal axis X (noted X ), versus the vertical one Y (noted Y ). The projection operator is either the arithmetic mean (noted 'Ar', then the projection is noted AXr), as illustrated in Figure 2, or the harmonic mean of the pixels on each column or each ligne of I (noted 'Ha', then we have HXa).
Then, we estimate the probability distribution function (pdf) of each pro l according to [7]. Considering that the sources are ergodic, we naly calculate each PEF equal to the normalized entropy (H(pdf )=log(#bins(pdf ))). We detail below each steps of the PEF extraction.
for each color of I of L(I) lignes and C(I) columns : oXp(I) = p^df ( oXp(I)), over nbinX (I) = round(pC(I)) bins, where oXp is the vertical projection with operator op, P EFX (I) = H( oXp(I))=log(nbinX (I)).
P EFYoYp((II)) == Hp^d(f (oYpoY(pI()I))=)l,ogo(vnerbinnbYi(nIY))(.I) = round(pL(I)) bins,
p^df (I) = pdf of all the pixels of I over nbinXY (I) = nbinX (I) nbinY (I) bins, P EF:(I) = H(p^df (I))=log(nbinXY (I)).
And we naly complete the PEF features by the usual mean and standard deviation of each normalized color of I.
We can calculate the PEF into three horizontal subimages (see Glotin Zhao VCDT CLEF papers for details). We note such PEF '='. We also calculate the PEF in three vertical subimages, we note these PEF 'kk'.
For each, we have 3 bands and 3 di erent PEF for each of the 3 colors, plus their mean and variance, thus we have 3 3 3 + 3 3 2 = 45 dimensions for '=' or for 'kk' features. We note '+' the feature concatenation of '=' and 'kk' features, which has then 90 dimensions. Considering the two mean type, the PEF concatenation without repetition of the mean and std color are quite compact with a total of 126 dimensions (= 2 (subimages type '=' or 'kk') * 3 (bands by subimages type) * 3 (rgL) * 4 (=4 types = (X or Y) * (Ar or Ha) ) + 1 (=H(I)) + 2 (= mean and std))). 5
Because of the very small length of the xml text data associated to the image, we selected almost 500 words de ning the requests. We count these word occurences in each xml le. The text score for each xml le and each query is then simple the sum of theses occurences for each selected words of the query. This lead to a very fast text classi cation : for the whole 150 000 xml les only few minutes are necessary on a pentium IV. Our goal was not to produce a very e cient text retrieval system, but to compare our visual concept approach to the most basic text retrieval. 70 60 50 40 300
signal RGB X profil 200 400 600 50 Y profil 100
150 R/L G/L L The support vector machine (SVM) [8] rst maps the data into a higher dimensional input space by some kernel functions, and then to learn a separating hyperspace to maximize the margin. Currently, because of its good generalization capability, this technique has been widely applied in many areas such as face detection, image retrieval, and so on. The SVM is typically based on an "-insensitive cost function, meaning that approximation errors smaller than will not increase the cost function value. This results in a quadratic convex optimization problem. So instead of using an "-insensitive cost function, a quadratic cost function can be used. The least squares support vector machines (LS-SVM) [9] are reformulations to the standard SVMs which lead to solving linear KKT systems instead. It is computationally attractive.
In our experiments, the RBF kernel
K(x1 x2) = exp( jx1 x2j2= 2) is selected as the kernel function of our LS-SVM. So there is a corresponding parameter, , to be tuned. A large value of 2 indicates a stronger smoothing. Moreover, there is another parameter, , needing tuning to nd the tradeo between to stress minimizing of the complexity of the model and to stress good tting of the training data points.
We trained 100 SVMs with di erent parameter values for each topic, and selected the best SVM using the validation set.
For Image or Text classi cation we do not make any Feedback nor Query Expansion techniques. So our runs are for NOFB type. In our experiments, we computed the average of the ranks of TXT and VISUAL. The process we adopt to implement the image retrieval in photo task is shown in Figure 3 and depicted as the following steps:
Step 1) According to the keywords of each topic, perform the text retrieval on the WIKI XML text database, and then get the rank result which is called Rank-Text.
Step 2) Split the VCDT labeled image dataset into 2 sets, namely training image dataset and The results for the wikipediaMM task have been computed with the trec eval tool (version 8.1). The submitted runs have been corrected (where necessary) so as to correpond to valid runs in the correct TREC format. The following corrections have been made: The runs comply with the TREC fomat as speci ed in the submission guidelines for the task When a topic contains an image example that is part of the wikipediaMM collection, this image is removed from the retrieval results, i.e., we are seeking relevant images that the users are not familiar with (as they are with the images they provided as examples). When an image is retrieved more than once for a given topic, only its highest ranking for that topic is kept and the rest are removed (and the ranks in the retrieval results are appropriately xed).
The interpolated recall precision averages are shown in Figure 4, and the summary statistics for the runs sorted by MAP are shown in Table 2, from which we see that TXT method did the best. 8
Our visual IMG NOFB run is the second best model in this campaign. So it can be concluded that our PEF VCDT visual concept partly t this task. We show that even a simple text analysis overcomes the best IMG NO FEEDBACK from Imperial College which has 0.0037 MAP, against 0.399 for our TXT NOFB text run. This emphases the fact that visual retrieval for Wiki task is very di cult. This can explain why our 3 basic fusions methods TXTIMG did not improve the TXT run. One can then conclude that the Feedback seems necessary for such task.
This work was partially supported by the French National Agency of Research (ANR-06-MDCA002).