In this paper, we present the LIP6 retrieval system which automatically ranks the most similar images to a given query constituted of both textual and/or visual information through a given textual-visual collection. The system first preprocesses the data set in order to remove stop-words as well as non-informative terms. For each given query, it then finds a ranked list of its most similar images using only their textual informations. Visual features are then used to obtain a second ranking list from a manifold and a linear combination of these two ranking lists gives the final ranking of images.
ImageCLEF’s WikipediaMM uses image collection created and employed by the INEX competition
[
In section 2, we propose a strategy to perform the retrieval process. Section 3 describes the models we used in our experiments and in section 4 we show the results we obtained. 2
We have a collection of textual-visual documents constituted of images and their corresponding textual notes. The aim is hence to find a list of ranked documents for each query in a such way that the most relevant documents have the highest ranks. A query can be in a textual or both textual and visual formats. In order to benefit from the whole given data, we introduce a retrieval strategy which combines two lists of ranked documents obtained separetely from textual and image informations for each given query. If a query is just described by a text, the system provides the ranked list obtained from the first textual strategy. In the following section we detail these different retreival strategies. 3
The textual strategy is based on the Term Frequency-Inverse Document Frequency Model [
In the TF-IDF model [
1 + |D| 1 + | {d | |wi ∈ d} | and |D| representes the total number of documents in the collection. 3.2
In the Language Model [
P (D|Q) = p(Q|D)P (D).
P (D|Q) ∝ Y P (qi|D)
qi∈Q This model is also known as the unigram model. By dissociating query terms occuring or not in D we get:
P (D|Q) ∝
Y Ps(qi|D) × Y Pu(qi|D) qi∈D qi∈/D (1) Ps is the probability model of occuring query terms in D and Pu is the probability model nooccuring terms. Pu(qi|D) could be estimated by αd × P (qi|C) in which αd is a constant factor and C is the collection of documents. Documents are then ranked according to the logarithm value of equation (1): log P (Q|D) = qi∈D X log Ps(qi|D)
N (qi, d) + |qi| × log αd + X log P (qi|C).
qi∈/D
Where N (qi, d) is a normalization factor. For the estimation of Ps(qi|D), we used the
JelinekMercer smoothing approach[
Ps(qi|D) = (1 − λ)Pml(qi|D) + λP (qi|C) , λ ∈ [
Pml(qi|D) =
|qi ∈ D| Pj=1...N |qj ∈ D| . 3.3
The goal of using Data Manifolds [
As the first step of preprocessing, all unuseful characters should be removed, like non-alphabetic ones. As we are sure that in Wikipedia task there exist only the English characters, we delete also the non-English characters. In next step of preprocessing, each word in the corpus existing in the anti-dictionary will be deleted. As third step, we try to retrieve the keywords and verbs’ roots hidden in the combined terms. In this task, all the words appeared in the topics will create our keywords set. The idea is based on searching substrings in each word or combined term to find the hidden keywords through them. As we are sure that in Wikipedia task there exist only the English characters, we delete also the non-English characters.
In next step of preprocessing, each word in the corpus existing in the anti-dictionary will be deleted. As third step, we try to retrieve the keywords and verbs’ roots hidden in the combined terms. In this task, all the words appeared in the topics will create our keywords set. Our strategy based on searching substrings in each word or combined term to find the hidden keywords through them.
After this step, we remove the words with an appearance frequency less than 10 times in the corpus unless the word is a keyword. The reason why we do this step is to reduce the size of the corpus to make it easier to process the documents in the models. We lost a part of information, but it is not so important because we try to delete the least frequent words which have usually the least importance.
Afterward, we remove all documents containing less than 5 words for the reason that they have not so much information to be important in models’ process. Here also, we prevent the documents containing any keywords being removed. During the preprocessing, we create the dictionary (vocabularies list) and we also index each document respectively to the dictionary to facilitate future calculation. Dictionary is created simply as a list of two columns: first column contains the words existing in the obtained corpus from the latter step. In second column, each line indicates the times of appearance of the word located in the corresponding line in first column. 4.2
All runs are fully automatic and do not use feedback nor query expansion. The visual features are composed of a simple global HSV histogram (8x3x3) and of the standard deviations of H, S and V. The visual vectors are so composed of 17 dimensions.
Run1 TFIDF A textual approach using TF-IDF. We used exclusively the text information here.
Each topic text (query) and document were characterized in the bag-of-word space using a TF-IDF encoding. By performing a cosines-similarity between the given query and documents, we had a value for each document represents the relevance. Descending sort on documents upon these obtained values made the most similar documents be ranked higher.
As the final result, we chose the 2000 first documents in the ranking list.
Run2 LM A textual approach using Language-Model. We used exclusively the text information again but this time we perform another textual model. For each topic we estimated the probability that every document in the collection generates the query using a unigram language model as explained in models section. At the end, the probability calculated for each document was counted as the relevance value. By sorting these values we obtained the ranked list of documents and we got the N first documents as the final result. Run3 TFUSION TFIDF LM A textual approach using a fusion of the results of LanguageModel and TF-IDF. For each query q, we obtained two lists of ranked documents according to the TF-IDF and the Language Model approaches as calculated in Run1 and Run2. The final ranked list for q is based on the fusion of the rank values upon these two lists. As the fusion in this part, we got 50% of each ranked list to combine.
Run4 visualonly A visual approach using Image only. We used exclusively the image information here. Each topic image (query) and document were characterized using global HSV image features. For each query, its most similar documents in the sense of the euclidean measure were chosen. We did not provide any results for those topics without an associated image query.
Run5 TIFUSION LMTF COS A texto-visual approach using Fusion Text-image (1). For each topic, we ranked documents upon two approaches considering only the text information as done in Run3. We got the 2000 first documents and try to re-rank them upon their similarity using image features, i.e the final rank of documents was obtained by re-sorting the 2000 first documents upon their similarity to the image(s) of the given topic. We used the Euclidean distances as the similarity measure.
Run6 TI LMTF MANIF A texto-visual approach using Fusion Text-image (2). For each topic, we ranked documents upon two approaches considering only the text information as done in Run3. We got the 2000 first ranked document and try to re-rank these 2000 document upon their similarity using image features. but here, we used manifold-based technique in place of the simple Euclidean distance. Briefly, we create the network of data manifold on base of these N documents, through them we search the most relevance to query and the values of data manifolds model was the final rank of documents.
Run7 TIFUSION LMTF MANIF A texto-visual approach using Fusion Text-image (3). In this part, we used two ranking lists obtained from Run5 and Run6 and the final ranking list was obtained from the fusion of rank values of these two lists. In the fusion in this part, we took 60% as the weight factor for the ranked list of Run5 and 40% as the weight factor for Run6. 1 TFIDF 2 LM 3 TFUSION TFIDF LM 4 visualonly 5 TIFUSION LMTF COS 6 TI LMTF MANIF 7 TIFUSION LMTF MANIF
TXT TXT TXT
IMG TXTIMG TXTIMG TXTIMG In this working note, we focus our efforts on the study of how to find automatically the most similar images through a given textual-visual collection to a given query which is a collection of textual and/or visual information. First, we perform a preprocessing on the textual part, then for each given query, we use the textual information retrieval models which give us ranking lists. We also use the visual features to obtain another ranking list. A linear combination of these ranking lists give us the new ranking lists which benefit both of visual and textual part of information. As a new idea, we also perform the method of ranking on data manifold in place of using euclidean distance to find the similarity.
As for several participants of WikipediaMM 2008, our text runs obtain better MAP than
textovisual runs, but our texto-visual runs give better P10 scores, then using a combination of textual
and visual information seems, in our case, improving the first results. That shows the difficulty
to obtain a good texto-visual fusion. We use the same TF-IDF model and the same Language
Model in ImageCLEFphoto 2008 as in WikipediaMM2008. Contrary to WikipediaMM task, in
ImageCLEFphoto 2008, these two models give similar scores when the query is composed of the
title words (see [
As perspectives, we will study how the feature extraction manner influence the results obtain
by the data manifold model. We will also consider the different parameters of language model and
how the preprocessing phase could be improved in our future works. We also expect to investigate
the effect of learning with both labeled and unlabeled data in the search phase [