The main goal of this paper it is to present our experiments in ImageCLEF 2010 Campaign (Wikipedia retrieval task). This edition we present a different way of using textual and visual information based on the assumption that the textual module better captures the meaning of a topic. So that, the TBIR module works firstly and acts as a filter, and the CBIR system reorder the textual result list. The CBIR system presents three different algorithms: the automatic, the query expansion and a logistic regression relevance feedback algorithm. We have submitted nine textual and eleven mixed runs. Our best run, at the 34th position (25% at the first result list), is a textual run using our own implemented algorithm based on a VSM approach and TF-IDF weights (included in the IDRA tool) and all languages for annotation and for the topics. Our best mixed run (51th position is at 60% first result list) is using the textual list and the logistic regression relevance algorithm at the CBIR module. Most of our runs are above the average of its own modality for the different measures. The new system architecture with the IDRA tool for the textual module and the logistic regression relevance algorithm for the visual module are the right track to maintain in our research lines.
Categories and subject descriptors
The UNED-UV is a research group formed by researchers from two different
universities in Spain, the Universidad Nacional de Educación a Distancia (UNED)
and the Valencia University (UV). This research group is working together [
The main goal of this paper it is to present our experiments in ImageCLEF 2010
Campaign (Wikipedia retrieval task) [
The TBIR subsystem includes the UNED own implemented tool IDRA (InDexing
and Retrieving Automatically) [
A more detailed presentation of the system, the submitted experiments, and the obtained results is included in the following sections.
The global system (shown at Fig. 1) includes three main subsystems: the TBIR, the
CBIR and the merging module. The TBIR subsystem uses the UNED own
implemented tool IDRA [
This module is in charge of the textual image retrieval using the metadata supplied for
the images in the collection. IDRA tool [
The textual retrieval task architecture can be seen in the Fig. 1. Each one of the components takes care of a specific task. These tasks will be sequentially executed: COLLECTION
(XML files) T O TITLEs P I Example C Images S COLLECTION (images)
Text Extraction
Queries Lang
Feature
Extraction
texts
PREPROCESS
queries Queries
File
Automatic
Query Expansion Relevance Feedback IDRA
Search M E R G E
MAX merge OWA Fusion
TXT Results
List TXTIMG Results
List
Text Extraction. Extracts the text from the files which contains the associated metadata. It uses the JDOM Java API to identify the content of each of the tags of the XML files.
Preprocess. This component process the text in two ways: 1) special characters deletion: characters with no statistical meaning, like punctuation marks, are eliminated; and 2) stopwords detection: exclusion of semantic empty words from specifics lists for each language. When processing multilingual text, a manually join of these lists is used.
Metadata Selection. With this component the system selects the text we want to index, depending on the chosen language “Index Lang” (EN, FR, DE or ALL). Therefore, 4 different indexations will be generated: one multilingual and three monolingual.
In the case of the monolingual indexations the selected text for the chosen language L= {EN, FR, DE}, from images metadata files, will be: 1) <name> whenever there is specific metadata for language L, or when there is not for any language; 2) <description> and <caption> whenever there is specific metadata for L; 3) <comment> when the text in this tag is not contained in <description> or <caption> and therefore will add new information. This time we did not use the text from the corresponding Wikipedia articles indicated in the <caption> attribute “article”.
When carrying out the multilingual indexation (ALL), the selected text will be the concatenation of the corresponding text for each of the three languages (EN+FR+DE), in the same way as explained for the monolingual cases.
Queries File. 4 different queries files are constructed for the experiments: one for each language (EN, FR, DE), and another for the multilingual case, which is indicated in “Queries Lang”. The strategy to select the text for each query is just to extract the information in the <TITLE> tag for the chosen language, and the concatenation of the three languages for the multilingual experiment.
IDRA Index. This component indexes the selected text associated with each image. The indexation is based on the VSM approach using TF-IDF (term frequency – inverse document frequency) weighted vectors. This approach consists in calculating the weights vectors for each one of the images selected texts. Each vector is compounded by the TF-IDF weights values of the different words in the collection. TF-IDF weight is a statistical measure used to evaluate how important a word is to a text in a concrete collection, and is calculated as shown in (1).
⎛ N ⎞ TF − IDF = ti, j * log 2 ⎜ ⎟ ⎝ ni ⎠ ti,j: number of occurrences of the word tj in caption text Ti.
N: total number of images captions in the collection. ni: number of captions in which appears the word ti. (1)
All weights values for each vector will be then normalized using the Euclidean distance. Therefore, for each one of the words appearing in the collection, the IDRA Index process updates and stores the following values: ni, ti,j, N (described in (1)), Ti: unique identifier of the image, idfj: inverse document frequency ( log2(N/ni) ) in Ti, Ei: Euclidean distance used to normalize, and wj,i: weight of word tj in Ti. IDRA Search. Is in charge of launching the queries against a concrete indexation for the experiment, and it obtains the corresponding “TXT Results List”. For each one of the queries, IDRA calculates its corresponding weights vector in the same way as in indexation. Then, the similarity between the query and an image text will depend on the proximity of their associated vectors calculated by the cosine measure: sim (Ti, q ) = cos( Ö ) =
∑ w j, i * w j, i ∑ w j, i *w j, i * ∑ w q , i * w q , i (2)
This similarity value will be calculated between the query and all the images metadata indexed. Images are ranked in descending order in the “TXT Results List”.
The VISION-Team at the Computer Science Department of the University of
Valencia has its own CBIR system, and that has also been used in previous
ImageCLEF editions (Photo-retrieval task in 2008 and 2009 [
As in most CBIR systems, a feature vector represents each image. The first step at the Visual Retrieval system is extracting these features for all the images on the database as for each of the cluster query topic images for each question. Instead of using the low-level features provided by the organization, we have used our own features. We use different low-level features describing color and texture to build a vector of features. The number of low-level features has been increased from the 114 components at ImageCLEF09 up to 296 components at the current edition. This increment is mainly due to the use of local HS histogram (10x3 bins) instead of local H histograms (10 bins) descriptors in previous editions.
• Color information: Color information has been extracted calculating both
local and global histograms of the images using 10x3 bins on the HS color
system. Local histograms have been calculated dividing the images in four
fragments of the same size. A bidimensional HS histogram with 10x3 bins is
computed for each patch. Therefore, a feature vector of 30 components for
the global histogram, and 192 components for the local histograms represent
the color information of the image.
• Texture information: Two types of texture features are computed: The
granulometric distribution function, using the coefficients that result of
fitting the distribution function with a B-spline basis. And, the Spatial Size
Distribution. We have used two different versions of it by using as the
structuring elements for the morphological operation that get size both a
horizontal and a vertical segment [
At this edition, the vision team has focus his work in testing three different visual
algorithms applied to the results retrieved by the text module: the automatic, the
relevance feedback and the query expansion. We assume that the conceptual meaning
of a question is better captured by the text module than by a visual module when they
work individually. Therefore, the task of the visual module is to re-order the textual
result list taking into account the information of the query images given at each topic.
Automatic algorithm. This is the typical algorithm in a CBIR system. The first step
is to calculate the feature vector that describes each image of the database as it has
been explained at the previous paragraph. The second step is to calculate the
similarity measurement between the feature vectors of each image on the database
and the N query images. The distance metric applied in our experiments is the
Mahalanobis distance that gives better results than the Euclidean one ([
As we have N query images, we will obtain N visual result lists, one for each query image in the topic. These N result lists are passed to the merging module to fuse them in one result list.
Query expansion algorithm. The query expansion algorithm works in the same way that the automatic algorithm, being the only difference that this algorithm expands the N query images to a wider set of images M. Thus, the M query images set is composed of the N images given by the topic and the N’ expanded images being M=N+N’. The N’ images set are the 3 first images of the textual result list. The M result lists are passed to the merging module.
Relevance feedback algorithm based on logistic regression. This algorithm works
differently to the two previous ones. Therefore, we will explain the concept of
relevance feedback and the adjustments made to get a good performance of the
algorithm for the proposed tasks [
We will explain the way the logistic regression relevance feedback algorithm works. Let us consider the (random) variable Y giving the user evaluation where Y=1 means that the image is positively evaluated and Y=0 means a negative evaluation. Each image in the database has been previously described by using low-level features in such a way that the j-th image has the k-dimensional feature vector xj associated. Our data will consist of (xj, yj), with j=1,…,n, where n is the total number of images, xj is the feature vector and yj the user evaluation (1=positive and 0=negative). The image feature vector x is known for any image and we intend to predict the associated value of Y. In this work, we have used a logistic regression where P(Y=1|x) i.e. the probability that Y=1 (the user evaluates the image positively) given the feature vector x, is related with the systematic part of the model (a linear combination of the feature vector) by means of the logit function. For a binary response variable Y and p explanatory variables X1,…,Xp, the model for π(x)=P(Y=1|x) at values x=(x1,…,xp) of predictors is logit[π(x)]=α+β1x1+…+βpxp, where logit[π(x)]=ln(π(x)/(1- π(x))). The model parameters are obtained by maximizing the likelihood function given by: n l(β ) = ∏π (xi ) yi [1 −π (xi )]1−yi (3)
The maximum likelihood estimators (MLE) of the parameter vector β are calculated by using an iterative method.
We have a major difficulty when having to adjust a global regression model in
which we take the whole set of variables into account, because the number of selected
images (the number of positive plus negative images) is typically smaller than the
number of characteristics. In this case, the regression model adjusted has as many
parameters as the number of data and many relevant variables could be not
considered. In order to solve this problem, our proposal is to adjust different smaller
regression models: each model considers only a subset of variables consisting of
semantically related characteristics of the image. Consequently, each sub-model will
associate a different relevance probability to a given image x, and we face the
question of how to combine them in order to rank the database according to the user’s
preferences. This problem has been solved by means of an ordered averaged weighted
operator (OWA) [
In our case, we have adapted the manual relevance feedback to an automatic performance. The examples and the counter-examples (positive and negative images) are automatically selected for each topic. The examples are the query images of the topic plus N images taken from the first positions of the textual result list. The counter-examples are the M latest positions of the textual result list. The relevance feedback algorithm is executed once. 2.4
Two merging algorithms are used in different steps with different purposes.
OWA Fusion. In the modality for textual and visual retrieval the approach that
follows this edition is based on the assumption that the conceptual meaning of a topic
is initially better captured by the text module itself than by the visual module. Thus,
the textual module works as a filter for the visual module, and the work of the visual
module is to re-order the textual results list. In this way, there has not been used an
explicit fusion algorithm to merge the textual result list and the visual result list.
However, the visual module generates N result visual lists depending on the number
of query images for the automatic and query expansion algorithms. These N lists are
merged in one result final list by using the Mathematical aggregation operators OWA
[
MAXmerge. This algorithm is used to fuse together different results lists in order to carry out some experiments related to multilingualism (UNED-UV8, UNED-UV9 described in next section). MAXmerge algorithm is included in IDRA tool and consists on, for each query, to select the results from the different lists which have a higher relevance/similarity value for the corresponding query, independently of the list the results appears in. 3
We have participated in two modalities: textual and mixed retrieval (visual and textual). Finally, 20 runs were submitted (9 textual, 11 mixed). A schematic description of these runs is shown in Table 1.
For textual modality, we present 9 runs. As it is explained in previous sections, 4 different indexations and 4 queries files were generated. From all possible combinations, we were interested in evaluate experiments with the 4 queries files against the multilingual indexation, obtaining 4 runs: UNED-UV1 (with multilingual queries), UNED-UV2 (with English queries), UNED-UV4 (with French queries), and UNED-UV6 (with Dutch queries). UNED-UV3, UNED-UV5 and UNED-UV7 correspond to monolingual experiments in which the language for the indexation is the same of the queries: English, French or Dutch, respectively. Finally, 2 more textual runs were submitted using the MAXmerge fusion algorithm: UNED-UV8 merging results lists from UNED-UV2, UNED-UV4 and UNED-UV6; and UNEDUV9 merging results from UNED-UV3, UNED-UV5 and UNED-UV7.
In the mixed modality, the textual module has passed through the visual module four different kinds of its basic textual algorithms corresponding to the UNED-UV1, UNED-UV2, UNED-UV3 and UNED-UV9 runs. The visual module has applied its three different algorithms (automatic, relevance feedback and query expansion) to the textual result lists in order to test the performance of these three algorithms over the different kind of text algorithms retrieval. For the [UNED-UV1] basic line the automatic, relevance feedback and query expansion algorithms have been applied getting the three corresponding runs [UNED-UV10], [UNED-UV11] and [UNEDUV12]. Following the same structure applying the three different visual algorithms to the [UNED-UV2] run the [UNED-UV13], [UNED-UV14] and [UNED-UV15] runs are obtained. From the [UNED-UV3] run the [UNED-UV16], [UNED-UV17] and [UNED-UV18]; and, from the [UNED-UV9] the [UNED-UV19], [UNED-UV20] and [UNED-UV21]. The last one was out of the maximum runs submitted. After the evaluation by the task organizers, our results for each of the submitted experiments are presented in Table 2. The table shows that our two best results are for the textual runs UNED-UV1 and UNED-UV9 (at the 34th and 40th position of the global result list, this is at the 25% first results). For the mixed modality, the best result is the UNED-UV11 at the 51th position (at the 60% first results). It is worth pointing out that the ranking position is computed by using the MAP measure (with a maximum MAP value of 0.1927 for our best run and a minimum MAP value of 0.1502 for our worst one). It can also be observed at Table 2 that most of our runs are above the average for each own modality (textual and mixed runs). These above results are marked in bold at the table.
With textual experiments this campaign we aimed to analyze multilingual issues. Comparing UNED-UV1 results with UNED-UV2, UNED-UV4 and UNED-UV6 ones, we can observe that best retrieval with the multilingual indexation is performed when we use the queries file constructed with the concatenation of all languages (MAP=0.1927). Launching English queries obtains better results (0.1627) that French (0.0920) or Dutch (0.09936), surely due to the metadata information for that language. Analyzing results for UNED-UV8 an UNED-UV9 runs, we observe that both of them obtain a good performance (only UNED-UV1 obtains higher MAP than them). Slightly higher results are for UNED-UV9 (0. 1865 > 0. 1790), so it is early to conclude when merging results from different languages, if it is better to launch the queries against monolingual indexations than against multilingual. At this moment, the effort in preprocessing has to be taken into account to decide.
The process to analyze our mixed modality results is by the comparison of the basic textual algorithms (UNED-UV1, UNED-UV2, UNED-UV3 and UNED-UV9) and their corresponding mixed runs (UNED-UV10-12 for the UNED-UV1, UNEDUV13-15 for UNED-UV2, and so on). We have improved the precision values at 10 and at 20 with the mixed runs, i.e. the UNED-UV11-Feedback (Prec@10 0.3914 and Prec@20 0.3629) improves its basic textual run UNED-UV1 (Prec@10 0.3914 and Prec@20 0.3564). The same improvement can be observed for the other mixed runs being compared with their corresponding textual runs. This result points out that visual algorithms can improve the textual result lists by back forward to the end of the list non relevant images retrieved by the textual module. However, MAP values are still lower than their corresponding textual runs. This could be due to the fact that more query images would be needed to get better results for higher precision measures (P@30, P@40 and so on), improving in that way the medium of the precision values (MAP).
Our best result is for the textual modality and it is at the position 34th, at the first 25% of the best results of the contest; and, our best result for the mixed modality is at the 51th position, at the first 60% of the global contest. Most of our runs in ImageCLEF10 are above the average for its own modality. These results mean that our main algorithms for textual and visual modules have got good marks, and they can be tuned to improve the current results.
Regarding multilinguality, the multilingual run it is our best (multilingual query launched to the multilingual index). It defeats the runs using monolingual queries (also on multilingual index). When using monolingual indexes and merging the results lists according to the query, only a slightly difference it is obtained (MAP value 0.1927 > 0. 1865 for UNED_UV9). It is early to conclude about, but the effort in preprocessing has to be taken into account to it.
The best result for mixed runs has been obtained with the logistic regression relevance feedback algorithm (UNED-UV11 at position 51), followed by the query expansion and the automatic one. Our new algorithms (logistic regression relevance feedback and query expansion) have markedly improved the results in comparison with the automatic algorithm used in previous editions. It is also important to notice that the best results of the contest are also achieved with a feedback algorithm. This reinforces our idea that the feedback algorithms are the right track to maintain in our future research lines.