This paper describes the participation of the MIRACLE consortium at the ImageCLEF Photographic Retrieval task of ImageCLEF 2008. In this is new participation of the group, our first purpose is to evaluate our own tools for text-based retrieval and for content-based retrieval using different similarity metrics and the aggregation OWA operator to fuse the three topic images. From the MIRACLE last year experience, we implemented a new merging module combining the text-based and the content-based information in three different ways: FILTER-N, ENRICH and TEXT-FILTER. The former approaches try to improve the text-based baseline results using the content-based results lists. The last one was used to select the relevant images to the content-based module. No clustering strategies were analyzed. Finally, 41 runs were submitted: 1 for the text-based baseline, 10 content-based runs, and 30 mixed experiments merging text and content-based results. Results in general can be considered nearly acceptable comparing with the best results of other groups. Obtained results from textbased retrieval are better than content-based. Merging both textual and visual retrieval we improve the text-based baseline when applying the ENRICH merging algorithm although visual results are lower than textual ones. From these results we were going to try to improve merged results by clustering methods applied to this image collection.
MIRACLE is a research consortium formed by research groups of three different universities in Madrid, Universidad Politécnica (UPM), Universidad Autónoma and Universidad Carlos III, along with DAEDALUS, a small/medium size enterprise (SME) founded in 1998 as a spin-off of UPM.
This paper describes our participation (Mir-FI, stands for Miracle subgroup at Facultad de Informática) at the ImageCLEF Photographic Retrieval task of ImageCLEF 2008. The goal of this task was fully described last year in [6]. The reference database is the IAPR TC-12 Benchmark [7, 8].
This year our experiments were due to evaluate our own tools for text-based and content-based retrieval. The text-based technique is based in the classical Vector Space Model (VSM) with TF-IDF weights and the tool for image-based retrieval includes different image color and texture descriptors [9, 10]. In addition, we have applied some merging algorithms to fuse together both textual and visual results in order to evaluate if this improve our baseline. All the 41 experiments and results are explained in the following sections.
We have a tool implementing different techniques for image-based retrieval, based on several components that allow different configurations in order to easily execute sequentially text-based, content-based and the merge of the results. Fig. 1 presents an overview of the system architecture.
annotations
topics
Extractor
Module
Our main goal was to evaluate both textual and visual retrieval baselines and the experimentation with different combinations of them. Thus, the system is built up from three main different components: Text-based retrieval module, Image-content based retrieval module and the Merging module that is in charge of combine the results lists from textual and visual retrieval using different approaches. A more detailed explanation is included in section 2.1 and 2.2. 2.1
MIRACLE-FI textual retrieval is based on the VSM approach using weighted vectors based on the TF-IDF weight. Applying this approach, a representing vector will be calculated for each one of the image annotations provided by the IAPR TC-12. The textual retrieval task architecture can be seen in the Figure 1. Each one of the components takes care of a specific task. These tasks will be sequentially executed:
Annotations/Topics Tags Selection. With these components, it is possible to select the desired XML tags of the annotations/topics files, which will compound the associated text describing each image/query. In the annotations files there are eight different tags (DOCNO, TITLE, DESCRIPTION, NOTES, LOCATION, DATE, IMAGE and THUMBNAIL) and in the topics ones there are seven (NUM, TITLE, CLUSTER, NARR, and 3 IMGAGE). In all our experiments, the selected tags from the annotations files had been four: TITLE, DESCRIPTION, NOTES and LOCATION. In the case of the topics, the selected tags were two: TITLE and NARR.
It takes less than 20 minutes to extract the text from the provided annotations files, to delete the special characters, and to exclude stopwords. To build and save the vector space with all the weights vectors corresponding to each annotation file, it takes almost 7 hours of processing in this first version. 2.2
This campaign MIRACLE team joined the VISION-Team at the Computer Science Department of the University of Valencia who has its own CBIR system mainly used for relevance feedback algorithms evaluation [9,10]. The low-level features of the original CBIR system have been adapted to be used at the ImageCLEFphoto for image retrieval and for merging image and text information retrieval.
We use different low-level features describing color and texture to build a vector of features with 68 components:
Color information: a feature vector of 30 components represents the color information. Each of these components represents a bin on a HS (hue-saturation) histogram of size 10 x 3. For this database the last 10 histogram (the highest saturated) where eliminated so that their values where almost zero. Therefore, a feature vector of 20 components has been used for extracting color information.
Texture information: six feature textures have been computed for this repository respectively. The first three ones use code from the implementation done by Smith and Burn in Meastex [13]; the rest have been implemented by the authors. The total of texture features builds a vector of 48 components. o Gabor Convolution Energies [5]. o Gray Level Coocurrence Matrix also known as Spatial Gray Level Dependence [4]. o Gaussian Random Markov Fields [2]. o The granulometric distribution function, first proposed by Dougherty [3]. We have used here not the raw distribution but the coefficients that result of fitting its plot with a B-spline basis. o Finally, the Spatial Size Distribution [1]. We have used two different versions of it by using as the structuring elements for the morphological operation that get size both a horizontal and a vertical segment.
The second step is to calculate the similarity distance between the feature vectors from each image on the database to the three topic images. We have used two distance metrics on the experiments: the Euclidean and the Mahalanobis distance. Therefore, three similarity distances from each image on the repository to the three query images are calculated so that only a content-based image list is needed.
Mathematical aggregation operators transform a finite number of inputs into a single output and play an important role in image retrieval. We decided to use the so-called OWA operators to aggregate the three lowlevel feature vectors of the topic images. These operators were introduced in [16].
With the OWA operator no weight is associated with any particular input; instead, the relative magnitude of the input decides which weight corresponds to each input. In our application, the inputs are similarity distances to each of the three topic images and this property is very interesting because we do not know, a priori, which image of the three will provide us with the best information.
The goal of the content-based image system is to evaluate the three different aspects used in content-based image retrieval system: the low-level features, the OWA aggregation methods, and the different distance metrics to measure the similarity. About the time of execution, the most demanding task is feature extraction that is done just once and then the values are stored on the database. Therefore, it takes less than 5 minutes the calculation of the content-based list for all the questions. 2.3 Textual and image results lists will be merged in two different ways, using the textual results lists (T) as principal list and the image ones (I) as a support list.
FILTER-N. This way of merging the image and textual results lists consists on checking which results in the T list are also included in the N first results of the I list. The value of N indicates the number of results taken into account from the I list when narrowing down the T list. The resulting merged list will have a maximum of 1000 results for each query to follow the ImageCLEFphoto indications.
This merging strategy tries to eliminate from the main list those results that are not considered sufficiently relevant according to the support list. We consider that a result is important in the support list if it is ranked in the N firsts positions. The value of N can be modified to demand a higher degree of relevancy in the support list. ENRICH. This kind of merging also uses two results lists, the main list and the support list. If a concrete result appears in both lists for the same query, the relevance of this result in the merged list will be increased in the following way: new Re l = main Re l +
sup Re l ( pos Re l + 1) where newRel: relevance value in the merged list supRel: relevance value in the support list mainRel: relevance value in the main list posRel: position in the support list Relevance values will be then normalized from 0 to 1.
Every results appearing in the support list but not in the main one (for each query), will be added at the end of the results for each query. In this case, relevance values will be normalized according with the lower value in this moment. In the submitted experiments this addition of the results from the support list not appearing in the main list seems not working correctly. Algorithm has already been modified to add these results in the proper way. The merged lists resulting will be limited to the same number of results per query (1000), to follow the task indications.
TEXT-FILTER. In this kind of experiments the text-based module is applied to the complete database and those images that have a relevance value above zero are passed to the content-based image module. In this experiment, the content-based image module only works with the images filter by the text module. Then, the content-based image module calculates the similarity of each feature vector of the text-filter images to each of the query images. Moreover, this three relevance values are merged with the different OWA aggregation operators as mentioned in section 2.2.
Finally it was sent one text-based run, 10 content-based runs and 30 mixed runs using a combination of both. The name of the runs identifiers indicate the different configurations applied. All the names of the runs begin with EN-EN-AUTO because the used language is English and all of them are fully automatic, avoiding any manual intervention.
The text-based run identifier, MirFIbaseline is based on the vector space model using the TF-IDF weight. There are 10 content-based experiments, built combining different distances for calculating similarity from each feature vector to the topic, and different aggregation OWA operators for combining the three topic feature vectors for each topic image. The two similarity distances are Euclidean and Mahalanobis, and five aggregation OWA operators for combining the three topic images are used (max, min, med, o3, o7). The name of the 10 content-based runs indicates which distance and aggregation operator has been used in each case. The name will be MirFIdistmerge where dist = {euc, maha} and merge = {max, med, min, o3, o7}.
The combination of the results obtained from both the textual and visual retrieval will form a set of 30 mixed runs. FILTER-10000 and ENRICH have been used to generate the first 20 runs. The last 10 runs have been obtained by the TEXT-FILTER method.
Run Identifier The following table shows all the submitted runs identifiers built for this edition of ImageCLEFphoto. After the evaluation by the task organizers, obtained results for the different experiments are presented in the following tables. Each table shows the run identifier, the mean average precision (MAP), the precision at 5, 10, 20 and 30 first results, and the number of relevant images retrieved (out of 2401 relevant images). Obtained results with the textual-based retrieval module can be considered acceptable, having into account that no linguistic processes were applied. The MAP (0.2253) is higher than the average MAP taken from the best 4 runs for each participating group (0.2187).
For the content-based image module was testing we can observe that the Mahalanobis distance outperforms the Euclidean distance, and the best aggregation method in both metrics is the minimum (AND), followed by the orness(W)_0.3 that is a smoothed AND. Our best result for this group of experiments is the combination of the Mahalanobis metrics with orness(W)_0.3 with a MAP(0.0213) and a P20(0.0679). Our best result is considerably lower than the best result for this group. The FILTER-10000 merge algorithm improves the baseline in the precision at low values (5, 10) but never improves the MAP value nor the number of relevant images retrieved. ENRICH merge method improves the baseline experiment in the MAP value and in the number of relevant images retrieved. Best MAP value (0.2401) is achieved merging the textual results with the visuals obtained using the Mahalanobis distance and the AND operator. This value is quite bigger than the average MAP taken from the best 4 runs from each participating group (0.2187). Applying this merge strategy, obtained results outperform the content-based ones in terms of both precision and MAP. Again, the best results correspond to the experiments which use the Mahalanobis distance and the AND operator.
In this participation in the task, results in general can be considered by us acceptable comparing with the best results of all the groups.
The MAP value obtained for the text-based baseline experiments was 0.2253, higher than the average MAP (0.2187) calculated from the best 4 runs from each participating group.
For the content-based image retrieval, the results have not been very successful. Our results are lower than the best top ten. However, our challenge this year was to test their different parameters such as the distance metrics and the aggregation methods. The most interesting conclusion in that the Mahalanobis distance works better than the Euclidean one, and the best aggregation method is the AND operator. For following editions more low-level features based on local color descriptors and shape descriptors will be included.
Merged results show that the ENRICH algorithm improves very lightly the baseline. This is important taken into account the poor results obtained from the visual retrieval. So if we achieve to improve these content-based results, may be better merged results using this algorithm will be obtained. FILTER-10000 algorithm improves the textual baseline results in terms of precision at low values.
This work has been partially supported by the Spanish R+D National Plan, by means of the project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007-67407-C03-03; by the Madrid R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267 (2006-09).