This paper describes the participation of MIRACLE-GSI research consortium at the ImageCLEFphoto task of ImageCLEF 2008. For this campaign, the main purpose of our experiments was to evaluate different strategies for topic expansion in a pure textual retrieval context. Two approaches were used: methods based on linguistic information such as thesauri, and statistical methods that use term frequency. First a common baseline algorithm was used in all experiments to process the document collection: text extraction, tokenization, conversion to lowercase, filtering, stemming and finally, indexing and retrieval. Then this baseline algorithm is combined with different expansion techniques. For the semantic expansion, we used WordNet to expand topic terms with related terms. The statistical method consisted of expanding the topics using Agrawal's apriori algorithm. Relevance-feedback techniques were also used. Last, the result list is reranked using an implementation of k-Medoids clustering algorithm with the target number of clusters set to 20. 14 fully-automatic runs were finally submitted. In general, results are on the average, comparing to other groups.
results. This is because a search engine that retrieves a diverse, yet relevant set of images at the top of a ranked list is supposed to be more likely to satisfy its users.
Participants are provided with a set of topics, reused from the previous campaigns, which are run on their image
search system to produce a ranking that in the top 20, holds as many relevant images that are representative of
the different subtopics within the results. Evaluation is be based on two measures: precision at 20 and instance
recall at rank 20 (also called S-recall), which calculates the percentage of different clusters represented in the top
20. The reference database for this campaign is the same as last year, IAPR TC-12 Benchmark [
Based on our experience in previous campaigns, we designed a flexible system in order to be able to execute a large number of runs that exhaustively cover all the combinations of the different techniques. Our system is composed of a set of small components that are easily combined in different configurations and executed sequentially to build the final result set.
Specifically, our system is composed of five different modules: the textual (text-based) retrieval module, which indexes image annotations in order to search and find the most relevant ones to the text of the topic; the expander module, which expands documents and/or topics with additional related terms using textual and/or statistical methods; the relevance-feedback module, which allows to execute reformulated queries that include the results of previous queries; the result combination module, which uses OR operator to combine, if necessary, the results of the previous subsystems; and, finally, a clustering module that reranks the result list to allow cluster diversity. Figure 1 shows an overview of the system architecture.
Linguistic tools for textual analysis and retrieval. • •
Sparse matrix based tools for statistical topic expansion, clustering and relevance-feedback.
A common baseline algorithm was used in all experiments to process the document collection. This algorithm is based on the following sequence of steps:
Text Extraction: Ad-hoc scripts are run on the files that contain image annotations in XML format. Tokenization: This process extracts basic textual components. Some basic entities are also detected, such as numbers, initials, abbreviations, and years. So far, compounds, proper nouns, acronyms or other types of entity are not specifically considered. The outcomes of this process are only single words, years in numbers and tagged entities. 3.
Conversion to lowercase: All document terms are normalized by changing all letters to lowercase.
4. Filtering: All words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from the University of Neuchatel’s resources page [
Porter stemmers [
This common baseline algorithm is complemented and combined with different expansion techniques in order to
compare the improvement given by semantic- versus statistical-based techniques. For the semantic expansion,
we used WordNet [
The statistical method consisted of expanding the topics using the Agrawal’s apriori algorithm [
Experiments are defined by the choice of different combinations of the previous modules with the different topic expansion techniques and including relevance-feedback or not.
EN/RND EN/RND EN/RND EN/RND EN/RND EN/RND
EN/RND
Method stem + stopwords baseline + k-Medoids clustering baseline + Apriori topic expansion + k-Medoids clustering baseline + WordNet topic expansion + k-Medoids clustering baseline + Relevance-Feedback (N=10, M=5) + k-Medoids clustering baseline + Apriori topic expansion + Relevance-Feedback (N=10, M=5) + k-Medoids clustering baseline + WordNet topic expansion + Relevance-Feedback (N=10,
M=5) + k-Medoids clustering Results are presented in the following tables. Each of them shows the run identifier, the number of relevant documents retrieved, the mean average precision (MAP), precision at 10, 20 and 30 first results, and cluster precision at 10, 20 and 30 first results. For English, the best result in terms of MAP is achieved by the baseline experiment. However, the best cluster precision (CR), which was the variable to maximize in this task, is achieved when k-Medoids algorithm is applied, thus proving to be valuable. The significant improvement in cluster precision is over 6% at CR10 and 21% at CR30. Again, as in the case of English, the best results in terms of cluster relevance are obtained in runs that include k-Medoids clustering. MAP value for English is significantly better than for the Random (mixed) language, probably due to the noisy nature of the multi-language annotation.
In general, with respect to MAP, the highest value is obtained with the baseline experiment; MAP values are similar in practice for experiments using topic expansion (Tag and AP) and significantly worse (0.08 against 0.18) in the case of relevance-feedback (RF). This shows that no strategy for topic expansion nor specially relevance-feedback has proved to be useful.
A preliminary analysis of the results, given the low precision values obtained in the experiments that make use of the relevance-feedback methods, shows that the reranking algorithm used for combining the different result lists is likely to be the main reason for the disappointing results. However, this impression has to be confirmed with a more in-depth analysis. However, even though all expansion processes produce a decrease in the appropriateness of the results, their recall, as shown in the number of relevant document retrieved) improves in a significant manner.
Another probable cause is the choice of the OR operator to combine the terms in the topic to build up the query. Due to time constraints to prepare this report, we were unable to repeat our experiments with the AND operator, but we think that MAP values should be significantly higher using this operator.
The last conclusion that can be drawn is that the application of clustering techniques smoothes the negative effect of the expansion processes, showing quite promising results. This work has been partially supported by the Spanish R+D National Plan, by means of the project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007-67407-C03-03 and by Madrid R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267.