This paper describes the participation of MIRACLE-GSI research consortium at the ImageCLEF 2009 Photo Retrieval Task. For this campaign, the main purpose of our experiments was to compare the performance of a “standard” clustering algorithm, based on the k-Medoids algorithm, against a more simple classification technique that makes use of the cluster assignment that was provided for a subset of topics by the task organizers. First a common baseline algorithm was used in all experiments to process the document collection: text extraction, tokenization, conversion to lowercase, filtering, stemming and finally, indexing and retrieval. Then this baseline algorithm is combined with these two different result reranking techniques. As expected, results show that any reranking method outperforms a standard non-clustering image search baseline algorithm in terms of cluster recall. In addition, using the information of cluster assignments leads to the best results.
MIRACLE team decided to split into two subgroups, MIRACLE-GSI (Grupo de Sistemas Inteligentes – Intelligent System Group) in charge of purely textual runs and MIRACLE-FI (Facultad de Informática, Computer Science Faculty) in charge of visual and mixed runs. This paper reviews the participation of MIRACLE-GSI at ImageCLEFphoto 2009. The participation of the other subgroup is described in an accompanying paper.
Our idea for this campaign was to continue the open line of research [
Based on our experience in previous campaigns, we designed a flexible system in order to be able to execute a large number of runs that exhaustively many combinations of different techniques. Our system is composed of a set of small components that are easily combined in different configurations and executed sequentially to build the final result set. Specifically, our system is composed of five modules: • • • • •
Linguistic processing module, which extract, parses and prepares the input text for subsequent modules, Expander module, which expands documents and/or topics with additional related terms using textual and/or statistical methods, Textual (text-based) retrieval module, which indexes image annotations in order to search and find the list of images that are most relevant to the text of the topic, Result combination module, which uses OR/AND operators to combine, if necessary, two different result lists,
Clustering module, which reranks the result list to allow cluster diversity. 1.
Text Extraction: Ad-hoc scripts are run on the files that contain image annotations in XML format. Tokenization: This process extracts basic textual components. Some basic entities are also detected, such as numbers, initials, abbreviations, and years. So far, compounds, proper nouns, acronyms or other types of entity are not specifically considered. The outcomes of this process are only single words, years in numbers and tagged entities.
Conversion to lowercase: All document terms are normalized by changing all letters to lowercase.
Filtering: All words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from the University of Neuchatel’s resources page [
Stemming: This process is applied to each one of the words to be indexed or used for retrieval. Standard
Porter stemmers [
Indexing and retrieval: Lucene [
The topic set was divided into two subgroups. The first 25 topics include a cluster assignment provided by the task organizers, i.e., some clues were given to guide the clustering process. For those topics, classification techniques can be used to produce the final result list. The rest of the topics did not include any cluster assignment, so “standard” clustering techniques had to be used to produce the final result list. On the one hand, the classification technique finds, for each topic, the list of images that are relevant to each given cluster, and, in addition, the list of images that are relevant to the topic but do not match any of the given clusters (“Others” cluster). For that purpose, the algorithm first builds as many subtopics as different clusters have been provided for a given topic. These subtopics contain the original topic terms combined with the terms of the cluster titles. For instance, if topic A has 2 clusters associated (A1, A2), the set of subtopics would be: {termsA AND termsA1} and {termsA AND termsAB} Second, the algorithm builds another subtopic that includes the topic terms but excludes the terms of all clusters. For the previous example, the subtopic for “Others” cluster would be:
{termsA AND NOT termsA1 AND NOT termsA2} Then those subtopics are given to the Lucene information retrieval engine to get the relevant list of images. Last, each image is assigned to the cluster that corresponds to the subtopic with which the image has the highest similarity.
On the other hand, the clustering technique is based on an implementation of k-Medoids clustering algorithm [
Results are presented in the following tables, showing the run identifier, the number of relevant documents retrieved, the mean average precision (MAP), precision at 10, 20 and 30 first results, and cluster precision at 10, 20 and 30 first results. Table 2 shows the results for the first 25 topics and Table 3 shows the results for the remaining topics. The following figures show the precision and cluster recall values for each run and allow comparing results achieved by the classification technique (MIRGSI_TCT_TXT) with respect to the clustering technique (MIRGSI2_T_TXT) in each subset of topics. Data series identified with “part 1” refer to topics 1-25 whereas data series identified with “part 2” refer to topics 26-50. The baseline experiment achieves the best result in terms of MAP. However, the best cluster recall (CR), which was the variable to maximize in this task, is achieved when other techniques are used, thus proving to be valuable. As it could be expected, the run that makes use of the manually assigned clusters (MIRGSI_TCT_TXT) achieves the best results in terms of cluster recall, and clearly outperforms the baseline experiment (0.782 vs 0.600 at CR30, 130%). Morever, the k-Medoid clustering is slightly better than the baseline experiment in cluster recall at any value.
After this preliminary analysis, the conclusion that can be drawn is that the application of clustering techniques improves the information retrieval process and shows quite promising results.
This work has been partially supported by the Spanish R+D National Plan, by means of the project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007-67407-C03-03 and by Madrid R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267.