SINAI at ImageCLEF 2005 M.T. Mart 0 1 2 n-Valdivia 0 1 2 M.A. Garc 0 1 2 a-Cumbreras 0 1 2 M.C. D 0 1 2 az-Galiano 0 1 2 L.A. Uren~a-L 0 1 2 A. Montejo-Raez 0 1 2 General Terms 0 1 2 Images, Indexing, Machine Translators Algorithms , Experimentation, Languages, Performance Campus Las Lagunillas , Ed. A3, e-23071, Ja University of Ja

In this paper, we describe our ¯rst participation in the ImageCLEF campaign. The SINAI research group participated in both the ad hoc task and the medical task. For the ¯rst task, we have used several translation schemas as well as experiments with and without pseudo relevance feedback (PRF). For the medical task, we have also submitted runs with and without PRF, and experiments using only textual query and using textual mixing with visual query.

 H 3 [Information Storage and Retrieval] H 3 1 Content Analysis and Indexing H 3 3 Information Search and Retrieval H 3 4 Systems and Software Introduction

This is the ¯rst participation for the SINAI research group at the ImageCLEF task. We have accomplished the ad hoc task and the medical task.

As a cross language retrieval task, a multilingual image retrieval based on query translation can achieve high performance, more than a monolingual retrieval. The ad hoc task involves to retrieve relevant images using the text associated to each image query.

The goal of the medical task is to retrieve relevant images based on an image query [ 1 ]. This year, a short text is associated to each image query. We ¯rst compare the results obtained using only textual query versus results obtained combining textual and visual information. We have accomplished several experiments with and without PRF. Finally, we have used di®erent methods to merge visual and text results.

Next section describes the ad hoc experiments. In Section 3, we explain the experiments for the medical task. Finally, conclusions and further works are presented in Section 4.

Experiment SinaiDuTitleFBSystran SinaiDuTitleNoFBSystran The goal of the ad hoc task is, given a multilingual query, to ¯nd as many relevant images as possible, from an image collection.

The proposal of the ad hoc task is to compare results with and without PRF, with or without query expansion, using di®erent methods of query translation or using di®erent retrieval models and weighting functions. 2.1

Experiment Description

In our experiments we have used nine languages: English, Dutch, Italian, Spanish, French, German, Danish, Swedish and Russian

The dataset is the same used in 2004: St Andrews. The St Andrews dataset consists of 28,133 photographs from St Andrews University Library photographic collection which holds one of the largest and most important collections of historic photography in Scotland. The collection numbers in excess of 300,000 images, 10% of which have been digitized and used for the ImageCLEF ad hoc retrieval task. All images have an accompanying textual description consisting of 8 distinct ¯elds. These ¯elds can be used individually or collectively to facilitate image retrieval.

The collections have been preprocessed, using stopwords and the Porters stemmer.

The collection dataset has been indexed using LEMUR IR system. It is a toolkit that supports indexing of large-scale text databases, the construction of simple language models for documents, queries, or subcollections, and the implementation of retrieval systems based on language models as well as a variety of other retrieval models. The toolkit is being developed as part of the Lemur Project, a collaboration between the Computer Science Department at the University of Massachusetts and the School of Computer Science at Carnegie Mellon University.

We have used online Machine Translator for each language pair English-other. After a complete research the best translators are ² Systran for Dutch, French, German, Italian, Russian and Swedish ² Prompt for Spanish (European) and Spanish (Latinoamerican)

One parameter for each experiment is the weighting function, such as Okapi or TFIDF. Another is the use or not of PRF. 2.2

 Results and Discussion The Medical Task

The main goal of medical ImageCLEF task is to improve the retrieval of medical images from heterogeneous and multilingual document collections containing images as well as text. This year, Experiment SinaiEnTitleNarrFB SinaiEnTitleNoFB SinaiEnTitleFB SinaiEnTitleNarrNoFB

Initial Query

title + narr title title title + narr

 Expansion

with without with without

 Experiment Initial Query

SinaiGerTitleFBSystran title SinaiGerTitleFBPrompt title SinaiGerTitleNoFBPrompt title SinaiGerTitleNarrFBSystran title + narr SinaiGerTitleNarrFBPrompt title + narr SinaiGerTitleNoFBSystran title SinaiGerTitleFBWordlingo title SinaiGerTitleNarrNoFBSystran title + narr SinaiGerTitleNarrNoFBPrompt title + narr SinaiGerTitleNoFBWordlingo title SinaiGerTitleNarrFBWordlingo title + narr SinaiGerTitleNarrNoFBSWordlingo title + narr Experiment Initial Query SinaiSpEurTitleFBPrompt title SinaiSpEurTitleFBEpals title SinaiSpEurTitleNoFBPrompt title SinaiSpEurTitleNarrFBEpals title + narr SinaiSpEurTitleNoFBEpals title SinaiSpEurTitleNarrFBSystran title + narr SinaiSpEurTitleNoFBSystran title SinaiSpEurTitleNoFBWordlingo title SinaiSpEurTitleFBSystran title SinaiSpEurTitleFBWordlingo title SinaiSpEurTitleNarrNoFBEpals title + narr SinaiSpEurTitleNarrNoFBPrompt title + narr SinaiSpEurTitleNarrNoFBSystran title + narr SinaiSpEurTitleNarrFBSystran title + narr SinaiSpEurTitleNarrNoFBSWordlingo title + narr SinaiSpEurTitleNarrFBWordlingo title + narr Experiment Initial Query SinaiSpLatTitleFBPrompt title SinaiSpLatTitleNoFBPrompt title SinaiSpLatTitleNoFBEpals title SinaiSpLatTitleNoFBSystran title SinaiSpLatTitleNoFBWordlingo title SinaiSpLatTitleFBSystran title SinaiSpLatTitleFBWordlingo title SinaiSpLatTitleFBEpals title SinaiSpLatTitleNarrNoFBSystran title + narr SinaiSpLatTitleNarrFBPrompt title + narr SinaiSpLatTitleNarrFBSystran title + narr SinaiSpLatTitleNarrFBEpals title + narr SinaiSpLatTitleNarrNoFBPrompt title + narr SinaiSpLatTitleNarrNoFBEpals title + narr SinaiSpLatTitleNarrNoFBSWordlingo title + narr SinaiSpLatTitleNarrFBWordlingo title + narr Rank 8/31 9/31 11/31 12/31 13/31 14/31 15/31 16/31 19/31 20/31 21/31 22/31 23/31 24/31 25/31 27/31 queries have been formulated with example images and a short textual description explaining the research goal. For the medical task, we have used the list of retrieved images by GIFT1 [ 2 ] which was supplied by the organizers of this track. Also, we used the text of topics for each query. For this reason, our e®orts concentrated in manipulating the text descriptions associated with these images and in mixing the partial results lists. Thus, our experiments do not make a content based image retrieval (CBIR), we only use the list provide by the GIFT system in order to expand textual queries. Textual descriptions of the medical cases have been used to try to improve retrieval results. 3.1

Textual Retrieval System

In order to generate the textual collection we have used the ImageCLEFmed.xml ¯le that links the collections and their images and annotations. It has external links to the images and the associated annotations in XML ¯les. It contains relative paths, from the root directory, to all the ¯les.

The entire collection consists of 4 datasets (CASImage, Pathopic, Peir and MIR) containing about 50,000 images. Each subcollection is organized into cases that represent a group of related images and annotations. Each case consists of a group of images and an optional annotation. Each image is part of a case and has optional associated annotations, which consist of metadata and/or a textual annotation. All of the images and annotations are stored in separate ¯les. ImageCLEFmed.xml only contains the connections between the collections, cases, images, and annotations.

The collection annotations are in XML format. The majority of the annotations are in English but a signi¯cant number is also in French (in the CASImage collection) and German (in the Pathopic collection), with a few cases that do not contain any annotation at all. The quality of the texts is variable between collections and even within the same collection.

We generate a textual document per image, where the identi¯er number of document is the name of the image and the text of document is the XML annotation associated to this image. The XML tags and unnecessary ¯elds such as LANGUAGE were removed. If there were several images of the same case, the text was copied several times.

We have used English language for the document collection as well for the queries. Thus, French annotations in CASImage collection were translated to English and then were incorporated to the collection. Pathopic collection has annotation in both English and German language. We only used English annotations in order to generate the Pathopic documents and German annotations were discarded.

Finally, we have added the text associated to each query topic as documents. In this case, if a query topic includes several images, the text was also copied several times.

Once the document collection was generated, experiments were conducted with the LEMUR2 retrieval information system. We have used the 3 di®erent weighting schemes available: TFIDF, Okapi and Kl-divergence. 3.2

 Experiment Description

Our main goal is to investigate the e®ectiveness of combining text and image for retrieval. For this, we compare the obtained results when we only use the text associated to the query topic and the results when we merge visual and textual information.

We have accomplished a ¯rst experiment that we have used as baseline case. This experiment simply consists of taking the text associated to each query as a new textual query. Then, each textual query is submitted to the LEMUR system. The resulting list is directly the baseline run. This result list from LEMUR system contains the most similar cases with respect to the text and a weighting (the relevance). The weighting was normalized based on the highest weighting in the list to get values between 0 and 1.

1http://www.gnu.org/software/gift/ 2http://www.lemurproject.org/

Experiment IPALI2R TIan (best result) SinaiEn okapi nofb Topics.imageclef2005 SinaiEn okapi fb Topics.imageclef2005 SinaiEn kl fb Topics.imageclef2005 SinaiEn kl nofb Topics.imageclef2005 SinaiEn t¯df fb Topics.imageclef2005 SinaiEn t¯df nofb Topics.imageclef2005

The remaining experiments start from the ranked lists provided by the GIFT. The organization provides a GIFT list of relevant images for each query. For each list/query we have used an automatic textual query expansion of the ¯rst ¯ve images from the GIFT lists. We have taken the text associated to each image in order to generate a new textual query. Then, each textual query is submitted to the LEMUR system and we obtain ¯ve new ranked lists. Again, the resulting list was normalized to 1. Thus, for each original query we have six partial lists. The last step consists of merging these partial result lists using some strategy in order to obtain one ¯nal list with relevant images ranking by relevance. Figure 1 describes the process.

The merging of the visual and textual results was done in various ways: 1. ImgText4: The ¯nal list includes the images present in at least 4 partial lists independently of these lists are visual or textual. In order to calculate the ¯nal image relevance simply we sum the partial relevance and divide by the maxim number of lists which the images are present. 2. ImgText3: This experiment is the same that ImgText4 but the image must be in at least 3 lists. 3. ImgText2: This experiment is the same that ImgText4 but the image must be in at least 2 lists. 4. Img1Tex4: The ¯nal list includes the images present in at least 4 partial lists but the image is necessary to be in the GIFT list (i.e., the image must be in the GIFT list and in at least other 3 textual lists). In order to calculate the ¯nal image relevance simply we sum the partial relevance and divide by the maxim number of lists which the images are present. 5. Img1Text3: This experiment is the same that Img1Text4 but the image must be in at least 3 lists (the GIFT list and at least 2 textual lists). 6. Img1Text2: This experiment is the same that Img1Text4 but the image must be in at least 2 lists (the GIFT list and at least 1 textual list).

These 6 experiments and the baseline experiment (that only uses textual information of the query) have been accomplished with and without PRF for each weighting schemas (TFIDF, Okapi and Kl-divergence). In summary, we have submitted 42 runs: 7 (di®erent experiments)*2 (PRF and no PRF) * 3 (weighting schemas). 3.3

 Results and Discussion

Experiment IPALI2R Tn (best result) SinaiEn kl fb ImgText2.imageclef2005 SinaiEn kl fb Img1Text2.imageclef2005 SinaiEn okapi fb Img1Text2.imageclef2005 SinaiEn okapi nofb Img1Text2.imageclef2005 SinaiEn kl nofb ImgText2.imageclef2005 SinaiEn okapi nofb ImgText2.imageclef2005 SinaiEn okapi fb ImgText2.imageclef2005 SinaiEn kl fb ImgText3.imageclef2005 SinaiEn kl nofb Img1Text2.imageclef2005 SinaiEn okapi nofb ImgText3.imageclef2005 SinaiEn kl fb Img1Text3.imageclef2005 SinaiEn okapi fb ImgText3.imageclef2005 SinaiEn kl nofb ImgText3.imageclef2005 SinaiEn okapi nofb Img1Text3.imageclef2005 SinaiEn okapi fb Img1Text3.imageclef2005 SinaiEn kl nofb Img1Text3.imageclef2005 SinaiEn okapi nofb ImgText4.imageclef2005 SinaiEn t¯df fb Img1Text2.imageclef2005 SinaiEn kl nofb ImgText4.imageclef2005 SinaiEn kl nofb Img1Text4.imageclef2005 SinaiEn kl fb ImgText4.imageclef2005 SinaiEn kl fb Img1Text4.imageclef2005 SinaiEn okapi nofb Img1Text4.imageclef2005 SinaiEn t¯df nofb Img1Text2.imageclef2005 SinaiEn okapi fb Img1Text4.imageclef2005 SinaiEn okapi fb ImgText4.imageclef2005 SinaiEn t¯df fb ImgText2.imageclef2005 SinaiEn t¯df fb Img1Text3.imageclef2005 SinaiEn t¯df fb ImgText3.imageclef2005 SinaiEn t¯df nofb Img1Text3.imageclef2005 SinaiEn t¯df nofb ImgText2.imageclef2005 SinaiEn t¯df fb ImgText4.imageclef2005 SinaiEn t¯df fb Img1Text4.imageclef2005 SinaiEn t¯df nofb ImgText3.imageclef2005 SinaiEn t¯df nofb Img1Text4.imageclef2005 SinaiEn t¯df nofb ImgText4.imageclef2005 There are no signi¯cant di®erences between results obtained with Okapi and Kl-divergence schemas. However, the worst results were obtained with the TFIDF schema.

On the other hand, the use of only two lists is better than mixing three or four lists of partial results. However, a substantial di®erence in the inclusion or not of the images in the GIFT list (Img1TextX experiments) is not appraised either. 4

 Conclusion and Further Works

In this paper, we have presented the experiment carried out in our ¯rst participation in the ImageCLEF campaign. We have only tried to verify if the use of textual information increases the e®ectiveness of the systems. Evaluation results show that the use of textual information signi¯cantly improves the retrieval.

The incorporation of some natural language processing techniques such as word sense disambiguation (WSD) or named entity recognition (NER) will focus our future work. We also plan to use some machine learning algorithm in order to improve the lists merging process. Thus, we should do a comparative study for di®erent fusion methods using basic algorithms (such as Round-Robin or Raw Scoring) and machine learning algorithms (such as logistic regression, neural networks, support vector machine). 5

 Acknowledgements

This work has been partially supported by a grant from the Spanish Government, project R2D2 (TIC2003-07158-C04-04)

Figure 1: The merging process of result lists

 [1] Clough , P. , Sanderson , M. , MuÄller , H.: The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004 . In Proceedings of the Cross Language Evaluation Forum (CLEF 2004 ), 2004 . [2] MuÄller , H. , Geissbhler , A. , Ruch ., P.: Report on the CLEF experiment: Combining image and multi-lingual search medical image retrieval . In Proceedings of the Cross Language Evaluation Forum (CLEF 2004 ), 2004