In this paper, we describe our first participation in the ImageCLEF campaign. The SINAI research group participated in both the ad hoc task and the medical task. For the first task, we have used several translation schemas as well as experiments with and without pseudo relevance feedback (PRF). For the medical task, we have also submitted runs with and without PRF, and experiments using only textual query and using textual mixing with visual query.
This is the first participation for the SINAI research group at the ImageCLEF task. We have accomplished the ad hoc task and the medical task.
As a cross language retrieval task, a multilingual image retrieval based on query translation can achieve high performance, more than a monolingual retrieval. The ad hoc task involves to retrieve relevant images using the text associated to each image query.
The goal of the medical task is to retrieve relevant images based on an image query [1]. This year, a short text is associated to each image query. We first compare the results obtained using only textual query versus results obtained combining textual and visual information. We have accomplished several experiments with and without PRF. Finally, we have used different methods to merge visual and text results.
Next section describes the ad hoc experiments. In Section 3, we explain the experiments for the medical task. Finally, conclusions and further works are presented in Section 4. 2 The Ad Hoc Task
The goal of the ad hoc task is, given a multilingual query, to find as many relevant images as possible, from an image collection.
The proposal of the ad hoc task is to compare results with and without PRF, with or without query expansion, using different methods of query translation or using different retrieval models and weighting functions.
In our experiments we have used nine languages: English, Dutch, Italian, Spanish, French, German, Danish, Swedish and Russian
The dataset is the same used in 2004: St Andrews. The St Andrews dataset consists of 28,133 photographs from St Andrews University Library photographic collection which holds one of the largest and most important collections of historic photography in Scotland. The collection numbers in excess of 300,000 images, 10% of which have been digitized and used for the ImageCLEF ad hoc retrieval task. All images have an accompanying textual description consisting of 8 distinct fields. These fields can be used individually or collectively to facilitate image retrieval.
The collections have been preprocessed, using stopwords and the Porters stemmer.
The collection dataset has been indexed using LEMUR IR system. It is a toolkit that supports indexing of large-scale text databases, the construction of simple language models for documents, queries, or subcollections, and the implementation of retrieval systems based on language models as well as a variety of other retrieval models. The toolkit is being developed as part of the Lemur Project, a collaboration between the Computer Science Department at the University of Massachusetts and the School of Computer Science at Carnegie Mellon University.
We have used online Machine Translator for each language pair English-other. After a complete research the best translators are • Systran for Dutch, French, German, Italian, Russian and Swedish • Prompt for Spanish (European) and Spanish (Latinoamerican) One parameter for each experiment is the weighting function, such as Okapi or TFIDF. Another is the use or not of PRF.
Tables 1, 2, 3, 4, 5, 6, 7, 8 and 9 show a summary of experiments submitted and results obtained for the seven languages used.
The results obtained show that in general the application of query expansion improves the results. Only one Italian experiment without query expansion gets a better result. In the case of the use of only title or title + narrative, the results are not conclusive, but the use of only title seems to get better results.
The main goal of medical ImageCLEF task is to improve the retrieval of medical images from heterogeneous and multilingual document collections containing images as well as text. This year, queries have been formulated with example images and a short textual description explaining the research goal. For the medical task, we have used the list of retrieved images by GIFT1 [2] which was supplied by the organizers of this track. Also, we used the text of topics for each query. For this reason, our efforts concentrated in manipulating the text descriptions associated with these images and in mixing the partial results lists. Thus, our experiments do not make a content based image retrieval (CBIR), we only use the list provide by the GIFT system in order to expand textual queries. Textual descriptions of the medical cases have been used to try to improve retrieval results.
In order to generate the textual collection we have used the ImageCLEFmed.xml file that links the collections and their images and annotations. It has external links to the images and the associated annotations in XML files. It contains relative paths, from the root directory, to all the files. The entire collection consists of 4 datasets (CASImage, Pathopic, Peir and MIR) containing about 50,000 images. Each subcollection is organized into cases that represent a group of related images and annotations. Each case consists of a group of images and an optional annotation. Each image is part of a case and has optional associated annotations, which consist of metadata and/or a textual annotation. All of the images and annotations are stored in separate files. ImageCLEFmed.xml only contains the connections between the collections, cases, images, and annotations.
The collection annotations are in XML format. The majority of the annotations are in English but a significant number is also in French (in the CASImage collection) and German (in the Pathopic collection), with a few cases that do not contain any annotation at all. The quality of the texts is variable between collections and even within the same collection.
We generate a textual document per image, where the identifier number of document is the name of the image and the text of document is the XML annotation associated to this image. The XML tags and unnecessary fields such as LANGUAGE were removed. If there were several images of the same case, the text was copied several times.
We have used English language for the document collection as well for the queries. Thus, French annotations in CASImage collection were translated to English and then were incorporated to the collection. Pathopic collection has annotation in both English and German language. We only used English annotations in order to generate the Pathopic documents and German annotations were discarded.
Finally, we have added the text associated to each query topic as documents. In this case, if a query topic includes several images, the text was also copied several times.
Once the document collection was generated, experiments were conducted with the LEMUR2 retrieval information system. We have used the 3 different weighting schemes available: TFIDF, Okapi and Kl-divergence.
Our main goal is to investigate the effectiveness of combining text and image for retrieval. For this, we compare the obtained results when we only use the text associated to the query topic and the results when we merge visual and textual information.
We have accomplished a first experiment that we have used as baseline case. This experiment simply consists of taking the text associated to each query as a new textual query. Then, each textual query is submitted to the LEMUR system. The resulting list is directly the baseline run. This result list from LEMUR system contains the most similar cases with respect to the text and a weighting (the relevance). The weighting was normalized based on the highest weighting in the list to get values between 0 and 1. The remaining experiments start from the ranked lists provided by the GIFT. The organization provides a GIFT list of relevant images for each query. For each list/query we have used an automatic textual query expansion of the first five images from the GIFT lists. We have taken the text associated to each image in order to generate a new textual query. Then, each textual query is submitted to the LEMUR system and we obtain five new ranked lists. Again, the resulting list was normalized to 1. Thus, for each original query we have six partial lists. The last step consists of merging these partial result lists using some strategy in order to obtain one final list with relevant images ranking by relevance. Figure 1 describes the process.
The merging of the visual and textual results was done in various ways:
1. ImgText4: The final list includes the images present in at least 4 partial lists independently of these lists are visual or textual. In order to calculate the final image relevance simply we sum the partial relevance and divide by the maxim number of lists which the images are present.
ImgText3: This experiment is the same that ImgText4 but the image must be in at least 3 lists.
This experiment is the same that ImgText4 but the image must be in at least 2 lists.
The final list includes the images present in at least 4 partial lists but the image is necessary to be in the GIFT list (i.e., the image must be in the GIFT list and in at least other 3 textual lists). In order to calculate the final image relevance simply we sum the partial relevance and divide by the maxim number of lists which the images are present.
Img1Text3: This experiment is the same that Img1Text4 but the image must be in at least 3 lists (the GIFT list and at least 2 textual lists).
6. Img1Text2: This experiment is the same that Img1Text4 but the image must be in at least 2 lists (the GIFT list and at least 1 textual list).
These 6 experiments and the baseline experiment (that only uses textual information of the query) have been accomplished with and without PRF for each weighting schemas (TFIDF, Okapi and Kl-divergence). In summary, we have submitted 42 runs: 7 (different experiments)*2 (PRF and no PRF) * 3 (weighting schemas).
Tables 10 and 11 show the results for medical task (text only and mixed retrieval) with the Sinai system. The total runs submitted for text only were 14 and for mixed retrieval were 86.
Best results were obtained when using Okapi without PRF for text only runs (experiment SinaiEn okapi nofb Topics.imageclef2005) and using Kl-divergence with PRF and ImgText2 experiment for mixed retrieval runs (experiment SinaiEn kl fb ImgText2.imageclef2005). There are no significant differences between results obtained with Okapi and Kl-divergence schemas. However, the worst results were obtained with the TFIDF schema.
On the other hand, the use of only two lists is better than mixing three or four lists of partial results. However, a substantial difference in the inclusion or not of the images in the GIFT list (Img1TextX experiments) is not appraised either.
In this paper, we have presented the experiment carried out in our first participation in the ImageCLEF campaign. We have only tried to verify if the use of textual information increases the effectiveness of the systems. Evaluation results show that the use of textual information significantly improves the retrieval.
The incorporation of some natural language processing techniques such as word sense disambiguation (WSD) or named entity recognition (NER) will focus our future work. We also plan to use some machine learning algorithm in order to improve the lists merging process. Thus, we should do a comparative study for different fusion methods using basic algorithms (such as Round-Robin or Raw Scoring) and machine learning algorithms (such as logistic regression, neural networks, support vector machine).
This work has been partially supported by a grant from the Spanish Government, project R2D2 (TIC2003-07158-C04-04)