In the ImageCLEF 2005 campaign, the LIC2M participated in the ad hoc task, the medical task and the annotation task. For both ad hoc and medical task, we perform experiments on merging the results of two independent search systems: a crosslanguage information retrieval system exploiting the text part of the query and a content-based image retrieval system exploiting the example images given with the query. The results show that a well-tuned merging may improve performance, but the tuning is made difficult because the performance of each system highly depends on the corpus and queries. Annotation task has been performed using a KNN classifier with the image indexes of our CBIR system.
ImageCLEF campaign aims at studying cross-language image retrieval, that potentially uses text and image matching techniques. The LIC2M participated in ImageCLEF 2005 to perform experiments on merging strategies to integrate the results obtained from the cross-language text retrieval system and the content-based image retrieval system that are developed in our lab.
In both ad hoc and medical tasks of the ImageCLEF 2005 campaign, text and visual information were provided for the queries. In ad hoc task, the basic query is textual (title and narrative are provided), but two example images are provided; in medical task, query images are given and a short textual description give precisions about the research goal. We applied the same strategy for the two tasks, using our general-domain systems for multilingual text retrieval and contentbased image retrieval, taking into account both text and visual part of the query and applying a posteriori merging strategies on the results provided independently by each system. We present in section 2 the retrieval systems for text and image and the merging strategies used. We then present the results obtained for the ad hoc task and the medical task in sections 3 and 4 respectively. The strategy and results for the annotation task are presented in section 5.
The multilingual text retrieval system used for these experiments is basically the same as the one used for the previous CLEF campaigns, and a more detailed description can be found in [1]. The system has not been specially adapted to work on the text of the ImageCLEF corpora, and has simply been used as is. In particular, for both the ad hoc and medical corpora, no special treatment has been performed to take into account the structure of the documents (such as photographer's name, location, date for the captions and description, diagnosis, clinical presentation in the medical annotations): all fields containing some text have been taken as is. No adaptation has been made to take into account the specificities of medical texts (specialized vocabulary). Notice that this system is not only cross-lingual but multilingual, because it integrates a concept-based merging technique to merge results found in each target language. Its basic principle is briefly described here.
The documents and queries are processed through a linguistic analyzer, that performs in particular a part-of-speech tagging, a lemmatization, and extracts compounds and named entities from the text. The elements extracted from the documents are indexed into inverted files. The elements extracted from the queries are used as query "concepts". Each concept is reformulated into a set of search terms for each target language, either using a monolingual expansion dictionary (that introduces synonyms and related words), or using a bilingual dictionary. Document Retrieval Each search term is searched in the index, and documents containing the term are retrieved. All retrieved documents are then associated with a concept profile, indicating the presence of query concepts in the document. This concept profile depends on the query concepts, and is language-independent (which allow merging results from different languages). Documents sharing the same concept profile are clustered together, and a weight is associated with each cluster according to its concept profile and to the weight of the concepts (the weight of a concept depends on the weight of each of its reformulated term in the retrieved documents). The clusters are sorted according to their weights and the first 1000 documents in this sorted list are retrieved.
The content-based image retrieval system we used in ImageCLEF 2005 is the system PIRIA (Program for the Indexing and Research of Images by Affinity) [3], developed in our lab. The query image is submitted to the system, which returns a list of images ranked by their similarity to the query image. The similarity is obtained by a metric distance that operates on every image signatures. These indexed images are compared according to several classifiers : principally Color, Texture and Form if the segmentation of the images is relevant. The system takes into account geometric transformations and variations like rotation, symmetry, mirroring, etc. PIRIA is a global one-pass system, feedback or "relevant/non relevant" learning methods are not used.
Color Indexing This indexer first quantifies the image, and then, for each quantified color, it computes how much this color is connex. It can also be described as a border/interior pixel classification [4]. The distance used for the color indexing is a classical L2 norm.
Texture Indexing A global texture histogram is used for the texture analysis. The histogram is computed from the Local Edge Pattern descriptors [2]. These descriptors describe the local structure according to the edge image computed with a Sobel filtering. We obtain a 512-bins texture histogram, which is associated with a 64-bins color histogram where each plane of the RGB color space is quantized into 4 colors. Distances are computed with a L1 norm.
Form Indexing The form indexer used consists of a projection of the edge image along its horizontal and vertical axes. The image is first resized in 100x100. Then, the Sobel edge image is computed and divided into four equal sized squares (up left, up right, bottom left and bottom right). Then, each 50x50 part is projected along its vertical and horizontal axes, thus giving a 400-bins histogram. The L2 distance is used to compare two histograms.
For both ad hoc and medical task, the queries contain textual and visual information. Textual information is used to search relevant text documents with multilingual text retrieval system. For ad hoc task, each text document corresponds to a single image: the images corresponding to the relevant texts are then given as results. For the medical task, a text document may be associated with several images. In that case, the score obtained by the text documents is given to each image it is associated with: the first 1000 images in this image list are kept.
Since the PathoPic corpus of the medical task contains annotations in English and German that are associated with the same image, the multilingual retrieval system may return both English and German annotations as relevant documents (maybe with different scores), creating duplicate elements in the result list. In this case, the score associated with the corresponding image is the best score returned. To make sure that the number of retrieved images is 1000, we set the number of retrieved documents for the text retrieval system at 2000 for the medical task 1 .
Independently, visual information was used by the CBIR system to retrieve similar images. Queries contain several images: a first merging has been performed to obtain a single image list from the results of each query image: the score associated to result images is set to the max of the scores obtained for each query image.
Merging the results obtained by each system is simply done by a weighted sum of the scores obtained by each system. To be comparable, the scores of each system are normalized, for each query, by the highest score obtained for the query. This merging is parameterized by a merging coefficient α: for a query q and an image document retrieved for this query d ∈ Ret(q), the merging score is
where s T (d) is the score of the text retrieval system and s I (d) the score of the image retrieval system.
A conservative merging strategy has also been tested: by conservative, we mean that we use the results obtained by one system only to reorder the results obtained by the other (results can be added at end of list if the number of documents retrieved by main system is less than 1000). The score of a document is modified using the same merging coefficient. For example, if the merging is conservative with the text results:
The results we obtained in ImageCLEF 2004 tend to show that this kind of conservative merging strategies gives good performances. We will use the term of expansionist merging strategy to denote standard merging strategy, as opposed to the conservative one.
In the ad hoc task, we used textual queries in English, French and Spanish. We tried using the title only (T) or the title and the narrative (T+N). Comparative results for textual retrieval only, using either T or T+N are given in Table 1. These results show that average precision is better when using the title only, but the number of relevant documents is generally better when using also the narrative part (except for French, for which it is a bit worse). This can be explained by the fact that narrative introduce more words that allow to increase the total number of documents retrieved (for English and Spanish, there are 6 queries for which the system does not find 1000 documents matching title only, only 3 for French), and the number of relevant documents. But narrative also introduces more noise, which makes the precision decrease. We present in Table 2 the results obtained by the merging of the two systems, using the texture indexer for the CBIR system. The results are presented for conservative and expansionist strategies and for different values of the merging coefficient α (when α = 1, the search is only based on text, when α = 0, the search is only based on images). Values below 0.5 are not presented but does not give better results. For expansionist strategy, the results are given for the mean average precision (map) and the number of relevant documents retrieved (relret); for conservative strategy, only the map is presented (relret is constant).
These results show that this simple merging of text and image results based on a weighted sum of the scores can increase the mean average precision (gain of 17 or 18%) and the best value for α is around 0.7 (though differences with surrounding values are small).
Concerning conservative/expansionist strategies, our previous experiments in ImageCLEF showed that the StAndrews collection, composed of old photographs, is not well adapted the kind of image indexers we use, that rely mostly on color for segmentation. We therefore chose the text retrieval as base for conservative merging. Looking at the relevant documents retrieved prove us right: text retrieval allow to retrieve 1246 relevant documents, whereas image retrieval only retrieve 367 relevant documents (239 of which were also found by the text retrieval system). However, the two merging strategies give comparable results, even though, as one can expect, the performance of the expansionist strategy decreases faster with α. Submitted runs for the ad hoc task in the ImageCLEF 2005 campaign were, for English, French and Spanish, text only (T and T+N), plus a conservative merging of text results based on title only and image results based on the texture indexer, with α = 0.9.
In the medical task, we tested text retrieval using queries in English, French and German (searching for each in all target languages).
Based on our experiments in ImageCLEF 2004, we assumed that image retrieval for the medical task gives good results. Submitted runs for the medical task in the ImageCLEF 2005 campaign include runs based on visual queries only (texture and color indexers), and for English, French and German, a conservative merging of image results based on the texture indexer and text results, with α = 0.9. Unfortunately, the use of texture or color indexer with the ImageCLEFmed 2005 visual queries gave poor results, and conservative merging based on these results does not give much better results2 .
We present in Table 4 the results obtained by the merging of text and image systems, using the texture indexer for the CBIR system, with different values of the merging coefficient α, and for conservative and expansionist merging strategies (conservative strategy based on text results).
Except for German (for which our linguistic processing is clearly not well adapted to medical text), the conservative merging strategy improves performances (the best merging coefficient seems to be around 0.5). Expansionist merging gives comparable results: improvement of mean average precision is less important, but the number of relevant documents retrieved is generally improved, which tends to prove that both systems retrieve different documents3 : conservative merging improves the ordering of documents retrieved by one system whereas expansionist merging improves the number of documents retrieved.
We present in Table 5 similar results using the color indexer for visual retrieval. Results are slightly worse, but the same kind of tendencies as for the texture indexer can be noticed.
For the automatic annotation task, we submitted three runs, each corresponding to one of the three indexers described in section 2 (Color, Texture and Form).
All images are first indexed with the chosen indexer. Then, a k-Nearest Neighbor classifier is used to classify the indexed images. Odd numbers from 3 indexer, and evaluated with the leave-one-out method. The best k were 3 for the form indexer and 9 for the color indexer and the texture indexer.
The attributed class is decided by a majority vote of the nearest neighbors. In case of ties, distances to nearest neighbors are used (for example, in 9-NN, if 4 neighbors are from a class A, 4 neighbors from a class B, and 1 from another class, we use the distances between the requested image and its neighbors to select the nearest class).
We present in Table 6 the results obtained for each of the indexers. It is not a surprise that the form indexer performed better than the others, as all the images in the database were in grey levels, and the form indexer is designed for such images, whereas the color and texture indexers are not well adapted to it (remember that the texture indexer includes a 64-bins color histogram). 9-NN Color 9-NN Texture 3-NN Form error rate 46.0 % 42.5 % 36.9 %
The experiments performed by the LIC2M in the ImageCLEF 2005 campaign show that merging results from different media may increase the performance of a search system: a well-tuned a posteriori merging of the results obtained by two general purpose systems (no particular adaptation of the systems was made for the two tasks) can improve the mean average precision by at least 15%.
The difficulty relies on the tuning of the merging strategy. We used a simple weighted sum of the scores given by each system but the importance given to each system should rely on the performance of the system on a particular corpus, that is not easily predicted (best strategy for the ImageCLEF 2004 medical task appears to be opposite to the best strategy for ImageCLEF 2005 medical task, that has a more varied corpus and more difficult visual queries).
Further experiments will be undertaken to try to make the systems give a confidence score associated with its results and adapt the merging strategy according to this confidence. Other more sophisticated merging strategies will also be considered.