For its rst participation in the ImageCLEF campaign, the LIC2M participated in both the ad hoc task and the medical task. Using our cross-language information retrieval system and content-based image retrieval system, our goal was to perform some rst experiments on merging the results of the two systems. The results show that the performance of each system highly depends on the corpus and the task: with the systems we used, text retrieval alone performs better on the ad hoc task, whereas image retrieval alone performs better on the medical task.
2.1
Document processing The documents are processed through a linguistic analyzer, that performs in particular a part-of-speech tagging, a lemmatization, and extracts compounds and named entities from the text. All these elements are indexed into inverted les. For both the StAndrews and CasImage corpora, no special treatment has been performed to take into account the structure of the documents (such as photographer's name, location, date for the captions and description, diagnosis, clinical presentation in the medical cases): all elds have been taken as a single text to be analyzed.
Query processing The query is rst processed through a similar analyzer (corresponding to the query language) to extract the informative elements of the text. These elements are used as query \concepts". Each concept is reformulated into a set of search terms, either using a monolingual expansion dictionary (that introduces synonyms and related words), or using a bilingual dictionary, depending on the index languages.
Search and merging Each search term is searched in the index, and documents containing the term are retrieved. All retrieved documents are then associated to a concept pro le, indicating the presence of query concepts in the document. Documents sharing the same concept pro le are clustered together, and a weight is associated to each cluster according to its concept pro le and to the weight of the concepts (the weight of a concept depends on the weight of each of its reformulated term in the retrieved documents). The clusters are sorted according to their weights and the rst 1000 documents in this sorted list are retrieved. 2.2
For Image retrieval, we used a system developed at our lab, the LIC2M, called PIRIA (Program
for the Indexing and Research of Images by A nity)[
Color Indexing PIRIA uses a global normalized color histogram. The choice of the color space is very important for a good color division. The model based on Hue, Saturation and Value is used to obtain a strong semantic content. Global histogram is used for the global image or after the segmentation of the image in several blocks. Splitting the image by blocks enables computation of spatial relationship. A more complex color analysis can be used with a region based segmentation. Color information of each region are mixed with form analysis (Fourier descriptors). The distance uses for the color indexing is a classical L1 norm.
Texture Indexing A global texture histogram is used for the texture analysis. The histogram
is computed from the Local Edge Pattern descriptors [
Merge of results The merging of results from several indexers is computed with a boundary fusion based on the position of the result's images. 3
For the ad hoc task, we used topics in English and French. For each of these topic languages, we submitted two runs. The rst one (lic2mSA*1t) was a simple text retrieval, with no use of the image retrieval system. The second one (lic2mSA2*ti) uses a simple merging strategy integrating the results of both text and image retrieval: in this case, the image used for the image retrieval was the example image provided for each topic. The merging strategy was quite straightforward: each image is given a score that is a weighted sum of the scores given by each retrieval systems.
The results of the four runs1 are presented in Figure 1 (precision/recall graph) and Table 1.
From these results, the merging strategy using both text and image does not show much better results than the direct text search. In the case of English topics, a little improvement of the average precision is noticed (not signi cant), which is mostly the e ect of a reordering of retrieved documents (the total number of relevant documents retrieved actually decreases). This is mainly due to the fact that the image retrieval does not perform well on this corpus (indeed, the images need a complex local analysis - based on interest points) The image retrieval alone (using the example images from the queries) retrieves only 122 relevant images out of the 829 relevant images of the partial-isec-total assessments. Of these 122 images, only 8 were not found by the original text retrieval, based on English topics (11 for French topics). Hence, the merging strategy seems to give too much importance to the image result and add noise to the text retrieval (removing relevant images retrieved). Nevertheless, this merging causes a reordering of relevant documents already retrieved that seems to be interesting (at least in the case of English topics).
Further experiments for merging the results of text and image are planned to try to minimize the introduction of non-relevant images in the retrieval results.
The image indexers will also be adapted to treat images such as the old photographs of the StAndrews collection: this image base is particularly di cult for the kind of image indexers we used in this experiments since most of the images are old photographs that are in a kind of 1The o cial results for the merging strategy (lic2mSA*2ti) are erroneous due to a misordering of the queries in the submitted runs (only 17 queries were taken into account out of the 25). monochrome color (with not always the same tone), so that a color segmentation of the image cannot be performed to identify the interesting elements of the images. 4
For the medical task, we submitted two runs. The rst one (lic2mCA1i) was a simple image retrieval search from the image query. The second one (lic2mCA2it) is based on the rst image retrieval search and implements an automated feedback using text information contained in the cases associated with the retrieved images. 4.1
The process of this feedback is the following (a schema presenting the outline of this feedback strategy is proposed in Figure 2):
age search im
t search tex images from similar cases query image image results
case descriptions 2. we collect the cases associated with the images retrieved by the CBIR system (we used only the top ve cases for this feedback); 3. we then use these cases as queries to the text retrieval system to retrieve similar cases, based on the textual description of the cases: we retrieve the 20 most similar cases. These cases are given a score by the text retrieval system (text score, denoted st); 4. we collect the images associated to the cases retrieved by the text retrieval system: these images are candidate images for feedback; 5. since the images retrieved must have the same modality than the query image, we lter these images associated to the similar cases by their similarity to the corresponding images collected in step 12. The similarity with the original image give a score to the candidate images ( ltering score, denoted sf ); 6. The set of feedback images is then used to enrich the rst set of retrieved images (step 1), either by increasing the score of an already retrieved image (function of its image score, the text score and the ltering score) or by adding new images, with an associated score that is a function of the image score of the image that lead to the new image, the text score and the ltering score of the new image. Since the scoring of the di erent systems are not easily comparable, the merging of the three scores is not obvious: we used in the submitted runs an arbitrary function de ned as follows: if the image was already retrieved the score is si + (1 ) f (st; sf ), otherwise, the score attributed is g(si; st; sf ), where f (st; sf ) and g(si; st; sf ) are weighted sums of the di erent scores. The parameter has been introduced to make sure that images retrieved by the rst step are still given more importance (in the experiments, = 0:9). 4.2
results The results of both runs are presented in Figure 3 (precision/recall graph) and Table 2.
In this case, the image retrieval alone performs better than the use of the text retrieval for feedback and enrichment of the retrieved images.
A deeper analysis of the feedback process show that this process produces 17656 image candidates for feedback (step 4: images from similar cases), in which 3195 images were already found 2We could have used directly the query image for ltering: the impact of such a choice should be studied. by direct image retrieval (of which 829 were relevant images) and in the 14461 other images, only 243 are relevant images.
Furthermore, the merging strategy does not succeed in including these 243 documents: only 74 new relevant documents are added, and on the other hand, 82 relevant documents previously included in the initial retrieval are lost. The scoring function we tested introduce too many nonrelevant images. Further testing will be performed on the scoring function, and on the number of cases to consider for feedback and the number of similar cases to look at.
Also, the task imposes that the retrieved images are of the same modality than the query images. Hence, a general similarity on the textual description of the cases is not enough: it can retrieve cases relative to the same kind of pathology but it is not obvious that the images associated to these cases will be similar to the original image We tried to avoid this problem using a second step of image similarity, but a deeper analysis of the text could be needed so that informations on the image modality and anatomic region are extracted from the case description.
Another possible reason for these results is that our text retrieval system is very general. A specialized corpus such as this medical corpus contains many technical words that are treated by the system as unknown words. A more adapted processing of the medical text, giving special importance to terms such as disease names, anatomic regions, medical acts should increase the relevance of the cases similarity. 5
These rst experiments in the ImageCLEF campaign are very interesting: with the same two general purpose systems (no particular adaptation of the systems was made for the two tasks), the results lead to very di erent conclusions according to the task and corpus.
The ad hoc task with the StAndrews collection of old photographs is not well adapted the kind of image indexers we used, that relies mostly on color for segmentation. On the other hand, this task is easier for text retrieval, since the descriptions of the images in the captions are small and precise and the elements in the queries are often found as is in the documents (even without treating the structure of the captions).
The medical task o ers a better eld for image retrieval, the images being \easier" to index (at least, to separate the images by their modality, quite di erent in nature and colors) but in that case, and given the particularity of the task and the specialization of the corpus, the feedback strategy using text information did not improve the results. More experiments should be undertaken to improve the feedback strategy. Other experiments using a training process to associate text to the medical images and use the results for adding text to the image query (textual query expansion) can also be imagined.