This paper describes the participation of MIRACLE research consortium at the ImageCLEF Medical Image Retrieval task of ImageCLEF 2007. For this campaign, our challenge was to research on different merging strategies, i.e. methods of combination of textual and visual retrieval techniques. We have focused on the idea of performing all possible combinations of well-known textual and visual techniques in order to find which ones offer the best results in terms of MAP and analyze if the combined results may improve the individual ones. Our system consists of three different modules: the textual (text-based) retrieval module, which indexes the case descriptions to look for those descriptions which are more relevant to the text of the topic; the visual (contentbased) retrieval component, which provides the list of case images that are more similar to the topic images; and, finally, the merging module, which offers different operators (AND, OR, LEFT, RIGHT) and metrics (max, min, avg, max-min) to combine and rerank the outputs of the two previous subsystems. These modules are built up from a set of basics components organized in four categories: (i) resources and tools for both general-domain and medical-specific vocabulary analysis, (ii) linguistic tools for text-based information retrieval, (iii) tools for image analysis and retrieval, and (iv) ad-hoc tools for result merging and reranking. We finally submitted 50 runs. The highest MAP was obtained with the baseline text-based experiment in English where only stemming plus stopword removal is performed. Neither tagging with UMLS medical concepts nor merging of textual and visual results proved to be of value to improve the precision with regards to the baseline experiment. However, the most interesting conclusion is that experiments that use the OR operator obtain higher MAP values than those with the AND operator.
The MIRACLE team is a research consortium formed by research groups of three different universities in Madrid (Universidad Politécnica de Madrid, Universidad Autónoma de Madrid and Universidad Carlos III de Madrid) along with DAEDALUS, a small/medium size enterprise (SME) founded in 1998 as a spin-off of two of these groups and a leading company in the field of linguistic technologies in Spain. MIRACLE has taken part in CLEF since 2003 in many different tracks and tasks, including the main bilingual, monolingual and cross lingual tasks as well as in ImageCLEF [7] [8], Question Answering, WebCLEF and GeoCLEF tracks. This paper describes our participation in the ImageCLEFmed task of ImageCLEF 2007. The goal of this task (fully described in [9]) is to improve the retrieval of medical images from heterogeneous and multilingual document collections containing images as well as text. The task organizers provide a list of topic statements (a short textual description explaining the research goal) in English, French and German, and a collection of images (from one to three) for each topic. The objective is to retrieve as many relevant images as possible from the given visual and multilingual topics. ImageCLEFmed 2007 extends the experiments of past editions with a larger database and even more complex queries.
Although this task certainly requires the use of image retrieval techniques and our areas of expertise do not include image analysis research, we do take part to promote and encourage multidisciplinary participation in all aspects of information retrieval, no matter whether it is text or content based.
All experiments are fully automatic, thus avoiding any manual intervention. We submitted runs using only text (text-based retrieval) or only visual features (content-based retrieval) and also mixed runs using a combination of both.
Our system is logically built up from three different modules: the textual (text-based) retrieval module, which indexes case descriptions in order to look for the most relevant ones to the text of the topic; the visual (contentbased) retrieval component, which provides the list of case images that are more similar to the topic ones; and, finally, the result combination module, which uses different operators to combine the results of the two previous subsystems. Figure 1 gives an overview of the system architecture. 2.1.
The system consists of a set of different basic components organized in two categories: • •
Resources and tools for medical-specific vocabulary analysis
Linguistic tools for textual analysis and retrieval.
Instead of using raw terms, the textual information of both topics and documents is parsed and tagged to unify all
terms into concepts of medical entities. This is similar to a stemming or a lemma extraction process, but the
output, instead of the stem or lemma, is the medical entity to which the term relates. The consequence of this
process is that concept identifiers [
Text Extraction: Ad-hoc scripts are run on the files that contain information about the medical cases in order to extract the annotations and metadata enclosed between XML tags. Table 2 shows the metadata which was considered from each collection.
Medical-vocabulary Recognition: All case descriptions and topics are parsed and tagged using a subset
of Unified Medical Language metathesaurus [
Lowercase words: All document words are normalized by changing all uppercase letters to lowercase.
5. Filtering: All words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from [
No feedback or any other kind of expansion was used.
Because the textual retrieval module is completely based on information about medical cases, the last step of module is to obtain the images that correspond to each case (block labeled as AnnotationToImage at Figure 2). 2.2.
For this part of the system, we resorted to two publicly and freely available Content-Based Information Retrieval
systems: GIFT (GNU Image Finding Tool) [
In the case of GIFT, the complete image database was indexed in a single collection, down-scaling each image to 32x32 pixels. For each ImageCLEFmed query, a visual query is made up of all the images contained in the query. Next, this visual query is used in GIFT to obtain the list of the most relevant images (i.e., images which are more similar to those included in the visual query), along with the corresponding relevance values. Although different search algorithms could be integrated as plug-ins in GIFT, only the provided separate normalization algorithm has been used in our experiments.
On the other hand, we directly used the results of the FIRE system kindly provided by the organizers, with no further processing. 2.3.
The textual and image result lists are then merged by applying different techniques, which are characterized by an operator and a metric for computing the relevance (score) of the result. Table 3 shows the defined operators: union (OR), intersection (AND), difference (AND NOT), and external join (LEFT JOIN, RIGHT JOIN). Each of these operators selects which images are part of the final result set. Then, results are reranked by computing a new relevance measure value based on their corresponding input results by using different metrics shown in Table 4.
max min avg mm score = max(a, b) score = min(a, b) score = avg(a, b) score = max(a, b) + min(a, b) *
min(a, b) max(a, b) + min(a, b)
Experiments are defined by the choice of different combinations of the previously described modules, operators and score computation metrics. A wide set of experiments was submitted: 8 text-based runs covering the 3 different topic languages, 9 content-based runs (built with the combination of results from GIFT and FIRE), and also 33 mixed runs (built with the combination of textual and visual experiments). (1) [Query language] > [Annotation language]; “all” refers to the concatenation of text in all languages Results are presented in the following tables. Each of them shows the run identifier, the number of relevant documents retrieved, the mean average precision (MAP), the R-precision and the precision at 10, 30 and 100 first results.
Table 8 shows the results of the text-based experiments. The highest MAP is obtained by the baseline experiment in English where only stemming plus stopword removal is performed. Surprisingly for us, tagging with UMLS thesaurus has proved to be of no use with regards to the simplest strategy. This issue has to be further investigated in case that there is some problem with the generation of the result sets.
TxtXN TxtENT TxtXT TxtFRN TxtFRT TxtDET TxtDEN Experiments using French and German languages achieve a very low precision (respectively, a decrease to 31% and 28% with regards to English). This result is similar to other experiments carried out in other CLEF tracks and may be attributed to deficient stemming modules.
The evaluation for the content-based experiments is shown in Table 9. (1) Evaluations for some experiments with OR operator are missing In general, MAP values are very low, which reflects the complexity and difficulty of the visual-only retrieval for this task. The best value (5% of the top ranked textual experiment) is obtained with the baseline visual experiment, which just uses GIFT. However, probably due to an oversight by the task organizers, the evaluations for the experiments with the OR operator (4 runs) are missing in the Excel files provided. Thus, no definitive conclusion can be extracted about the usage of any merging strategy, as the restrictive AND operator filters out many images (165 instead of 532 relevant images retrieved).
Finally, Table 10 in next page shows the evaluation for the mixed runs. Although the MAP of the best ranked mixed experiment is lower than the MAP of the best textual one (77%), we cannot conclude that the combination of textual and visual results with any kind of merging strategy fails to improve the precision because. The same as before, some experiments with OR operator (11 runs) are missing from the table, thus, it is impossible to extract any valuable conclusion on this issue.
However, observe that the best ranked runs are those with the RIGHT operator, which implicitly includes an OR (see definition in Table 4). In addition, the use of this operator (visual RIGHT textual) shows that textual results are preferred over visual results (RIGHT prioritizes the second result list).
Another conclusion that can be drawn from these results is that the textual retrieval is the best strategy for this task. We think that this is because many queries include semantic aspects such as medical diagnoses or specific details present in the image, which a purely visual retrieval cannot tackle. This issue will be considered for future participations.
The best experiment at ImageCLEFmed 2007 reaches a MAP value of 0.3962, 112% better than ours. Despite this difference, MIRACLE participation is ranked 3rd out of over 12 groups, which is indeed considered to be a very good position. (1) Evaluations for some experiments with OR operator are missing
The highest MAP is obtained with the baseline text-based experiment in English where only stemming plus stopword removal is performed. Neither tagging with UMLS medical concepts nor merging of textual and visual results have proved to be of value to improve the precision with regards to the baseline experiment. However, evaluations for some of our experiments were missing, so this issue cannot be confirmed and has to be further investigated. In addition, experiments using French and German languages get a very low precision. This result is similar to other experiment carried out in other CLEF tracks and may be attributed to deficient stemming modules. We will invest more effort in these languages in future participations.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267. [1] Apache Lucene project. On line http://lucene.apache.org [Visited 10/08/2007].
On line
line