This paper describes the participation of MIRACLE research consortium at the ImageCLEFmed task of ImageCLEF 2009. The main purpose of our experiments was to determine if any improvement of the linguistic expansion modules that were developed for the previous CLEF campaign, in terms of precision and recall, was possible. Again, we focused on runs using text features only. First a common baseline algorithm was used in all experiments to process the document collection: text extraction, medical-vocabulary recognition, tokenization, conversion to lowercase, filtering, stemming and indexing and retrieval. Then this baseline algorithm was combined with different semantic expansion techniques. Documents were tagged based on the MeSH concept hierarchy using UMLS entities as basic root elements. Relevance-feedback techniques were also used. Average results were obtained.
reformulated query, i.e., both the original terms and the expanded terms were combined with the OR operator. This implied that documents that contained any of those terms were considered as relevant, no matter if the term belonged to the original topic or it was included in the expansion process.
We concluded that a combination of OR and AND operators should have been used to be sure that documents do contain the original topic terms and, optionally, any of the expanded terms:
(original1 OR expanded1) AND (original2 OR expanded2) In addition, we found that the reranking algorithm used for combining the different results list could be one of the reasons for the low precision values obtained in the experiments that make use of the relevance-feedback methods.
Thus, the objective of this year’s experiments was to try to solve those bugs and be able to analyze and compare the performance in terms of precision and recall of the different query expansion techniques. Again, all runs were based on textual features only. All experiments were fully automatic, with no manual intervention.
The architecture of our system is composed of four different modules: the textual (text-based) retrieval module,
which indexes descriptions in order to search and find the most relevant ones to the text of the topic; the
expander module, which performs the expansion of the content of documents and/or topics with related terms
using textual algorithms; the relevance-feedback module, which allows to execute reformulated queries that
include the results of an initial seed query; and, finally, the result combination module, which uses OR operator
to combine, if necessary, the result lists provided by the previous subsystems.
For indexing, instead of using raw terms, the textual information of both topics and documents is parsed and
tagged to unify all terms into concepts of medical entities. This is similar to a stemming or a lemma extraction
process, but the output, instead of the stem or lemma, is the medical entity to which the term relates. The result is
that concept identifiers are used instead of terms in the text-based process of information retrieval.
For this purpose, a terminological dictionary was created by using a subset of the Unified Medical Language
System (UMLS) metathesaurus (US National Library of Medicine) [
Text Extraction: Ad-hoc scripts are run on the files that contain information about the medical cases so as to extract the annotations and metadata enclosed between XML tags.
Medical-vocabulary Recognition: All case descriptions and topics are parsed and tagged using the
UMLS-based terminological dictionary [
Tokenization: This process extracts basic textual components, detecting and isolating punctuation symbols. Some basic entities are also detected, such as numbers, initials, abbreviations, and years. So far, compounds, proper nouns, acronyms or other types of entity are not specifically considered. The outcomes of this process are only single words, years in numbers (e.g. 1995, 2004, etc.) and tagged entities. 4.
Conversion to lowercase: All terms are normalized by changing all uppercase letters to lowercase.
5. Filtering: All words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from the University of Neuchatel’s resources page [
Porter stemmers [
This common baseline algorithm is complemented and combined with semantic expansion techniques. For the
semantic expansion, we used the MeSH concept hierarchy [
Experiments are defined by the choice of different combinations of the previous modules with the different topic expansion techniques, and including relevance-feedback or not.
Language Method EN, FR, DE stem + stopwords + tagged with UMLS thesaurus (baseline) EN, FR, DE baseline + MeSH topic expansion EN, FR, DE baseline + Relevance-Feedback (N=5, M=5) EN, FR, DE baseline + MeSH topic expansion + Relevance-Feedback (N=5, M=5)
EN stem + stopwords + tagged with UMLS thesaurus (baselineEN) EN baselineEN + MeSH topic expansion EN baselineEN + Relevance-Feedback (N=5, M=5) EN baselineEN + Relevance-Feedback (N=10, M=5) EN baselineEN + MeSH topic expansion + Relevance-Feedback (N=5, M=5)
EN baselineEN + MeSH topic expansion + Relevance-Feedback (N=10, M=5) Results are presented in the following tables, which show the run identifier, the number of relevant documents retrieved, the mean average precision (MAP), and the precision at 5, 10, 30 and 100 first results. The best results are highlighted in bold. Overall results achieved for all topics are shown in Table 3. Tables 4, 5 and 6 show the individualized results for visual, mixed and semantic topics, respectively. Independently of the topic type, the highest MAP is achieved with the baseline experiment in English. As in previous campaign, topic expansion using MeSH doesn’t seem to be useful and relevance retrieval leads to noticeably worse results. After a preliminary evaluation, the reranking algorithm used for combining the different results list is again the reason for the low precision values obtained in the experiments that make use of the relevance-feedback methods. Obviously other combination operators must be studied, in special those that assign a higher weight to documents that correspond to the initial query and a lower weight to documents found by the relevance feedback query.
If a comparison among the different topic types is made, it can be clearly observed that topics tagged as visual or mixed achieve noticeably better results than semantic topics. This can be explained by the fact that semantic topics have a very low number of terms (i.e., topics are quite short sentences) as compared to other topic types, and this issue negatively affects experiments based on purely textual information retrieval.
As in previous participation, the value for early precisions (P5, P10) quickly decreases as more documents are considered for the calculation and therefore decreasing the final MAP value. This shows that, although the first results may be appropriate, we probably fail to filter non-relevant documents out of the result list, or perhaps to sort out relevant documents that are “more difficult” to find. Some effort has to be again invested to research on this issue.
After a preliminary analysis, it can be observed that the low number of relevant documents retrieved and the low precision (MAP) values achieved in all experiments in general may be caused by the fact that common terms such as body parts (“head”, “lungs”) that are not directly related to the pathology or diagnosis referenced by the topic, are more frequent in the image description that the actual terms that model or characterize the medical concept (“cancer”, “aneurysm”), which produces that the relevance of the result is determined by those wrong terms instead of the others.
For future participations, we will try to isolate the terms that actually describe each medical case (those terms that refer to any pathology or diagnosis technique) and use them to determine the relevance with respect to the topic, for example using a reranking algorithm to calculate the result list. This work has been partially supported by the Spanish R+D National Plan, by means of the project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007-67407-C03-03 and by Madrid R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267.