In 2008, we participated in medical ImageCLEF in order to compare different strategies of query and document expansion. Since we are a group specialized in Natural Language Processing, we discarded the visual aspect and we only dealt with the textual fields of the documents. We chose descriptors belonging to the Medical Subject Headings (MeSH) in order to expand the queries and the documents; therefore, these metadata were supposed to improve the retrieval process, and to be an interlingua between the collection and the topics for the multilingual tasks. MeSH descriptors for query and document expansion could be automatically computed via two strategies. Each document of this collection is provided with several fields describing the image such as title or caption; so we applied a local lexical MeSH categorizer to these fields in order to automatically extract a set of MeSH descriptors. Moreover, as each document is linked to a journal article via a PMID, we harvested the MeSH descriptors assigned to this article in MEDLINE in order to obtain a second set of MeSH descriptors. For the 2008 official runs, we chose to compare both strategies, but we subsequently showed in an unofficial run that combining them, by merging both sets of MeSH descriptors, led to the best performances. Therefore, combining both strategies increased the Mean Average Precision (MAP) of our best English official run from 0.176 in 2008 to 0.321 in 2009. Results for German and French runs were respectively MAP 0.231 and MAP 0.295.
Images are getting more important and varied in the medical domain, as they become available in digital form.
Despite the fact that images are language-independent, they are often accompanied by textual notes in various
languages. These textual notes can strongly improve the retrieval quality [
In medical ImageCLEF 2009, the size of the collection slightly raised, while the ad hoc task remained unchanged. We then combined both expansion strategies in order to compute our primary run, and take benefits in competition from what our subsequent experiments showed after medical ImageCLEF 2008.
The 2009 collection consisted of around 75’000 images (67’000 in 2008), along with their title, caption, and a PubMed identifier (PMID). The PMID linked the image to a publication contained in MEDLINE, provided with metadata such as authors and abstract, and a set of manually assigned MeSH descriptors. Thus, MeSH descriptors for a given image could be computed via two different strategies: whether being extracted from its title and caption (expansion with captions strategy), or being collected from the assigned MeSH descriptors of the linked publication (expansion with articles strategy). Both sets of MeSH descriptors are then used for document expansion.
The 2009 collection is qualitatively identical to the 2008 collection. More descriptions are given in [
For each image, we aggregated the title and the caption. Then, we applied a local lexical MeSH categorizer in
order to automatically extract a set of MeSH descriptors (Fig. 1). The output of the MeSH categorizer was a
ranked list of MeSH descriptors along with a confidence score. According to past studies, we chose to keep 5
MeSH descriptors per image [
Image in a 12-year-old boy depicts 15-mm nodule (arrows) that is separated from the capsule by intervening thyroid parenchyma. C = carotid artery, E = esophagus, T = thyroid gland, Tr = trachea. </caption> Score 112222 104202 75010 74248 54489 40848 34960 27948 ……..
MeSH descriptor thyroid glands thyroid cancer carotid arteries tracheas esophagus diagnosis capsules carcinomas ……………
A deep analysis of the MeSH descriptors extracted via this strategy showed that they were particularly
relevant to describe diseases and anatomical concepts [
For each image, we extracted the associated PMID. Then, using the PubMed e-utils [5], we collected the assigned MeSH descriptors for this publication in MEDLINE (Fig. 2). For this set of MeSH descriptors, all descriptors were kept for document expansion.
Adenocarcinoma, Follicular ; Adenoma ; Adolescent ; Calcinosis ; Carcinoma, Papillary ; Child ; Diagnosis, Differential ; Female ; Goiter, Nodular ; Humans ; Image Enhancement ; Image Processing, Computer-Assisted ; Imaging, Three-Dimensional ; Male ; Neoplasm Staging; Neovascularization, Pathologic ; Prospective Studies ; Sensitivity and Specificity ; Thyroid Neoplasms ; Thyroid Nodule ;
Ultrasonography, Doppler
A deep analysis of the MeSH descriptors collected via this strategy showed that they were particularly
relevant to describe diagnostic imaging concepts [
Our MeSH categorizer was used in order to compute MeSH descriptors for the queries and to perform query
expansion. According to past studies [
Moreover, we applied a set of manually generated rules in order to describe the diagnostic with the correct MeSH descriptor, according to the best practices in MEDLINE. For instance, for English, when the word radiograph was mapped, we expanded the query with the MeSH descriptor Tomography, X-Ray Computed.
Our MeSH categorizer was developed with EasyIR, a local Natural Language Processing toolkit. EasyIR combines a fuzzy mapping module with a vector-space model. Our MeSH categorizer compares a textual input with indexed MeSH descriptors and their synonyms, and then outputs a ranked list of MeSH descriptors along with a confidence score. More information on EasyIR is available in [7].
The Information Retrieval process was performed with Terrier [8]. We chose classic settings that showed their competitiveness in previous competitions. We used Porter stemming, and the default stopwords list available in Terrier. The weighting scheme was BM25.
Document expansion was simply performed by adding the selected MeSH descriptors at the end of the
document. According to past studies [
<doc> <docID>110931</docID> <title>Diagnosis of thyroid cancer in children: value of gray-scale and power doppler US</title> <caption> Figure 1a. Transverse gray-scale US images. (a) Papillary thyroid carcinoma. Image in a 14year-old girl depicts 9-mm subcapsular hypoechoic nodule (arrows). (b) Follicular thyroid adenoma.
Image in a 12-year-old boy depicts 15-mm nodule (arrows) that is separated from the capsule by intervening thyroid parenchyma. C = carotid artery, E = esophagus, T = thyroid gland, Tr = trachea. </caption> <mesh> thyroid glands D013961; thyroid cancer D013964 ; carotid arteries D002339 ; tracheas D014132 ; esophagus D004947 ; adenocarcinoma, follicular D018263 ; adenoma D000236 ; adolescent D000293 ; calcinosis D002114 ; carcinoma, papillary D002291 ; child D002648 ; diagnosis, differential D003937 ; female D005260 ; goiter, nodular D006044 ; humans D006801 ; image enhancement D007089 ; image processing, computer-assisted D007091 ; imaging, three-dimensional D021621 ; male D008297 ; neoplasm staging D009367 ; neovascularization, pathologic D009389 ; prospective studies D011446 ; sensitivity and specificity D012680 ; thyroid neoplasms D013964 ; thyroid nodule D016606 ; ultrasonography, Doppler D018608 </mesh> </doc>
In medical ImageCLEF 2009, we took benefits from what we showed in subsequent experiments from
ImageCLEF 2008, i.e. combining our both expansion strategies led to great improvements [
Language Best MAP in 2008 Best MAP in 2009
English 0.176 0.321
French 0.115 0.295
German 0.118 0.231
Our best English run reached a Mean Average Precision (MAP) of 0.321 in 2009. By comparison, our best English run in ImageCLEF 2008 was computed with the expansion with articles strategy and achieved a MAP of 0.176. Thus, we confirmed in competition conditions that the merge of both expansion strategies significantly improves the performance of the document expansion. Nevertheless, we cannot assume that this rise is only due to the merge of both expansion strategies, because runs computed exactly in same conditions reached a MAP of 0.254 for the 2008 ImageCLEF. It seems that the task was slightly easier in 2009 than in 2008.
The MAP of the French run (0.295) is not so distant from the English run. We assume that it is due to the language-independent property of our document expansion strategy, because the MeSH plays the role of an interlingua between the different languages. Even if our MeSH categorizer was not evaluated for each language, we assume that it is because the MeSH categorizer has weaker performances for German than the German run was more outdistanced (MAP of 0.231). Moreover, additional French and German runs, computed without Google translator, reached respectively a MAP of 0.275 and a MAP of 0.22.
Finally, we applied a new expansion strategy that was to expand the MeSH descriptors with the set of all their synonyms provided by the MeSH. For instance, when we expanded the query 10 by the MeSH descriptor Pulmonary Edema (D011654), we extended the query with all its synonyms too, such as Wet Lung. This synonyms strategy run reached for English a MAP of 0.205 (- 37%). It seems that expanding with all synonyms is too brutal.
In 2009 medical ImageCLEF, we evaluated a new document expansion strategy, by combining the both strategies applied in 2008: to expand with MeSH descriptors automatically extracted from the image’s metadata, and to expand with the MeSH descriptors collected from the image’s associated MEDLINE publication. We then confirmed in competition conditions that this global expansion strategy led to great improvements compared to a baseline run computed without document expansion. Moreover, this strategy shows interesting language independent properties. However, the best run for all teams reached a MAP of 0.43 in 2009, while it reached a MAP of 0.29 in 2008. Hence, our improvements between 2008 and 2009 must be analyzed with caution, as it seems that the 2009 task was significantly easier than the 2008 one.
5. http://www.ncbi.nlm.nih.gov/corehtml/query/static/eutils_help.html. 6. http://translate.google.com. 7. Gobeill J., Müller H. and Ruch P., “Query and Document Translation by Automatic Text Categorization: A Simple Approach to Establish a String Textual Baseline for ImageCLEFmed 2006”, CLEF working notes 2006, Alicante, Spain, 2006. 8. Ounis I., Lioma C., Macdonald C. and Plachouras V.. “Research Directions in Terrier: a Search Engine for Advanced Retrieval on the Web”, Novatica/UPGRADE Special Issue on Next Generation Web Search, vol 8, pp 49-56, 2007.