We report on simple textual strategies with thesaural resources in order to perform document and query translation for cross-language information retrieval in a collection of annotated medical images. The keystone of our strategy for the previous medical ImageCLEF was to enrich documents and queries with Medical Subject Headings (MeSH) terms extracted from them, in order to translate the more important concepts into an intermediate language. The core technical component of our cross-language search engine is an automatic text categorizer, which associates a set of MeSH terms to any input text, with a top precision at above 90%. Nevertheless, in the new 2008 collection, images are given with more verbose captions, and with an associated article relative to a specific case study. Therefore, our strategy to enrich each document is either to collect MeSH terms from the associated article, either to extract them from the caption. Our results are fair, as we stand on the first part of the participants (0.176 for mean average precision). Nevertheless, it appears that MeSH terms collected from the relative article are not always relevant, as this article can concern a huge set of images in general, and can not to describe precisely the associated image. Moreover, the MeSH terms directly extracted from the captions lead to worst performances, possibly due to the more verbose captions. We try different strategies on weighting scheme or retrieval on articles, but without significant improvements. In conclusion, a mixed strategy to combine the two origins of the MeSH terms should be planned for the next ImageCLEF, while better performances should be obtained in the future by tuning the system with the existing benchmark.
Cross-Language Information Retrieval (CLIR) is increasingly relevant as network-based resources become
commonplace. In the medical domain it is of strategic importance in order to fill the gap between clinical
records, written in national languages and research reports massively written in English. Images are also getting
increasingly important and varied in the medical domain, and they become available in digital form. Despite the
fact that images are language-independent, they are most often accompanied by textual notes in various
languages and these textual notes can strongly improve retrieval quality (
Historically, the most traditional approach to IR in general and to multilingual retrieval in particular, uses a controlled vocabulary for indexing and retrieval. In this approach, a librarian selects for each document a few descriptors taken from a closed list of authorized terms. A good example of such a human indexing is found in the MEDLINE database, where records are manually annotated with Medical Subject Headings (MeSH). The MeSH is a terminology maintained by the National Library of Medicine and which exists in a dozen languages. However, it can be difficult for users to think in terms of a controlled vocabulary. Actually, the use of terminology-based systems – like most Boolean-supported engines – is often performed by professionals rather than general users. Therefore, it can be more efficient for realistic search engine to automatically handle the documents enrichment and query expansion by MeSH concepts.
The Cross Language Evaluation Forum (CLEF) is a challenge which occurs each year since 2000. The goal of this challenge is to evaluate the participants on a common multilingual task, to establish a state of the art of the techniques used in a domain, and to build a benchmark for future evaluations. Medical ImageCLEF has started in 2004 with the goal to retrieve relevant medical images in a multilingual document collection, using visual features – images – or textual features – associated captions, titles and articles.
Our group is specialized in Natural Language Processing; nevertheless, we always participate to
medical ImageCLEF, applying textual strategies based on the picture’s metadata, and the use of MeSH as an
intermediate language (
In 2008, the collection is entirely new, as organizers were able to obtain images from the GoldMiner system (
<imageLocalName>r99ja45g1x.jpeg</imageLocalName> </Record>
Several differences between this new collection and the previous ones must be noted. Firstly, the 2008 collection contains only English texts, contrary to the previous ones which also contained French and German texts. Queries are, as previous years, asked in English, French and German. Secondly, as metadata give a PubMed id (PMID) to each document, human-generated MeSH terms can be automatically collected and associated by following the link to PubMed. Thirdly, an article is provided for each document, even if a set of images belongs to the same article – there are 4961 articles for 67115 images.
The strong point of our strategy, for we participate to ImageCLEF, is focused on associating MeSH
terms to any textual components – documents or queries – in order to enrich the text with language-independent
descriptors, and to perform a standard Information Retrieval process. The core technical component of our
crosslanguage search engine is an automatic text categorizer, which associates a set of MeSH terms to any input text;
the precision at high ranks of this engine for MeSH terms is above 90% (
MeSH D010437 : Peptic Ulcer Synonyms : Ulcère gastroduodénal, Gastroduodenal Ulcer, Marginal Ulcer, Ulcus pepticum, Ulcus marginale, Ulcus gastroduodenale
We merge three versions of MeSH (English, German and French (see figure 2)) in order to enrich each
document with several MeSH terms – between 3 and 8 in 2006, 15 in 2007 – and their unique identifier, making
them efficient regardless of the original language of the document (see figure 3). The number of terms is an
important parameter, as in 2006, the more MeSH terms were added, the best the run was. We finally showed in
2007 that with the previous collection, the ideal number of MeSH concepts per document was around 15 (
Upper Gastrointestinal Ulcers Images from the National Endoscopic Database Endoscopy, Gastrointestinal; Peptic Ulcer
Location: Duodenal Bulb, Not bleeding, clear ulcer base. b) 0127714|peptic ulcer|D010437 0043948|endoscopy|D004724 0039258|endoscopy, gastrointestinal|D016099 0043827|bleeding|D006470 c) Upper Gastrointestinal Ulcers
Images from the National Endoscopic Database Endoscopy, Gastrointestinal; Peptic Ulcer Location: Duodenal Bulb, Not bleeding, clear ulcer base. peptic ulcer D010437 endoscopy D004724 endoscopy, gastrointestinal D016099 bleeding D006470
The same MeSH categorization is then performed on queries; according to past studies, the ideal number
of terms associated by each query is 3 (
With the new collection used for ImageCLEF 2008, and the PMID associated to each document, our strategy is lightly different. Human-generated MeSH terms can be collected for each document, thanks to the PMID contained in the metadata. So, even if our MeSH categorizer obtains good results, we can suppose that “official” descriptors are more accurate and more complete. So, one strong skill of our strategy becomes needless, but we choose to enrich images with the MeSH terms attached to their PMID. Nevertheless, we choose to keep a run where metadata are enriched with MeSH terms found by our categorizer. We try different weighting schemes too, and different combinations between captions, full texts and MeSH terms. The last strategy is to supply a textual run to another team of University and Hospitals of Geneva, Xin Zhou and Henning Muller, who work on visual Information Retrieval, in order to produce a mixed run. More details for each run are given in the Results part.
An important point for us is the use of the three languages into the same query. We think that having the
same queries, perfectly translated into three languages, is not a realistic task. No human user asks a question into
three different perfect translations in a system. So, we think preferable, for each query, to make a run for each
language: English (EN), French (FR) and German (GE). This is an opinion that we defend in each ImageCLEF
we participate, even if we can lose some performance, as seen in 2007 (
Automatic text categorization has been studied largely and has led to an impressive amount of papers. A partial
list of machine learning approaches applied to text categorization includes naïve Bayes (
The mean average precision (map): is the main measure for evaluating ad hoc retrieval tasks (for both
monolingual and bilingual runs). Following (
Two main modules constitute the skeleton of our categorization system: the regular expression (RegEx)
component, and the vector space (VS) component. Each of the basic classifiers implements known approaches to
document retrieval. The first tool is based on a regular expression pattern matcher (
Vector space classifier. The vector space module is based on a general IR engine with the tf.idf weighting
schema. The engine uses a list of 544 stop words. As for setting the weighting factors, we observed that cosine
normalization was especially effective for our task. This is not surprising, considering the fact that cosine
normalization performs well when documents have a similar length (
Classifier fusion. The hybrid system combines the regular expression classifier with the vector-space classifier. We do not merge our classifiers by linear combination, because the RegEx module does not return a scoring consistent with the vector space system. The combination does not use the RegEx's edit distance, and instead it uses the list returned by the vector space module as a reference list, while the list returned by the regular expression module is used as boosting list, which serves to improve the ranking of terms listed in RL.
To translate the medical ImageCLEF textual contents (queries or documents), we transform the English MeSH
mapping tool described above, attributing MeSH terms to English abstracts or queries. Thus, the English,
French, and German version of the MeSH are simply merged in the categorizer. We use the weighting schema
and system combination ( [ dtu.dtn | ltc.atn ] + RegEx ) as described in (
We then describe each run separately.
The Baseline run (BL) is submitted to evaluate the performance of our Information Retrieval Engine alone. The strategy is simple: for each image, the caption and the title are indexed. Then, queries are submitted in 3 languages, without add of MeSH descriptors.
map EN FR GE
BL 0.136 0.069 0.07
As captions and titles are in English, it is not stunning that the English run is the best one. Nevertheless, map of French and German runs, without any strategy, is quite high. Actually, map is very different depending on the queries, because some German and French queries have very specific disease or anatomic names, which are unchanged across languages. For example, the query 24 deals with “malformation de Budd-Chiari” in French. “Budd-Chiari” is a very specific term which is supposed to be a strongly discriminant feature: the query 24 has a map of 0.89 for French. On the contrary, the query 21 “photographies de tumeur” in French has no chance to be associated with a relevant English word contained in a caption: map for the query 21 is 0.002.
The MeSH run (MH) is finally the best one, while it uses the simplest strategy. For each image, the caption, the title and the MeSH terms extracted from MEDLINE with their PMID – MeSH term + MeSH id as seen in figure 3 – are indexed. Queries are enriched by 3 MeSH terms too (see figure 4), and are then submitted. map EN FR GE 0.176 0.105
The benefit for English is +30%, for French +52%, and for German +8%. This is nearly the same order of performance than in the previous ImageCLEF for our strategy. But while we thought that MeSH terms collected from PubMed will be more accurate and more complete, it’s not always the case. For example, the relevant document g01oc14g23x deals with “Budd-Chiari syndrome”. But the MeSH concept “Budd-Chiari syndrome” (D006502) is not a MeSH term for the corresponding PMID (11598252). While the query 24 is enriched with the concept “Budd-Chiari syndrome D006502”, the relevant document is not; so we lose the discriminant power of the feature D006502, which is the keystone of our strategy.
The associated article – and its MeSH terms – seems to be sometimes too general; it perhaps describes more a case study, through a set of images, than the specific image. For example, the PMID 11598252 corresponds to 46 images and deals with hepatocellular carcinoma; the image showing a Budd-Chiari syndrome is not necessarily relevant with this general set, and with the associated MeSH terms.
A problem that we manually discover with the German queries is the complexity of the words used, especially when several words are aggregated in only one. For instance, for the German query 25 “Merkelzellkarzinom”, our categorizer returns no MeSH terms, while we can suppose that some concepts can be mapped if the word was split in “Merkel” “Zell” and “Karzinom”. As the query is enriched with no MeSH terms, the retrieval step returns no documents.
A simple strategy that we choose to beat this difficulty is to use OVID. OVID is a multilingual search
engine in MEDLINE, developed and maintained by the University and Hospitals of Geneva (
map GE
0.107
The benefit for German is +40%. We think it’s a better strategy, at least for our approach based on MeSH descriptors, than trying to translate the German query with an automatic translator – as Babelfish – because we translate directly the query into the chosen intermediate language, i.e. the MeSH. Nevertheless, it could be interesting to compare these results with runs composed with translations strategies.
All following runs are performed with German queries enriched with this strategy. 4.4 ltc run The ltc run is the same as the MeSH run, but the weighting scheme used for Information Retrieval is ltc.atn instead of dtu.dtn. ltc.atn is a weighting scheme which showed good performances at TREC Genomics last years. See 3.1 for more details.
The utilization of this weighting scheme shows no improvements for English and French, while it lightly increase German map.
The weight mix run (mixWeight) is computed by linearly combining the two weighting schemes: dtu.dtn and ltc.atn. Document’s scores from two runs are normalized, then merged.
ltc
There is no improvement obtained for English, but we obtain our best run for German. We can suppose that, as the MeSH strategy is less effective with German (see 4.2) and captions are relatively short documents, articles bring more textual data in order to compare with the German queries. Nevertheless, for English, articles seem to bring more noise than more precisions; there still needs to work on the coefficients of the combination in order to confirm this hypothesis.
For this run, the MeSH terms associated to captions and title are not extracted from MEDLINE thanks to their PMID, but they are mapped with our MeSH categorizer. We choose to keep 15 MeSH terms per document.
While the linear combination increases the map for German and French, it doesn’t for English, which brings the best results. It seems that the combination should be obtained with different factors than 50-50; nevertheless, these factors have to be tuned, and it was impossible to do before having a benchmark.
The mix papers run (mixPapers) is the more sophisticated one. We start working from the MH run. All scores are normalized. Then, we index the full texts attached to the images, and perform a retrieval from the query on this collection: the top relevant documents are then used in order to boost the associated images in the first run (score increased by 10%).
map EN FR GE map visual mixed semantics capMH
The improvement for English is null. Nevertheless, the improvement is significant for French (+34%). It appears that while the MeSH terms collected from the associated article are not relevant for each image, the MeSH terms extracted with our categorizer are certainly not precise enough. We can suppose that the more verbose captions of the images, compared to the previous years’ ones, leads to poorer performances of our MeSH categorizer. The solution could be to mix the two origins of MeSH terms in order to obtain a more complete set of descriptors for each image.
We split the results of our best run, MeSH run for English (see 4.2) in the three types of queries: visual (
This is obviously the semantics queries which achieved the best results. Nevertheless, we notice that this run is nearly at the same rank compared to all participants’ runs, no matter we split the queries in visual, mixed or semantics ones. Actually, it appears that the teams which choose a visual strategy obtain poor results on this benchmark. Taking a closer look to participants’ results shows that some textual runs performs better in the visual queries than in the mix ones (SINAI-sinai_CT_Mesh for instance), and some visual runs performs better in the mixed queries than in the visual ones (GE-GE_GIFT8). We suppose that these anomalies should disappear when the visual techniques will be more adapted to this collection. 4.9 combination of the best run with a visual run from another participating team To obtain these last runs, we supply our best run (MH-EN) to another team of University and Hospitals of Geneva, Xin Zhou and Henning Muller, who are specialized in Visual Information Retrieval.
Map Visual
Mixed Semantics all
0.015 0.061 0.027 0.035 0.076 0.117 0.061 0.085
X Zhou and H Muller performed runs relied mainly on GIFT (
The keystone of our strategy for the previous medical ImageCLEF was to enrich documents and queries with Medical Subject Headings (MeSH) terms extracted from them, in order to translate the more important concepts into an intermediate language. Nevertheless, in the new 2008 collection, images are given with more verbose captions, and with an associated article relative to a specific case study. It appears that MeSH terms of the associated article collected from MEDLINE are not always relevant, as this article can concern a huge set of images dealing with a more general subject, and can not to describe precisely the associated image. Moreover, the MeSH terms extracted directly from the captions leads worst performance, possibly due to the more verbose captions.
For the future ImageCLEF, a mixed strategy to combine the two origins of the MeSH terms should be planned. Better performances should be obtained too by tuning the system with the existing benchmark.