This work was part of SUNY at Buffalo's overall participation in cross-language retrieval of image collections (ImageCLEF). Our main goal was to explore the combination of Content-Based Image Retrieval (CBIR) and text retrieval of medical images that have clinical annotations in English, French and German. We used a system that combined the content-based image retrieval system GIFT and the well-known SMART system for text retrieval. Translations of English topics to French were performed by mapping the English text to UMLS concepts using the 2005 UMLS meta-thesaurus. Results show that combining both CBIR and Text retrieval yields significant improvements of retrieval performance.
The organizers of the conference provided a list of the top 1000 retrieved images returned by GIFT for each of the 25 topics. We used this list as the output of the CBIR system. From the textual part of the query we used only the English version provided in the query. This Text was processed with MetaMap,. The current Version of UMLS includes vocabularies with translations in 13 languages (Including French and German which are the ones of interest for this task). For these experiments we decided to use the French terms present in UMLS as translations. We decided not to use German due to the fact that the German documents come from the Pathopic collection and constitute a parallel collection of English and German. Since our initial query is in English we consider redundant to try to get German translations for the topics. However, German terms might be added if they rank high in the automatic relevance feedback process. The corresponding English and French UMLS terms associated with the concepts identified by MetaMap were added to the original English Queries. The text was processed by SMART and the top 10 cases retrieved (together with the cases from top 10 image retrieved by GIFT) are used for expand the query. Observe that this is a multilingual expansion since terms in French, German or English could be added by the query expansion process. The images associated with each case are given the same retrieval score from the text retrieval. Finally we combined the image scores obtained from the CBIR system and from the text retrieval system. We weighted this combination and the results indicate that a 3:1 weighting in favor of text retrieval achieve the best performance in our system.
Results We submitted five runs that included different weights for the contribution of text and images, and several variations on the number of terms used for expansion. The best combination of our official results was obtained by weighting the text results 3 times higher than the visual results. The parameters for the relevance feedback used the top 10 results (both images and text cases), and expand the queries with the top 50 ranked terms using Rocchio’s formula (with α=8, β=64, γ=16). This combination showed a significant improvement for retrieval performance (35% with respect to using text only and 150% using only image retrieval). Our runs were ranked 4th overall and our system was among the best 3 participating in the conference.
( UBimed_en-fr.T.Bl )
n=10, m=50 GIFT results
MAP 0.1746 0.0864 Our preliminary analysis showed significant improvements of 35% over text-only and 173% over queries that used only visual features. We still need to do the query by query analysis to try to identify if there are specific types of queries for which our approach works better.
Our future research plans include exploring in more detail the contribution of the translation using MetaMap and UMLS, types of queries that benefit the most from either text or image retrieval, and sensitivity of the system to different parameters.
Acknowledgements We want to thank the National Library of Medicine for their support of Miguel Ruiz as a visiting faculty at the National Library of Medicine and for allowing us to use UMLS and MetaMap for this research. We want to thank AT&T for supporting this research by providing the necessary hardware used in our experiments through a small grant for research at the UB School of Informatics. Bibliography