This paper presents the contribution of IPAL group on the CLEF 2006 medical retrieval task (i.e. ImageCLEFmed). The main idea of our group is to incorporate medical knowledge in the retrieval system within a multimodal fusion framework. For text, this knowledge is in the Uni ed Medical Language System (UMLS) sources. For images, this knowledge is in semantic features that are learned from examples within structured learning framework. We propose to represent both image and text using UMLS concepts. The use of UMLS concepts allows the system to work at a higher semantic level and to standardize the semantic index of medical data, facilitating the communication between visual end textual indexing and retrieval. The results obtained with UMLS-based approaches show the potential of this conceptual indexing, especially when using a semantic dimension ltering, and the bene t of working within a fusion framework, leading to the best results of ImageCLEFmed 2006. We also test a visual retrieval system based on manual query design and visual task fusion. Even if it provides the best visual results, this purely visual retrieval provides poor results in comparison to the best textual approaches.
retrieval methods could help researchers, lecturers, and student nd relevant images from large repositories. Visual features not only allow the retrieval of cases with patients having similar diagnoses but also cases with visual similarity but di erent diagnoses.
Current CBIR systems [
Among the limited research e orts of medical CBIR, classi cation or clustering driven feature
selection and weighting has received much attention as general visual cues often fail to be
discriminative enough to deal with more subtle, domain-speci c di erences and more objective ground
truth in the form of disease categories is usually available [
The semantic gap can also be reduced by exploiting all sources of information. In
particular, mixing text and image information generally increases the retrieval performance [
In this paper, we present our work on medical image retrieval that is mainly based on the
incorporation of medical knowledge in the system within a fusion framework. For text, this
knowledge is in the Uni ed Medical Language System (UMLS) sources produced by NML1. For
images, this knowledge is in semantic features that are learned from examples and do not rely
on robust region segmentation. In order to manage large and complex sets of visual entities (i.e.,
high content diversity) in the medical domain, we developed a structured learning framework that
facilitates modular design and extract medical visual semantics. We developed two complementary
visual indexing approaches within this framework: a global indexing to access image modality,
and a local indexing to access semantic local features. This local indexing does not rely on region
segmentation but builds upon patch-based semantic detector [
To bene t e ciently from both modalities, we propose to represent both image and text using UMLS concepts in our principal retrieval system. The use of UMLS concepts allows our system to work at a higher semantic level and to standardize the semantic index of medical data, facilitating the communication between visual end textual indexing and retrieval. We propose several fusion approaches and a visual modality ltering is designed to remove visually aberrant images according to the query modality concept(s).
Besides this UMLS-based system, we also investigate the potential of a closed visual retrieval system where all queries are xed and manually designed (i.e. several examples are manually selected to represent each query).
Textual, visual, and mixed approaches derived from these two systems are evaluated on the medical task of CLEF 2006 (i.e. imageCLEFmed).
1National Library of Medicine -http://www.nlm.nih.gov/ UMLS is a good candidate as a knowledge base for medical image and text indexing. It is more than a terminology base because terms are associated with concepts. There exists also di erent type of links. The base is large (more than 50,000 concepts, 5.5 million of terms in 17 languages), and is maintained by specialists with two updates a year. Unfortunately, UMLS is a merger of di erent sources (thesaurus, terms lists), and is neither complete, nor consistent. In particular, the links among concepts are not equally distributed. UMLS is a \meta thesaurus", i.e. a merger of existing thesaurus. It is not an ontology, because there is no formal description of concepts, but its large set of terms and variation restricted to medical domain only, enable us to experiment a full scale conceptual indexing system. In UMLS, all concepts are assigned to at least one semantic type from the Semantic Network. This provides consistent categorization of all concepts in the meta-thesaurus at the relatively general level represented in the Semantic Network. This partially solves the problem of merging existing thesaurus hierarchy during the merging process.
Despite the large set of terms and terms variation available in UMLS, it still cannot cover all
possible (potentially in nite) term variation. So we need a concept identi cation tool that manages
terms variation. For English texts, we use MetaMap[
The extracted concepts are then organized in conceptual vectors, like a conventional vector
space IR model. We then use the same weighting scheme provided by our XIOTA indexing system
[
We tested six retrieval approaches based on this conceptual indexing and an approach - corresponding to run IPAL Textual TDF - based on an indexing using MeSH2 terms.
Each conceptual text retrieval approach uses a Vector Space Model (VSM) for representing each document and a cosine similarity measure to compare the query index to the database medical report. The tf idf measure is used to weight the concepts. The mapping text-concept is separate for three languages, query vectors of concepts of three languages are fusioned, and used for interrogation using the three indexing separately. Then, the three relevance status values are fusioned together.
One major criticism we have against VSM is the lack of structure of the query. VSM is known 2Medical Subject Headings, MeSH, is the controlled vocabulary thesaurus of the U.S. National Library of Medicine. It is included in the meta-thesaurus UMLS. to perform well using long textual queries but ignoring query structure. The ImageCLEFmed 2006 queries are rather short. Moreover, it seems obvious to us that it is the complete query that should be solved and not only part of it. After query examination, we found out that queries are implicitly structured according to some semantic types (e.g. anatomy, pathology, modality). We call this the "semantic dimensions" of the query. Omitting correct answer to any of these dimensions may lead to incorrect answers. Unfortunately VSM does not provide a way to ensure answers to each dimension.
To solve this problem, we decided to add a semantic dimension ltering step to the VSM, in order to explicitly taking into account the query dimension structure. This extra ltering step retains answers that incorporate at least one dimension. We use semantic structure on concepts provided by UMLS. Semantic dimension of a concept is de ned by its UMLS semantic type, grouped into semantic groups: Anatomy, Pathology and Modality. Only a conceptual indexing and a structured meta-thesaurus like UMLS enable us to do such a semantic dimension ltering (DF). This ltering discards noisy answers regarding to the dimension query semantic structure. The corresponding run is \IPAL Textual CDF". We also test a similar dimension ltering based MESH terms (run \IPAL Textual TDF"). In this case, the association between MeSH terms and a dimension had to be done manually. According to Table 1, using UMLS concepts more than terms improves the results of 2 Mean Average Precision (MAP) points (i.e. from 21% to 23%).
Another solution to take into account query semantic structure is to re-weight answers according to dimensions (DW). Here, Relevance Status Value output from VSM is multiplied by the number of concepts matched with the query according to the dimensions. This simple reweighting scheme strongly emphasizes the presence of maximum number of concepts related to semantic dimensions. This re-weighting step implicitly do the previous dimension ltering (DF), as the relevance value is multiplied by 0. According to our results in Table 1 this DW approach - corresponding to the run \IPAL Textual CDW" - produces the best results of 2006 ImageCLEFmed with 26% of MAP. This result outperforms any other classical textual indexing reported in ImageCLEFmed 2006. Hence we have shown here the potential of conceptual indexing.
In run \IPAL Textual CPRF", we tested Pseudo-Relevance Feedback (PRF). From the result of late fusion of text-image retrieval results, three top relevant documents retrieved are taken and all concepts of these documents are added into query for query expansion. Then, a dimension ltering is applied. In fact, this run should have been classi ed in the mixed runs as we also use the image information to have a better precision in the three rst images. This PRF approach improves slightly the results obtained with a simple dimension ltering. This is principally due to the fact errors can be present in the three rst documents, even with the best mixed retrieval result. Using a manual Relevance Feedback (RF), we obtained a MAP of 25% that is 2 points higher than the result obtained with a simple dimension ltering. In this last run - named \IPAL Textual CRF" - a maximum of 4 top relevant documents were chosen by human judgment over 20 rst retrieved image. All the concepts from these documents are added into the query for query expansion.
We also tested document expansion using the UMLS semantic network. Based on UMLS hierarchical relationships, each database concept is expanded by concepts positioned at a higher level in the UMLS hierarchy and connected to this concept with respect to the semantic relation \is a". The expanded concepts have a higher position than document concept in UMLS hierarchy. For example a document indexed by the concept \molar teeth" would be also indexed by the more general concept "teeth". This document would be thus retrieved if the user ask for a teeth photography. This expansion does not seem relevant according to the Table 1 as the run \IPAL Textual CDE" - that uses a document expansion technique and a dimension ltering - is 4 points below the simple dimension ltering. 3.1
UMLS-based visual indexing and retrieval In order to manage large and complex sets of visual entities in the medical domain, we developed a structured learning framework to facilitate modular design and learning of medical semantics from images. This framework allows to index images using VisMed terms, that are typical semantic tokens characterized by a visual appearance in medical image regions. Each VisMed term is expressed in the medical domain as a combination of UMLS concepts. In this way, we have a common language to index both image and text, which facilitates the communication between visual and textual indexing and retrieval. We developed two complementary indexing approaches within this statistical learning framework: a global indexing to access image modality (chest X-ray, gross photography of an organ, microscopy, etc.); a local indexing to access semantic local features that are related to modality, anatomy, and pathology concepts.
After a presentation of both approaches in Sections 3.1.1 and 3.1.2, retrieval procedures and experimental results are given in Section 3.1.3. 3.1.1
Global UMLS Indexing The global UMLS indexing is based on a two level hierarchical classi er according to mainly modality concepts. This modality classi er is learned from about 4000 images separated in 32 classes: 22 grey level modalities, and 10 color modalities. Each indexing term is characterized by a UMLS modality concept (e.g. chest X-ray, gross photography of an organ) and, sometimes, a spatial concept (e.g. axial, frontal, etc), or a color percept (color, grey). The training images come from the CLEF database (about 2500 examples), from the IRMA3 database (about 300 examples), and from the web (about 1200 examples). The training images from ImageCLEFmed database was obtained from modality concept extraction using medical reports. A manual ltering step on this extraction process to remove irrelevant examples had to be performed. We plan to automate this ltering in the near future.
The rst level of the classi er corresponds to a classi cation for grey level versus color images.
Indeed, some ambiguity can appear due to the presence of colored images, or the slightly blue or
green appearance of X-ray images. This rst classi er uses the rst three moments in the HSV
color space computed on the entire image. The second level corresponds to the classi cation of
modality UMLS concepts given that the image is in the grey or the color cluster. For the grey
level cluster, we use grey level histogram (32 bins), texture features (mean and variance of Gabor
coe cients for 5 scales and 6 orientations), and thumbnails (grey values of 16x16 resized image).
For the color cluster, we have adopted HSV histogram (125 bins), Gabor texture features, and
thumbnails. Zero-mean normalization [
yj2) jx yj = 1 XF jxf F f=1
Nf yf j (1) (2) where x = fx1; :::; xF g and y = fy1; :::; yF g are feature vectors, xf ; yf are feature vectors of type f, Nf is the feature vector dimension, and F is the number of feature types: F = 1 for the grey versus color classi er, F = 3 for the conditional modality classi ers: color, texture, thumbnails.
3http://phobos.imib.rwth-aachen.de/irma/index_en.php where C and G denote the color and the grey level clusters respectively, and the conditional probability P (MODijz; V ) is given by: (3) (4) (5) P (cjz; V ) =
expDc(z)
Pj2V expDj(z) where Dc is the signed distance to the SVM hyperplane that separate class c from the other classes of the cluster V .
After learning - using SVM-Light software4 [
Local UMLS Indexing
To better capture the medical image content, we propose to extend the global modeling and
classi cation with local patch classi cation of local visual and semantic tokens (LVM terms).
Each LVM indexing term is expressed as a combination of Uni ed Medical Language System
(UMLS) concepts from Modality, Anatomy, and Pathology semantic types. A Semantic Patch
Classi er was designed to classify a patch according to the 64 LVM terms. In these experiments,
we have adopted color and texture features from patches (i. e. small image blocks) and a classi er
based on SVMs and the softmax function [
After learning, the LVM indexing terms are detected during image indexing from image patches
without region segmentation to form semantic local histograms. Essentially, an image is tessellated
into overlapping image blocks of size 40x40 pixels after size standardization. Each patch is then
classi ed into one of the 64 LVM terms using the Semantic Patch Classi er. An image containing
P overlapping patches is then characterized by the set of P LVM histograms and their respective
location in the image. An histogram aggregation per block gives the nal image index : M N
LVM histograms. Each bin of the histogram of a given block B corresponds to the probability of
a LVM term presence in this block. This probability is computed as follows:
We use = 1 in all our experiments. This just-in-time feature fusion within the kernel combines
the contribution of color, texture, and spatial features equally [
The probability of a modality MODi for an image z is given by:
P (MODijz) =
P (MODijz; C)P (Cjz) if MODi 2 C
P (MODijz; G)P (Gjz) if MODi 2 G P (VMTijB) =
Pz jz \ Bj P (VMTijz)
Pz jz \ Bj where B is a block of a given image, z denotes a path of the same image, jz \ Bj is the area of the intersection between z and B, and P (VMTijz) is given by Equation (4). To facilitate spatial aggregation and matching of image with di erent aspect ratios , we design 5 tiling templates, namely M N = 3 1, 3 2, 3 3, 2 3, and 1 3 grids resulting in 3, 6, 9, 6, and 3 probability vectors per image respectively. 3.1.3
Visual retrieval using UMLS-based visual indexing We propose three retrieval methods from query by example(s) based on the two UMLS-based visual indexing. When several images are given in the query, the similarity between a database image z with the query is given by the maximum value among the similarities between z and each query image.
The rst method - corresponding to run \IPAL Visual MC" - is based on the global indexing scheme according the modality. An image is represented by a semantic histogram, each bin corresponding to a modality probability. The distance between two images is given by the Manhattan distance (i.e. city-block distance) between the two semantic histograms.
The second method - corresponding to run \IPAL Visual SPC" - is based on the local UMLS
visual indexing. An image is then represented by M N semantic histograms. Given two images
represented as di erent grid patterns, we propose a exible tiling (FlexiTile) matching scheme to
cover all possible matches [
The last visual retrieval method - corresponding to run \IPAL Visual SPC+MC" - is the fusion of the two rst approaches. This approach combines thus two complementary sources of information, the rst concerning the general aspect of the image (global indexing according to modality), the second concerning semantic local features with spatial information (local UMLS indexing). The similarity to a query is given by the mean of the similarity to a query according each index.
The 2006 CLEF medical task was particularly di cult this year for purely visual approaches. Indeed, the queries were at a hight semantic level for a general retrieval system. As a proof, the best automatic visual result was less than 8% of MAP. Mixing the local and global indexing, gives us the third place with 6% of MAP as showed in Table 25. We believe than we can improve these results using also the textual query in the retrieval process. Indeed, besides the usual similaritybased queries, our semantic indexing allow semantic-based query. Tests are in course on 2005 and 2006 medical tasks, providing promishing results.
Manual Query Construction and Visual Task Fusion
To see how far we can go with a purely visual approach, we propose here a closed visual system
based on manual query construction and visual task fusion. This work is similar to what we
did in ImageCLEFmed 2005[
5The Mean Average Precision and the Recall-precision computed in imageCLEFmed for the run IPAL Visual SPC+MC was 6:34% and 10:48% respectively because we only submitted - by error - the 25 rst queries.
In this year's attempts, we have submitted 10 runs based on the fusion of six sub-runs. The generated sub-runs are denoted D1,D2,D3,D4, D5 and D6. D1,D2 and D3 use \Dist" retrieval system but di erent features (D1 using \icon", D2 using \blob", D3 using \blob+icon"), D4 and D5 use \SVM" with di erent features (D4 using \blob", D2 using \icon"), and D6 uses the UMLSbased system presented in Section 3.1 (D6 corresponds to the run \IPAL Visual MC" that is based on global UMLS indexing).
Di erent from the work done in 2005, we also use the probability estimation of each image about its modality that is given by Equation (3). Some of these sub-runs (D1-D6) are linearly combined together to produce a score for each image. Each score may then be multiplied by the probability and all the results are sorted to yield the nal retrieval ranking lists. The runs \IPAL CMP D1D2D4D5D6", \IPAL CMP D1D2D3D4D5", \IPAL CMP D1D2D3D4D5D6"6, and \IPAL CMP D1D2D4D5" used the probability estimations. We also applied a color lter to remove those images whose number of color channels are less than that of the query images, except for runs \IPAL D1D2D4D5D6" and \IPAL D1D2D4D5". The performance of these runs is given in Table 3.
Applying a probability estimation of each image about its modality and imaging anatomy helps (compare \IPAL CMP D1D2D4D5D6" (MAP=0.1596) and \IPAL cfD1D2D4D5D6" (MAP=0.155)); Color ltering generally improves performance, but not signi cantly; Using D6 in the combination, the performance is improved. Compare \IPAL cfD1D2D4D5D6" (MAP=0.155) versus \IPAL cfD1D2D4D5" (MAP = 0.1461) and \IPAL D1D2D4D5D6" (MAP=0.1551) versus \IPAL D1D2D4D5" (MAP=0.1461). 4
We propose three types of fusion between text and images: a simple late fusion (run \IPAL Cpt Im"); a fusion that uses a visual ltering according to modality concept(s) (runs \IPAL ModFDT Cpt Im", \IPAL ModFST Cpt Im", \IPAL ModFDT TDF Im", and \IPAL ModFDT Cpt"); an early fusion of UMLS-based visual and textual indexes (run \IPAL MediSmart 1" and \IPAL MediSmart 2").
The rst fusion method is a late fusion of visual and textual similarity measures. The similarity between a mixed query Q = (QI ; QT ) (QI : image(s), QT : text) and a couple composed of an image and the associated medical report (I; R) is then given by: (Q; I; R) =
V (QI ; I) max V (QI ; z) z2DI + (1 )
T (QT ; R) max T (QT ; z) z2DT where V (QI ; I) denotes the visual similarity between the visual query QI and an image I,
T (QT ; R) denotes the textual similarity between the textual query QT and the medical report R, DI denotes the image database, and DT denotes the text database. The factor allows the control of the weight of the textual similarity with respect to the image similarity. After some experimentations on imageCLEFmed 2005, we choose = 0:7. In order to compare similarities in the same range, each similarity is divided by the corresponding maximal similarity value on the entire database. The result of the corresponding run, \\IPAL Cpt Im", given in Table 4 show the good complementarity of the visual and textual indexing: from 26% for the textual retrieval and 6% for the visual retrieval, the mixed retrieval provides 31% of MAP. The best results on imageCLEFmed 2006 in terms of MAP and R-precision (i.e. precision after R retrieved images, where R is the number of relevant images) were obtained with this simple late fusion.
The second type of fusion exploits directly the UMLS index of images. Indeed, it is based on a direct matching between concepts extracted from the textual query and conceptual image indexes. This direct matching is done automatically with the use of the Uni ed Medical Language System. More speci cally, a comparison between the query concepts related to modality and the image modality index is done in order to remove all aberrant images. The decision rule is the following: an image I is admissible for a query modality MODQ only if:
P (MODQjI) > (MODQ) where (MODQ) is a threshold de ned for the modality MODQ. This decision rule de nes a set of admissible images for a given modality MODQ: fI 2 DI : P (MODQjI) > (MODQ)g. The nal result is then the intersection of this set and the ordered set of images retrieved by any system. This modality lter is particularly interesting for ltering textual retrieval results. Indeed, several images of di erent modalities can be associated to the same medical report. The ambiguity is thus removed when using a visual modality ltering. We test this approach with, rst, a xed threshold for all modality (MODQ) = 0:15 (\ModFST") based on experimental tests on 2005 CLEF medical task, and, second, an adaptive threshold for each modality according a con dence degree given to the classi er according this modality (\ModFDT"). The adaptive thresholding performs slightly better than the constant thresholding (compare \IPAL ModFDT Cpt Im" and \IPAL ModFST Cpt Im" in Table 4). In fact, we have over-estimated these thresholds for most modalities. Indeed, when this modality ltering is applied to the late fusion results the results 6\IPAL CMP D1D2D3D4D5D6" corresponds to \IPAL CMP D1D2D3D4D5D" in ImageCLEFmed where the last letter was missing (6) (7) n o i isc 0.4 e r P 0.7 0.6 0.5 0.3 0.2 0.1 IPAL_ModFDT_Cpt_Im
IPAL_ModFDT_Cpt
IPAL_Cpt_Im
IPAL_Textual_CDW IPAL_Visual_SPC+MC decrease of 2 points (compare \IPAL ModFDT Cpt Im" and \IPAL Cpt Im"). That means that this ltering not only removes aberrant images but also relevant images. This ltering nevertheless increases the results of the purely textual retrieval approach from 26% to 27% (see \IPAL ModFDT Cpt" in Table 4 and \IPAL Textual CDW" in Table 1). Moreover, this ltering is relevant if the user - which is often the case - is more interested in the precision in the rst retrieved images that in the mean average precision. Indeed, the Figure 1 shows that the adaptive modality ltering on the late fusion results (\IPAL ModFDT Cpt Im") and even directly on the textual results (\IPAL ModFDT Cpt") provides a better precision than the late fusion results (\IPAL Cpt Im") for the rst retrieved documents (until 30 when applied on textual results, until 50 when applied on the mixed results).
20 40 60 80 100 120 Number of documents 140 160 180 200
We also submitted two runs concerning the early fusion of UMLS-based visual and textual
indexes. A Semantic level fuzzy cation algorithm takes into account the frequency, the localization,
the con dence and the source of the information [
In this paper, we have proposed a medical image retrieval system that represents both texts and images at a very high semantic level using concepts from the Uni ed Medical Language System. Textual, visual, and mixed approaches derived from this system were evaluated on ImageCLEFmed 2006. A structured framework was proposed to bridge the semantic gap between low-level image features and the semantic UMLS concepts. A closed visual system based on manual query construction and visual task fusion was also tested to go as far as possible using a purely visual approach. From the results on ImageCLEF 2006, we can conclude that the textual approaches capture more easily the semantics of the medical queries, providing better results than purely visual retrieval approaches. Indeed, the best visual approach in 2006 - that corresponds to a result of our closed system - only provides 16% of MAP against 26% of MAP for the best textual results. Moreover, the results show the potential of conceptual indexing, especially when using a semantic dimension ltering: we obtained the best textual and mixed results in imageCLEF 2006 using our UMLS-based system. The bene t of working in a fusion framework has been demonstrated. Firstly, visual retrieval results are enhanced by the fusion of global and local similarities. Secondly, mixing textual and visual information improves signi cantly the system performance. Besides precision in the rst documents increases when using a visual modality ltering, allowing 68% of mean precision on the 10 rst documents and 62% of mean precision for the 30 rst documents on the 30 queries of ImageCLEF 2006. We are currently investigating the potential of an early fusion scheme using appropriate clustering methods. In the near future, we plan to use the LVM terms from local indexing for semantics-based retrieval (i.e. cross-modal retrieval: processing textual query on LVM-based image indexes). A visual ltering based on local information could also be derived from the semantic local indexing.