This paper present the details of participation of DEMIR (Dokuz Eylul University Multimedia Information Retrieval) research team to the ImageCLEF 2012 Medical Retrieval task. This year, we evaluated impact of our proposed Integrated Combination method and Explicit Graded relevance feedback on the most descriptive low level features of images and best text retrieval result. We improved results by examination of different level of integrated approach for retrieved text data and low level features. We tested multi-modality image retrieval in ImageCLEF 2012 medical retrieval task and obtained the best rank in visual retrieval due to our experiments. The results clearly show that proper combination of different modalities improve the overall retrieval performance.
2
It seems that the best performance of multi-modality fusion appears when we can put all relevant retrieved documents of each fused modalities in response to a query into relevant retrieved document set of fused modalities. More formally, Union of modalities’ relevant retrieved document sets can achieve the best result in fusion. However, there are some documents that are relevant but not appear in any individual modalities’ result set. This is the limit of multimodality fusion. Because of that, in literature of multimodal data fusion, some of the authors claimed that data fusion algorithms are competitive in performance and is not devoid of risks and sometimes can degrade the retrieval performance.
Our previous studies illustrated that main reason for this downside
originated from threshold on number of fused document (
In order to evaluate the impact of our proposed method on improvement of overall system performance, we designed and implemented an experimental integrated combination multimodal retrieval system. In this system, each modality was preprocessed and indexed separately. Then the results obtained from different modalities in response to each query combined based on selected combination methods and produced the final result set of retrieved documents. This combination can be inter-modality (between results of different modalities) or intra-modality (between different features of a modality).
In order to simplify the work, we split the XML file for textual metadata
and represented each image in the collection as a structured document of
xml file. We also expanded the XML file using related article full text, abstract
and title as new tags. We used Terrier IR Platform API, open source search
engine written in Java and is developed at the School of Computing Science,
University of Glasgow (
Since the choice of the weighting model may crucially affect the performance
of any information retrieval system, first of all we decided to work on
evaluating the relative merits and drawbacks of different weighting models
using Terrier IR Platform (
Since choice of the weighting model may crucially affect the performance of any information retrieval system, we performed our experiments on textual features using TF-IDF, the most common and famous weighting model in IR systems.
The visual content of the images is mapped into a new space named the
feature space. A feature is a set of characteristics of the image. In addition, a
feature can further be enriched with information about the spatial
distribution of the characteristic that it describes. Selection of right features,
typically represented as vectors in this space, is the major aspect to attain
discriminative and sufficient retrieval systems. We extracted features for all
images in test collection and query examples using Rummager tool (
The CEDD includes texture information produced by the six-bin histogram
of the fuzzy system that uses the five digital filters proposed by the MPEG-7
EHD. Additionally, for color information the CEDD uses the 24-bin color
histogram produced by the 24-bin fuzzy-linking system. Overall, the final
histogram has 144 regions. Each Image Block interacts successively with all
the fuzzy systems. In the Texture Unit, the Image Block is separated into four
regions called Sub Blocks. The value of each Sub Block is the mean value of
the luminosity of the pixels it contains. The luminosity values are derived
from a YIQ color space transformation. Each Image Block interacts with the
five digital filters proposed by MPEG-7 EHD, and with the use of the
pentagonal diagram it is classified in one or more texture categories. Then, in
the Color Unit, every Image Block is converted to the HSV color space. The
mean values of H, S and V are calculated and become inputs to the fuzzy
system that produces the fuzzy ten-bin histogram. Then, the second fuzzy
system uses the mean values of S and V as well as the position number of the
bin (or bins) resulting from the previous fuzzy ten-bin unit, calculates the hue
of the color and produces the fuzzy 24-bin histogram. The combination of
the three fuzzy systems will finally classify the Image Block. The process is
repeated for all the image blocks. At the completion of the process, the
histogram is normalized and quantized (
The FCTH descriptor includes the texture information produced in the
eightbin histogram of the fuzzy system that uses the high frequency bands of the
Haar wavelet transform. For color information, the descriptor uses the
24bin color histogram produced by the 24-bin fuzzy-linking system. Overall, the
final histogram includes192 regions. Each Image Block interacts successively
with all the fuzzy systems in the exact manner demonstrated in CEDD
production. Each Image Block is transformed into the YIQ color space and
transformed with the Haar Wavelet transform. The fLH, fHL and fHH values
are calculated and with the use of the fuzzy system that classifies the f
coefficients, this Image Block is classified in one of the eight output bins.
Next, the same Image Block is transformed into the HSV color space and the
mean H, S and V block values are calculated. These values become inputs to
the fuzzy system that forms the ten-bin fuzzy color histogram. Then, the next
fuzzy system uses the mean values of S and V as well as the position number
of the bin (or bins) resulting from the previous fuzzy ten-bin unit, to calculate
the hue of the color and create the fuzzy 24-bin histogram. The combined
three fuzzy systems therefore classify the Image Block. The process is
repeated for all the blocks of the image. At the completion of the process,
the histogram is normalized and quantized (
This feature is very similar to FCTH feature. The main difference from FCTH feature is using brightness instead of color histogram. This descriptor uses brightness and texture characteristics as well as the spatial distribution of these characteristics in one compact 1D vector. The most important characteristic of the proposed descriptor is that its size adapts according to the storage capabilities of the application that is using it. This characteristic renders the descriptor appropriate for use in large medical (or gray scale) image databases. To extract the proposed descriptor, a two unit fuzzy system is used. To extract the brightness information, a fuzzy unit classifies the brightness values of the image’s pixels into L_{Bright} clusters. The cluster centers are calculated using the Gustafson Kessel Fuzzy Classifier. The texture information embodied in the proposed descriptor comes from the Directionality histogram. This feature is part of the well-known Tamura texture features. Fractal Scanning method through the Hilbert Curve or the Z-Grid method is used to capture the spatial distribution of brightness and texture information (7) After extracting features, we examined the performance of all extracted feature and perceived that compact composite features like CEDD and FCTH have satisfactorily retrieval result on our image collection and require noticeably lower computational power and storage space.(as Figure 1) 800 600 400 200 0 547 603 519
530 329 352 167 120
265 num_rel_ret Also, we assessed performance of different similarity function on Compact Composite features and comprehended Euclidean distance on CEDD and FCTH features and Cosine distance function on SPCD and BTDH produce the best performance(as illustrated in Figure2). Then we calculate the similarity between query and dataset objects using Euclidean distance in matching phase. Then we sorted all of dataset images in a descending list based on the value of similarity score in corresponding to each query example image. 516 319 568 525 499 305 568 521 476 291
441 481 443 297
CEDD FCTH SPCD BTDH
The idea behind relevance feedback is to take the results that are initially returned from a given query and to use information about whether or not those results are relevant to perform a new query. Firstly we examined blind or Pseudo relevance feedback on ImageCLEFmed 2011 data collection and assumed that the top 3 ranked documents were most relevant then considered them as query image for each topic but found that this approach could not increase the performance of system. Therefore we performed Explicit Graded relevance feedback to indicate the most relevant images retrieved from a typical query based on assessment of non-professional person. Then we employ those images as query image of the topics.
The database includes over 300,000 images of the biomedical open access literature from journals of BioMed Central at the PubMed Central database associated with their original articles in the journals. Therefore, there are many irrelative generic biomedical illustrations that make the collection so noisy. This noisy documents influence on reduction of system precision. To solve this problem, we filter out such documents using classification decision tree methods and narrowing data collection down to about 35000 images.
In order to assess the above mentioned methods, we set up a set of experiments on the data collection of ImageCLEF 2012 Medical retrieval. We submitted 24 runs to ImageCLEF Ad-hoc Medical Retrieval in five categories as following:
This category includes the runs for baseline retrieval in textual modality on xml metadata, called as DEMIR_R1 to DEMIR_R4. Difference between runs of this category is on tags that are indexed. DEMIR_R1 index just caption of images in original article while DEMIR_R2 indexed full articles correspond to image. Caption of the images and abstract of the corresponded articles were indexed at DEMIR_R3. Title of the article joined to indexed contents in DEMIR_R4. As illustrated in Table1 results shows that caption of images is comprehensive and effective content of metadata to index. On the other hand, although full article has more irrelative content but is more effective than abstract.
As we mentioned in previous sections, CEDD, FCTH and CLD descriptors outperform other low level features. We submitted retrieval result of them as DEMIR_R5, DEMIR_R6 and DEMIR_R7 respectively. Beside these runs, we also investigate on impact of intra-modality combination between these features. We applied our proposed integrated approach on these features in DEMIR_R8 and DEMIR_R9. Also we used feature concatenation method on the synchronous feature vectors of all images in data collection and topics and concatenated them as the joint feature vector in DEMIR_R10 and DEMIR_R11. Results show that the best performance of this category appears in DEMIR_R9, where we combined all of these descriptors based on our method, using CombSUM function.
In this category, we examine our claims in ImageCLEF and confirm them again. We performed integrated weighted CombSUM combination such that coefficient of text modality was 1.7 folds of visual modality. Consequently, system performance arise to the best performance in contrast with other fusion methods in same condition. It is obvious that system performance in DEMIR_R14 and DEMIR_R15 that combined baseline textual and visual runs in integrated approach using weighted sum, are more batter than performance of DEMIR_R12 and DEMIR_R13 respectively. This viewpoint is true also in subject of DEMIR_R17 compared by DEMIR_16. In these later runs, we apply two-level integrated combination; in intra-modality level between low level image features and in inter-modality level between result of pervious level as a visual modality (DEMIR_R9) and baseline run of textual modality (DEMIR_R1).
In this category, we repeated some of submitted runs on new narrowed down and filtered out dataset but any of these new versions of runs did not improve the performance of total system. We submitted this category as DEMIR_R18 till DEMIR_R21.
To submit runs of this class, firstly we chose three most relevance document from retrieved document set of each base run , if it was existence in top 50 retrieved document set. Then these chosen images were joint to image query set of each topic then run retrieval system again to obtained new result. In our experiments, selection of most relevant document is based on only textual, only visual and both of visual and textual information respectively in DEMIR_R22, DEMIR_R23, DEMIR_R24. Although this manner decrease the performance of system when we consider at textual properties for selection of relevant documents, but it improve the effectiveness of system in visual modality so that DEMIR_R23 could obtain the best rank of visual runs. Textual Textual R1 R6 R9 R17 R1 R9 R17 0.2773
0.3 0,0591 In this year, we examined the impact of our integrated combination method on different level of multimodality retrieval system and found that proper combination model can improve the performance of multimodal retrieval systems. In order to investigate on possible combinations methods, we first do some evaluations to determine the appropriate low-level features and distance functions. Then we presented a two level integrated combination method for multimodal CBIR systems. We also perform a visual explicit relevance feedback method and examine its effects on visual, textual and mixed runs. Meanwhile we narrowed down data collection by filtering irrelative generic biomedical illustrations out of data collection using visual low level features.
Our finding illustrated that: Explicit relevance feedback based on visual similarity of image improves the performance of medical image retrieval system.
Elimination of noisy and irrelevant images from dataset can progress effectiveness and efficiency of medical image retrieval system. It is clear that integrated combination multimodal retrieval improves the performance of result system in all combination level, regardless of inter or intra-modality level.
In the best combination of textual and visual modalities, weight for textual modality is about 1.7 folds of visual modality weight.
Our study can be extended in several ways, in future. First, it would be good to apply this experimentation results to other medical image collection and verify that our findings produces the similar results on similar CBIR systems. Second, the impact of combination functions on system performance can also be further investigated. Lastly, our study can be extended into other domain of CBIR Systems rather than medical domain. 7 Analysis for Multimedia Interactive Services (WIAMIS), IEEE Computer Society,. pp. 191-196.