The main goal of this paper is to present our experiments in the classification modality and in the ad-hoc image retrieval tasks with the Medical collection at ImageCLEF 2012 Campaign. This edition we focus on applying new strategies for both the textual and the visual subsystems included in our multimodal retrieval system. The visual subsystem has focus on extending the low-level features vector with concept features. These concept features have been calculated by means of a logistic regression model. The textual subsystem has focus on applying a query reformulation to remove general and domain stop-words, trying to produce a query with only medical-related terms. We have not obtained the results as good as obtained at the Photo annotation retrieval subtask using similar techniques. Therefore, a deep analysis for the Medical collection will be done.
In this paper we present our experiments in ImageCLEF 2011 Campaign at Medical
campaign, we participate in two sub-tasks of the Medical Retrieval Tasks: Image
Modality Classification and Ad-hoc Image Retrieval. The work done in this edition is
building on the knowledge acquired in previous participations both at Medical
Retrieval Task [
Regarding the textual retrieval subsystem, we apply partially the successful
technique tested last year [
Concerning to the visual retrieval subsystem it uses the low-level features for
image retrieval. This low-level information although gives quite enough results
depending on the visual information of the query is not able to reduce the “semantic gap” in a
semantic complex query. Our proposal [
It is our first participation at the classification task with four visual runs submitted. We have adapted our regression model to act as a classifier for the classification task. A model for each of the categories have been trained and tested.
Section 2 describes the visual approach based on a regression model acting as a classifier for the modality classification subtask. Section 3 explains our multimodal retrieval system use for the ad-hoc image-based retrieval subtask. After that section 4 shows the submitted runs and the results obtained for modality classification and retrieval. Finally, in section 5 we extract conclusions and outlines possible future research lines. 2
We train a logistic regression model [
The medical classification task gives to the participants a training set, , for each of the categories. Being the training image set for each category (the relevant images), and the set that not belong to a certain category (non relevant images). The logistic regression analysis calculates the probability for a given image to belong to a certain category. Each image of the training set, is represented by a K-dimensional low-level features vector { }. The relevance probability for a certain category for a given image will be represented as ( ). A logistic regression model can estimate these probabilities. Let us consider for a binary Y, and k explanatory variables , the model for (x) = P(Y=1 X ) (probability ) for the x values [ ] , where logit ((x))=ln((x) / (1-(x)). The model parameters are obtained by maximizing the likelihood estimator (MLE) of the parameter vector β by using an iterative method.
We have a major difficulty when having to adjust an overall regression model in which we take the whole set of variables into account because the number of selected images (the number of positive plus negative images, k) is typically smaller than the number of characteristics (k < p). In this case the adjusted regression model has as many parameters as the amount of data and many relevant variables could be not considered. In order to solve this problem our proposal is to adjust different smaller regression models: each model considers only a subset of variables consisting of semantically related characteristics of the image. Consequently each sub-model will associate a different relevance probability to a given image x and we have to combine them in order to rank the database according to the image probability or image score (Si).
The explanatory variables to train the model are the visual
lowlevel features based on color and texture information that are calculated by our group.
We have a low-level features vector of 293 components divided by five different
visual information families.
Color information: We calculate global and local histograms of the image.
─ Global color: It is a feature vector of 30 components represents the color
information of the complete image. Each of these components represents a bin on a
HS (hue-saturation) histogram of size 10 x 3.
─ Local color: Local histograms have been calculated by dividing the images into
four fragments of the same size. A bi-dimensional HS histogram with 12x4 bins
is computed for each patch, being 48 components for each patch, and a total of
192 components.
Texture information: Two types of texture feature are computed:
─ The granulometric distribution functions [
The overall system includes three main subsystems: the TBIR (Text-Based Image Retrieval), the CBIR (Content-Based Image Retrieval), and the Fusion subsystem (see Fig. 2). Both the textual (TBIR) and the visual subsystem (CBIR) obtain a ranked list of images based on similarity scores (St and Si) for a given query. Firstly, TBIR uses the textual information from the annotations (metadata and articles) to obtain these scores (St). This textual pre-filtered list is then used by the CBIR sub-system. It extracts the visual information from the given example images of the query and generates a similarity score (Si). The fusion sub-system is in charge of merging these two lists of results, taking into account the scores and rankings, in order to obtain the final result list. 3.1.1
This module is in charge of the textual-based indexing and retrieval, using the text associated with each image in the collection.
In order to be able to manage the textual information of the collection, a preprocessing step is carried out, before of the indexing. Later, it has been carried out the indexing of the images for their subsequent retrieval. To indexing the collection, Solr1, a search platform from Lucene2 project, is used. The retrieval process is done through Solr too. The result of this retrieval process is a normalized image list for each query. Below, is explained in more detail each of the different stages performed by TBIR module:
Collection Articles
Text Topics
Images Collection Images
Query Reformulation TBIR CBIR Feature Extraction
Preprocess
Image Text Queries
Index
Search Expanded Conceptual Vector
Similarity Module: Logistic Regression Relevance feedback
IMG Results (Si)
Txt Results (St)
St*Si
FUSION
Txt_Img
Results
Query Reformulation: The original queries are reformulated in order to remove
common and domain stopwords (e.g: image). No other process is done.
Preprocess: Textual information (both at images description and queries
description) is preprocessed : 1) special characters deletion: characters with no statistical
meaning, like punctuation marks or blanks, are eliminated; 2) stopwords detection:
deletion of semantic empty words in English language (e.g: the, an…), 3)
stemming: reduction of word to their base form, for this purpose we use a Porter
Algorithm implementation provided by Solr and, finally, 4) convert all words to lower
case.
Indexing: Because the collection of this edition is different from the last edition, it
was necessary to index the new collection. The indexing is done automatically by
Solr, using Lucene operation.
Searching: The search process is also automatically done by Solr over Lucene
operation. The score function used for calculating the similarity between a given
query and the documents is BM25. The results are transformed to the TRECEval
format, in order to merge these textual results with visual results and check the
results using the UV tool [
The work of the CBIR subsystem is based on three main stages: Extraction of the low-level and the concept features of the images, and the calculation of the similarity (Si) of each of the images to the image examples given by a query. 1. Extraction of low-level features: The first step in the CBIR system is to extract the visual low-level and the concept features for all the images of the database as well as from the example images given in each question. The low-level features we use are calculated by our group and give color and texture information about the images. These features are the same that we have used for the modality classification task (see section 2.1 for more detailed information). 2. Calculating the Concept features vector: The regression models trained for each of the concepts gives for each image on the database and for the example query the probability of the presence of each concept ( ). With this probability information for each concept, we extend the low-level features vector to m components, being m the number of concepts trained. Each image on the database is described by the extended vector ( ) } . 3. Similarity Module: The similarity module instead of using the classical distance method to calculate the similarity of each of the images of the database to the example images for a given topic uses our own logistic regression relevance algorithm to get the probability of an image belonging to the query set. The sub-models regressions are set to five features inside each features family that are the number of example images given for each topic (see more details of the regression method at section 2.1.). The relevant images are the example images, and the non-relevant images are randomly taken from outside the pre-textual filtered list. 3.1.3
The fusion subsystem is in charge of merging the two score result lists from the TBIR and the CBIR subsystem. In the present work we use the product fusion algorithm (Si*St). The two results lists are fused together to combine the relevance scores of both textual and visually retrieved images (St and Si). Both subsystems will have the same importance for the resulting list: the final relevance of the images will be calculated using the product. 4 4.1
In this our first participation on the medical modality classification subtask, we have only participated with visual modality runs. Our objective for this edition has been to test the behavior of our logistic regression model for the classification task, and to adjust the parameters for the regression model explained at the section 2. The parameters to be defined to model each of the categories are: The automatic election for the relevant and non-relevant images for the model to train each of the categories (positive and negative images).
The organization gives a training set for each of the categories being these training sets the relevant images for our logistic regression model. The number of images for each category differs from 5 images at the lower range (DSEC and DSEM categories) to 49 images at the highest range (COMP, DRCT and GGEL categories). The non-relevant images are the nearest N image to the centroid images of the set of images of the other categories different to the one being trained. The number of non-relevant images will be the double of the number of relevant images. We present two approaches for the number of relevant images to be used: for the first approach all available images for each given category are taken as relevant images (runs 1, 3 and 4), and for the second approach we limit the number of relevant images to a MAX number of images. The MAX number chosen is 30 because is the average low-level features components for each visual information family (run 2). The different subgroups to adjust smaller regression models.
As it has been explained above the number of positive plus negative images, k (5 + 5*2 for the minimum set of training image category is smaller than the number of characteristics p (292 low-level featured vector) (k < p). We present four different approaches to group the low-level features: a regression model for each family low-level vector (run1), a regression model for each 30 components (run2) being 30 images the number of relevant images, a regression model for the lowest number of relevant images given that for this collection is 5 images (run3), and an adaptive regression model strategy different for each category depending on the minimum number of given relevant images or to the minimum number of components for the low-level featured family vector (run4). For all runs, the different submodels are merged by the average function. [30] [31] [31] [31] [31] [31] [31] The multimodal experiments (runs 2 to 9) have been designed to test the behavior of the expanded concept features vector. The runs marked as visual at the modality column at Table 2 use only the visual score, Si, to re-rank the final list. Meanwhile, those marked as Mixed use both textual and visual score to re-rank the final list by the product, St*Si. The third column shows which features vector has been used by the CBIR system to obtain the visual score, Si, with the following codes meaning: [LF], uses only the low-level features vector } ;[CF], using only the concept/category features for visual information } ; [LF…CF], uses the extended concept vector } as a unique vector; and finally, [LF]*[CF], uses the extended concept vector as two different vectors obtaining two probabilities, for the low-level features vector, and for the concept vector that are merged by the product
The textual retrieval approach we have proposed this time, based on a query
reformatted process, which focuses on the semantic of the queries by try to use only
medical terms, has not obtained the expected results. We will analyze this bad
performance of the textual retrieval process at the Medical collection, given that this
technique was successfully tested last year at the Wikipedia collection [
For the multimodal approaches presented for the ad-hoc image-based retrieval
subtask, our combination of the textual pre-filtered list as input to the visual system does
not outperform the textual baseline, as it has already been tested in other ImageClef
collections, Wikipedia [
The results obtained at the classification modality subtask suffered from the fact that our visual approach is a retrieval approach adapted for the classification modality task. Nevertheless, the regression model system proposed as a modality classifier will be analyzed query-by-query to improve its classification performance.
Acknowledgments. This work has been partially supported for Regional Government of Madrid under Research Network MA2VIRMR (S2009/TIC-1542), for Spanish Government by project BUSCAMEDIA (CEN-20091026) and by project MCYT TEC2009-12980. 6