This article describes IPL's participation to the image CLEF ad-hoc textual and visual medical retrieval for 2011. We report on our approaches and methods from a systematic experimental investigation on fusion from visual and textual sources of images. We also explore ways to enrich our searches using external sources like Wikipedia. In this article we give an overview of the application of our methods to ad-hoc medical image retrieval and present the results of our submitted runs. The goal of the CLEF medical retrieval task is to advance the performance of multimedia objects retrieval in the medical domain combining techniques from Information Retrieval and Content Based Image Retrieval (CBIR). Results so far, show that textual retrieval of images outperforms retrieval based only on low level features [7] [8]. This was also the case in the results reported by the CLEF campaign in the medical retrieval task this year. Thus, techniques of merging the retrieval results of multimedia objects out of multisources, remains an interesting and hot research topic. Most of our e orts this year were concentrated on data fusion techniques of di erent low level visual and textual features. To achieve our goal we combine techniques from Information Retrieval, CBIR and Natural Language Processing. In all our submitted runs, the values of the fusion parameters were estimated based on analysis with the CLEF 2009 and 2010 collections [9], [1]. We also explore ways of enriching the textual queries applying relevance feedback on articles derived from the medical category of Wikipedia. In the following sections we overview our approaches and present the results of our runs. Finally conclusions are drawn with proposals for further work.
This year's collection contains a subset of PubMed Central of 231,000 images
from various online magazines and journals [
In the case of multi eld retrieval weights are assigned to each eld at indexing time. Since this year's database has the same structure as the 2009 and 2010 databases we have used the same weights.
The topics are also in XML format. Each topic is identi ed by a unique number and is characterized as visual, semantic or mixed. Visual topics may contain more than one image. (1) (2) 2.2
In our data fusion strategy we used a linear function de ned by: W eightedSU M (q; d) =
X wiScorei(q; d) i where wi is a weight proportional to the performance of the i-th retrieval component. In the case of CBIR since di erent retrieval systems generate di erent ranges of similarity scores, it is necessary to normalize these scores by the formulae (2).
N ormScorei =
Scorei M axScore
M inScore
M inScore All variables in (2) are related to a given query, q, and a given result list. MaxScore and MinScore are the maximum and minimum scores in the result list, respectively; Scorei is the score that a document (image), d, obtained initially; and N ormScorei is the normalized score that d should obtain.
For our runs we have used two di erent sets of values for the weights wi:
{ A set of values estimated empirically from the CLEF '09 and '10 collections.
{ and a set of values wi = M APi. The weight of each system is determined as
a function of its performance [
A third set of values was also investigated based on a machine learning
technique. Although no runs were submited with those values we present brie y the
approach and results are given in section 4.2. These weights are estimated by
training a linear classi er. For training set was served the 2010 CLEF database
with the 2010 topics and the set of Qrels. A training example is de ned by the
normalized tuple of scores (score1; :::; scorek) together with the label of the
retrieved document: +1 if it is relevant and -1 if it is nonrelevant. A value scorei
denotes the score of the image with respect to a eld for the textual case or to a
low level feature in the case of the visual retrieval. We have used a Modi ed
Perceptron linear classi er proposed in [
In medical articles we encounter a large number of compound words, i.e. words that contain two or more single words. In the 2010 CLEF campaign we have noticed a signi cantly low performance in queries containing compound words. For example, the query "images of dermato broma" returns zero documents since the word "dermato broma" does not occur in the text collection (the words "images" and "of" are stopwords in our system). If we split the compound word in its constituent words (derma and broma) the system retrieves 10 relevant documents out of 29 in total.
There are two data sources which we could take advantage of in order to modify our analyzer to deal with compound words: The UMLS5 (Uni ed Medical Language System), a set of les that brings together many health and biomedical vocabularies and the Merriam Websters Medical Dictionary 6. We have used these two sources together with the Lucene's API, DictionaryCompoundWordTokenFilter7 to split up the compound words. However, in order for this class to work, one should provide a dictionary of simple words. The two language resources were used to construct such a dictionary. Initially we construct a table of medical terms from the UMLS meta-thesauruses. Then we proceed constructing for each term in the table all the n-grams of size greater or equal to 4. From these, we remove the n-grams with frequency less than 5. For the remaining n-grams we are assisted by the online dictionary Merriam Webster (MW), which has a medical subdomain. We have developed a script which for each candidate morpheme of the table, looks it up in the MW dictionary by using a simple HTTP call. It parses the response text and decides whether or not it is a medical term. In the case of a medical term it stores the primary form of its entry. Thus from all the n-grams we keep in our dictionary only those which exist in the MW dictionary in terms of medical lemmas or morphemes in combining form. For example, the word dermato broma is indexed as dermat, br, dermato broma. 2.4
Usually the initial query re ects the user's rst attempt in the process of
information seeking. The initial query, however, might be more generic or more
speci c than it should be. As a result, the user should modify his original query
to get better results. In our implementation we have applied a pseudo-relevance
feedback on the top k retrieved documents using Rocchio's algorithm [
Wikipedia is a free, web-based, collaborative, multilingual encyclopedia project and provides for public use the latest dump of its entire database. This huge amount of data can be very useful for several tasks in Information Retrieval and Natural Language Processing. In our runs we applied pseudo-relevance feedback using the Wikipedia subcollection on the medical domain to expand our initial query. Thus, the initial query is submitted to wikipedia database and we keep the top 5 documents retrieved for further analysis using Rocchio formula. Finally we select the two top terms, those with the highest weights (other than the terms in the initial query) to form the re ned query.
For CBIR we have used LIRE (Lucene Image Retrieval)8, a light weight open
source Java library [
In table 1 we give the de nitions of our textual runs. The last part of the run's identi er denotes the similarity function used. As we have already mentioned in section 2.2 we have used two types of weights all derived from the analysis in the CLEF '09 and '10 databases.
From our textual results, in Table 2 it is evident that the weighted multi eld retrieval outperforms retrieval in a single eld. Another interesting observation was that BM25 and BM25F were worst than the default similarity function in all the cases.
The weights were estimated by the MAP values from each individual feature in the 2009 database. The methods Default, Extensive and Fast de ne di erent combinations of color, texture and edge histograms.
For multi-image queries Q = fim1; :::; imkg the similarity score of an image, Image, was estimated by: k SCORE(Q; Image) = X score(imj; Image) j=1 (3) 4.3
For mixed retrieval we have used two di erent approaches. One was with a linear combination of the textual and visual results, de ned by:
SCORE(q; d) = 0:39 scoretextual(q; d) + 0:01 scorevisual(q; d) (4) where 0.39 was the MAP value of the textual retrieval and 0.01 the MAP of the visual retrieval on the CLEF 2009 database. The results from this approach are listed in the rst four lines in table 4. The second fusion approach was a ltering task of CBIR on a set of images retrieved from a textual query. The top 1000 documents retrieved from the textual queries are used in the CBIR. These documents are re-ranked according to their content based score. The results from this approach are listed in the last four lines in table 4. Finally in Table 5 we present our results based on the machine learning approach presented in section 2.2 from textual and visual retrieval on the databases over the past three years. Although no o cial runs were submitted from this approach results are promising and therefore we have included them in this working paper. 5
Retrieving documents from di erent sources surely improves performance of information retrieval. However, e cient balancing the e ect of each source to the nal result needs a deeper investigation. From our experiment studies, not submitted o cially this year seems that the values estimated by the machine learning technique are very promising. Further investigation, however is needed, to nd out whether the training data remain linearly separable when the number of queries for training the classi er is getting larger. Another positive feature of this approach is that users may save their own queries (with positive answers) obtaining in this way a personalized classi er that will better t to their needs.
In the medical domain, dealing with the compound words is very crucial. The medical dictionary of word lemmas and morphemes we have constructed is a positive contribution to that direction. Wikipedia is a very rich resource of data and can be used for query expansion in the medical domain.
The performance of the visual retrieval still is very poor compared to the textual retrieval. It seems that global features of the images do not have a good discrimination value. Thus techniques for image segmentation using local features may improve CBIR while keeping the complexity to acceptable levels. Finally the restriction of the CBIR retrieval to the top documents (say 1000) returned by a textual query makes CBIR scalable to large image collections.