This article describes the participation of the Image and Text Integration (ITI) group in the 2012 ImageCLEf medical retrieval and classi cation tasks. We present our methods for each of the three tasks and discuss our submitted textual, visual, and mixed runs as well as their results. While our methods generally perform well for each task, our best ad-hoc image retrieval submission was ranked rst among all the submissions from the participating groups.
This article describes the participation of the Image and Text Integration (ITI) group in the ImageCLEF 2012 medical retrieval and classi cation tasks. Our group is from the Communications Engineering Branch of the Lister Hill National Center for Biomedical Communications, which is a division of the U. S. National Library of Medicine.
The medical track [
In the following sections, we describe the textual and visual features that
comprise our image and article representations (Sections 2{3) and our methods
for the modality classi cation (Section 4) and medical retrieval tasks (Sections
5{6). Our textual approaches primarily utilize the Uni ed Medical Language
System R (UMLS R ) [
In Section 7 we describe our submitted runs, and in Section 8 we present our results. For the modality classi cation task, our best submission achieved a classi cation accuracy of 63:2% and was ranked within the submissions from the top three participating groups. Our best submission for the ad-hoc image retrieval task was ranked rst overall, achieving a mean average precision of 0.2377, which is a statistically signi cant improvement over the second ranked submission. For the case-based article retrieval task, our best submission achieved a mean average precision of 0.1035 and was ranked within the submissions from the top four participating groups, but this submission is statistically indistinguishable from our other case-based submissions. 2
We represent the images contained in biomedical articles using a combination of the textual and visual features described below.
We represent each image in the collection as a structured document of imagerelated text called an enriched citation. Our representation includes the title, abstract, and MeSH R terms1 of the article in which the image appears as well as the image's caption and \mentions" (snippets of text within the body of an article that discuss an image). These features can be indexed with a traditional text-based information retrieval system, or they may be exposed as term vectors and combined with the visual feature vectors described below.
In addition to the above textual features, we also represent the visual content of images using various low-level visual descriptors. Table 1 summarizes the descriptors we extract and their dimensionality. Due to the large number of these features, we forego describing them in any detail. However, they are all well-known and discussed extensively in existing literature.
Cluster Words. To avoid the computational complexity of computing distances
between the above visual descriptors, we create a textual representation of visual
features that is easily integrated with our existing textual features. For each
visual descriptor listed in Table 1, we cluster the vectors assigned to all images
using the k-means++ [
Article Representation for Case-based Retrieval We represent articles using the textual features of each image appearing in the article. Thus, each article enriched citation consists of its title, abstract, and MeSH terms as well as the caption and mention of each contained image. 4
We experimented with both at and hierarchical modality classi cation methods. Below we describe our at classi cation strategy, an extension of this approach that exploits the hierarchical structure of the classes, and a post-processing method for improving the classi cation accuracy of illustrations.
Figure 1a provides an overview of our basic classi cation approach. We utilize multi-class support vector machines (SVMs) as our at modality classi ers. First, we extract our visual and textual image features from the training images (representing the textual features as term vectors). Then, we perform attribute selection to reduce the dimensionality of the features. We construct the lowerdimensional vectors independently for each feature type (textual or visual) and combine the resulting attributes into a single, compound vector. Finally, we use the lower-dimensional feature vectors to train multi-class SVMs for producing textual, visual, or mixed modality predictions.
Unlike the at classi cation strategy described above, it is possible to exploit the
hierarchical organization of the modality classes in order to decompose the task
into several smaller classi cation problems that can be sequentially applied. Based
on our visual observation of the training samples and our initial experiments, we
modi ed the original modality hierarchy [
We train at multi-class SVMs, as shown in Figure 1a, for each meta-class. For
recognizing compound images, we utilize the algorithm proposed by Apostilova et
al. [
Because our initial classi cation experiments resulted in only modest accuracy for the fourteen \Illustration" classes shown in Figure 1b, we concluded that our current textual and visual features may not be su cient for representing these gures. Therefore, in addition to the aforementioned machine learning modality classi cation methods, we also developed several complimentary rulebased strategies for increasing the classi cation accuracy of \Illustration" classes.
A majority of the training samples contained in the \Illustration" meta-class,
unlike other images in the collection, consist of line drawings or text superimposed
on a white background. For example, program listings mostly consist of text;
thus, the use of text and line detection methods may increase the classi cation
accuracy of Class GPLI. Similarly, polygons (e.g., rectangles, hexagons, etc.)
contained in owcharts (GFLO), tables (GTAB), system overviews (GSYS), and
chemical structures (GCHE) are a distinctive feature of these modalities. We
utilize the methods of Jung et al. [
In this section we describe our textual, visual and mixed approaches to the ad-hoc image retrieval task. Descriptions of the submitted runs that utilize these methods are presented in Section 7.
To allow for e cient retrieval and to compare their relative performance, we
index our enriched citations with the Essie [
We organize each topic description into a frame-based (e.g., PICO4)
representation following the method similar to that described by Demner-Fushman
and Lin [
Our visual approaches to image retrieval are based on retrieving images that appear visually similar to the given topic images. We compute the visual similarity between two images as the Euclidean distance between their visual descriptors. For the purposes of computing this distance, we represent each image as a combined feature vector composed of a subset of the visual descriptors listed in Table 1 after attribute selection.
We explore several methods of combing our textual and visual approaches. One such approach involves the use of our image cluster words. For performing multimodal retrieval using cluster words, we rst extract the visual descriptors listed in Table 1 from each example image of a given topic. We then locate the clusters to which the extracted descriptors are nearest in order to determine their corresponding cluster words. Finally, we combine these cluster words with words taken from the topic description to form a multimodal query appropriate for either Essie or Lucene/SOLR.
While the use of cluster words allows us to create multimodal queries, we can instead directly combine the independent outputs of our textual and visual approaches. In a score merging approach, we apply a min-max normalization to the ranked lists of scores produced by our textual and visual retrieval strategies. We then linearly combine the normalized scores given to each image to produce a nal ranking. Similarly, a rank merging approach combines the results of our textual and visual approaches using the ranks of the retrieved images instead of their normalized scores. To produce the nal image ranking using this strategy, we re-score each retrieved image as the reciprocal of its rank and then repeat the above procedure for combining scores.
4 PICO is a mnemonic for structuring clinical questions in evidence-based practice and represents Patient/Population/Problem, Intervention, Comparison, and Outcome.
Another means of incorporating visual information with our retrieval approaches is through the use of a modality classi er. Using our hierarchical modality classi cation approach, we can rst determine the most probable modalities for a topic's example images. After retrieving a set of images using either our textual or visual methods, we can eliminate retrieved images that are not of the same modality as the topic images. An advantage of performing hierarchical classi cation is that we can lter the retrieved results using the meta-classes within the hierarchy (e.g., \Radiology").
Finally, we often combine the retrieval results produced by several queries into a single ranked list of images. We perform this query combination, or padding, by simply appending the ranked list of images retrieved by a subsequent query to the end of the ranked list produced by the preceding query. 6
Our method for performing case-based retrieval is analogous to our textual approaches for ad-hoc image retrieval. Here, we index the enriched citations described in Section 3 using the Essie and Lucene/SOLR search engines (for performance comparison). We generate textual and mixed queries appropriate for both search engines according to the approaches described in Sections 5.1.
As a form of query expansion for case-based topics, we also explore the
possibility of determining relevant disease names to correspond with signs and
symptoms found in a topic case. To determine a set of potential diseases, we rst
use the Google Search API5 to search the World Wide Web using a topic case as
a query. We then process the top ve documents with MetaMap [
In this section we describe each of our submitted runs for the modality classi cation, ad-hoc image retrieval, and case-based article retrieval tasks. Each run is identi ed by its submission le name or trec_eval run ID and mode (textual, visual or mixed). All submitted runs are automatic.
We submitted the following nine runs for the modality classi cation task: M1. Visual only Flat.txt (visual): A at multi-class SVM classi cation using selected attributes from a combined visual descriptor of 15 features (all descriptors in Table 1 except LCH and SCONCEPT).
M2. Visual only Hierarchy.txt (visual): Like Run M1 but classi cation is performed hierarchically.
M3. Text only Flat.txt (textual): A at multi-class SVM classi cation using selected attributes from a combined term vector created from four textual features (article title, MeSH terms, and image caption and mention). M4. Text only Hierarchy.txt (textual): Like Run M3 but classi cation is performed hierarchically. 5 https://developers.google.com/custom-search/v1/overview M5. Visual Text Flat.txt (mixed): A at multi-class SVM classi cation combining the feature representations used in Runs M1{3.
M6. Visual Text Hierarchy.txt (mixed): Like Run M5 but classi cation is performed hierarchically.
M7. Visual Text Flat w Postprocessing 4 Illustration.txt (mixed): Like Run M5 but additional post-processing is applied for \Illustration" classes. M8. Visual Text Hierarchy w Postprocessing 4 Illustration.txt (mixed): Like
Run M7 but classi cation is performed hierarchically.
M9. Image Text Hierarchy Entire set.txt (mixed): Like Run M6 but applied to all the images contained in the retrieval collection. 7.2
We submitted the following ten runs for the ad-hoc image retrieval task: A1. nlm-se (mixed): A combination of three queries using Essie. (A1.Q1) A disjunction of modality terms extracted from the query topic must occur within the caption or mention elds of an image's enriched citation; a disjunction of the remaining terms is allowed to occur in any eld. (A1.Q2) A lossy expansion of the verbatim topic is allowed to occur in any eld. (A1.Q3) A disjunction of the query images' cluster words must occur within the global image feature eld.
A2. nlm-se-cw-mf (mixed): A combination of Query A1.Q1 with the additional query below using Essie. (A2.Q2) A lossy expansion of the verbatim topic is allowed occur in any eld of an image's enriched citation and a disjunction of the query images' cluster words can optionally occur within the global image feature eld. Additionally, the retrieved images are ltered so that they share a least common ancestor modality with the query images, as determined by the modality classi er used in Run M9. Query A2.Q2 is distinct from Queries 1.Q2{3 in that the occurrence of a lossy expansion of the topic is not necessarily weighted more heavily than the occurrence of image cluster words.
A3. nlm-se-scw-mf (mixed): Like Run A2 but image cluster words are only considered if the modality classi er used in Run M9 identically labels all the example images of a topic.
A4. nlm-lc (mixed): A combination of three queries using Lucene with BM25 similarity and UMLS synonymy. (A4.Q1) A fuzzy phrase-based occurrence of the verbatim topic is allowed in any eld of an image's enriched citation. (A4.Q2) A disjunction of the topic words is allowed to occur in any eld. (A4.Q3) A disjunction of the query images' cluster words must occur within the global image feature eld.
A5. nlm-lc-cw-mf (mixed): A combination of Query A4.Q1 with the additional query below using Lucene with BM25 similarity and UMLS synonymy. (A5.Q2) A disjunction of the topic words is allowed occur in any eld of an image's enriched citation and a disjunction of the query images' cluster words can optionally occur within the global image feature eld. Additionally, the retrieved images are ltered so that they share a least common ancestor modality with the query images, as determined by the modality classi er used in Run M9.
A6. nlm-lc-scw-mf (mixed): Like Run A5 but image cluster words are only considered if the modality classi er used in Run M9 identically labels all the example images of a topic.
A7. Combined Selected Fileterd Merge (visual): Similarity matching using 62 min-max normalized attributes selected from a combined visual descriptor of 15 features (all descriptors in Table 1 except LCH and SCONCEPT). Retrieval is performed separately for each query image, and the retrieved results are ltered, according to the modality classi er used in Run M9, so that they share the top two modality levels with the query. Images are scored according the query image resulting in the maximum score. A8. Combined LateFusion Fileterd Merge (visual): Like Run A8 but similarity matching is performed separately for seven features (CLD, GLCM, SCONCEPT, and the color, Gabor, shape, and Tamura moments from Table 1) whose scores are linearly combined with prede ned weights.
A9. Txt Img Wighted Merge (mixed): A combination of visual Run A7 with a textual run consisting solely of Query A1.Q2 using score merging. A10. Merge RankToScore weighted (mixed): A combination of visual Run A8 with a textual run consisting solely of Query A1.Q2 using rank mering. 7.3
We submitted the following eight runs for the case-based article retrieval task: C1. nlm-se-max (textual): A combination of three queries for each topic sentence using Essie. (C1.Q1) A disjunction of modality terms extracted from the sentence must occur within the caption or mention elds of an article's enriched citation; a disjunction of the remaining terms is allowed to occur in any eld. (C1.Q2) A lossy expansion of the verbatim sentence is allowed to occur in any eld. (C1.Q3) A disjunction of all extracted words and discovered diseases in the sentence is allowed to occur in any eld. Articles are scored according to the sentence resulting in the maximum score. C2. nlm-se-sum (textual): Like Run C1 but articles are scored according to the sum of the scores produced for each sentence.
C3. nlm-se-frames-max (textual): A combination of the query below with Query C1.Q2 for each topic sentence using Essie. (C3.Q1) An expansion of the frame-based representation of the sentence is allowed to occur in any eld of an article's enriched citation. Articles are scored according to the sentence resulting in the maximum score.
C4. nlm-se-frames-sum (textual): Like Run C3 but articles are scored according to the sum of the scores produced for each sentence.
C5. nlm-lc-max (textual): A combination of two queries for each topic sentence using Lucene with language model similarity, Jelinek-Mercer smoothing, and UMLS synonymy. (C5.Q1) A fuzzy phrase-based occurrence of the verbatim sentence is allowed in any eld of an article's enriched citation. (C5.Q2) A disjunction of all words and discovered diseases in the sentence is allowed to occur in any eld. Articles are scored according to the sentence resulting in the maximum score.
C6. nlm-lc-sum (textual): Like Run C5 but articles are scored according to the sum of the scores produced for each sentence.
File Name Visual Text Hierarchy w Postprocessing 4 Illustration.txt Visual Text Flat w Postprocessing 4 Illustration.txt Visual Text Hierarchy.txt Visual Text Flat.txt Visual only Hierarchy.txt Visual only Flat.txt Image Text Hierarchy Entire set.txt Text only Hierarchy.txt Text only Flat.txt Mixed Mixed Mixed Mixed Visual Visual Mixed Textual Textual C7. nlm-lc-total-max (textual): A combination of the query below with Queries C5.Q1{2 (as C7.Q2{3) using Lucene with language model similarity, JelinekMercer smoothing, and UMLS synonymy. (C7.Q1) A fuzzy phrase-based occurrence of the entire verbatim topic is allowed in any eld of an article's enriched citation. Articles are scored according to the sentence resulting in the maximum score.
C8. nlm-lc-total-sum (textual): Like Run C7 but articles are scored according to the sum of the scores produced for each sentence. 8
We present and discuss the results of our modality classi cation, ad-hoc image retrieval, and case-based article retrieval task submissions below.
Table 2 presents the classi cation accuracy of our submitted runs for the modality
classi cation task. Visual Text Hierarchy w Postprocessing 4 Illustration.txt, a
mixed approach, achieved the highest accuracy (63.2%) of our submitted runs
and was ranked fth overall. However, it ranked within the submissions from the
top three participating groups. This result validates our post-processing method
used to improve the recognition of \Illustration" classes, and provides, with our
previous experience [
While our submitted runs were only judged on their ability to identify each of
the thirty-one modality classes [
Table 4 presents the mean average precision (MAP), binary preference (bpref),
and early precision (P@10) of our submitted runs for the ad-hoc image retrieval
task. nlm-se achieved the highest MAP (0.2377) among our submitted runs and
was ranked rst overall. Merge RankToScore weighted, the run achieving our
second highest MAP (0.2166), was ranked second overall. Comparing these two
runs using Fisher's paired randomization test [
That the two highest ranked runs were multimodal, as apposed to textual, is an encouraging result, and provides evidence that our ongoing e orts at integrating textual and visual information will be successful. In particular, the use by nlm-se and other runs of cluster words, which are indexed and retrieved using a traditional text-based information retrieval system, is an e ective way, not only of incorporating visual information with text, but of avoiding the computational expense common among content-based retrieval methods. Furthermore, Merge RankToScore weighted demonstrates the value of rank merging when combining textual and visual retrieval results. Some of our other mixed runs, in utilizing the results of our modality classi ers, may have been weakened due to the modest performance of our classi cation methods.
Table 5 presents the MAP, bpref, and P@10 of our submitted runs for the casebased article retrieval task. nlm-lc-total-sum, a textual approach using language model similarity, achieved the highest MAP (0.1035) among our submitted runs and was ranked seventh overall. However, it ranked within the submissions from the top four participating groups. Using Fisher's paired randomization test, we nd that there is no statistically signi cant di erence (p < 0:05) in MAP among any of our submitted runs. The relatively low performance of most of the ImageCLEF 2012 case-based submissions may be due, in part, to the existence in the collection of only a small number of case reports, clinical trials, or other types of documents relevant for case-based topics. 9
This article describes the methods and results of the Image and Text Integration (ITI) group in the ImageCLEF 2012 medical retrieval and classi cation tasks. For the modality classi cation task, our best submission was ranked within the submissions from the top three participating groups. Our best submission for the ad-hoc image retrieval task was ranked rst overall. Finally, for the case-based article retrieval task, our best submission was ranked within the submissions from the top four participating groups, though we found no statistical signi cance between this run and our other case-based submissions. The e ectiveness of our multimodal approaches are encouraging and provide evidence that our ongoing e orts at integrating textual and visual information will be successful. Acknowledgments. We would like to thank Antonio Jimeno-Yepes for assisting in expanding case-based topics with disease names, Russell Loane for proving source code for converting frame-based topics to Essie queries, and Srinivas Phadnis for constructing enriched citations and extracting visual features.