We present results from Oregon Health & Science University's participation in the medical image retrieval task of ImageCLEF 2008. We created a web-based retrieval system built on a full-text index of the annotations using a Ruby on Rails framework. The text-based search engine was implemented in Ruby using Ferret, a port of Lucene. In addition to this textual index of annotations, supervised machine learning techniques using visual features were used to classify the images based on image acquisition modality. All images were annotated with the purported modality. Our system provides the user with a number of search options including those for limiting the search to the desired modality, UMLS-based term expansion and Natural Language Processing based techniques. Purely textual runs as well as mixed runs using the purported modality were submitted. We also submitted interactive runs using a number of user specified search options. Latent semantic analysis of the visual features was used to reorder results. The use of the UMLS Metathesaurus increased our recall. However, our system is primarily geared towards precision. Consequently, many of our multimodal automatic runs using the custom parser as well as interactive runs had high early precision. Our runs also performed well using the bpref metric, a measure that is more robust in the case of incomplete judgments.
Image retrieval systems do not currently perform as well as their text counterparts [
Retrieval performance has shown demonstrable improvement by fusing the results of textual and visual
techniques. This has especially been shown to improve early precision [
The ImageCLEF 2008 medical image retrieval test collection consists of about 66,000 medical images and annotations associated with them. This collection is a set of images and captions from Radiology and Radiographics , two RSNA journals. We have created a flexible database schema that allows us to easily incorporate new collections while facilitating retrieval using both text and visual techniques. The captions and titles in the collection are currently indexed and we continue to add indexable fields for incorporating visual information.
The data distribution included an XML file with the image ID, the captions of the images, the titles of the journal articles in which the image had appeared and the PubMed ID of the journal article. In addition, a compressed file containing the approximately 66,000 images was provided.
We used the Ruby programming language, with the open source Ruby On Rails web application framework1, 2. A PostgreSQL relational database was used to store mapping between the images and the various fields associated with the image. The title, full caption and precise caption, as provided in the data distribution, were indexed. The user interface for the search engine is given below in Fig.1.
The image itself has important visual characteristics such as color and texture that can help in the retrieval process. We created additional tables in the database to store image information that was created using a variety of image processing techniques in MATLAB3. These include color and intensity histograms as well as measures of texture using gray-level co-occurrence matrices and discrete cosine transforms8. These features can be used to find images that are visually similar to the query image. We used this in the interactive, mixed mode to reorder the images obtained from the textual search such that images that are visually similar to an image marked relevant by the user are returned at the top of the list.
Images that may have had information about the imaging modality or anatomy or view associated with them
as part of the DICOM header can lose that information when the image is compressed to become a part of a
teaching or on-line collection, as the image format used by these collections is usually compressed JPEG. In
previous work [
The system presents a variety of search options to the user including Boolean OR, AND, and “exact match.”. There are also options to perform fuzzy searches, as well as a custom query parser. A critical aspect of our system is the query parser, written in Ruby. Ferret, a Ruby port of the popular Lucene system, was used in our system as the underlying search engine4. The custom query parser first performs stop word removal using a modified stop word list. The custom query parser is highly configurable, and the user has several configuration options from which to choose. The first such option is modality limitation. If the user selects this option, the query is parsed to extract the desired modality, if available. Using the modality fields described in the previous section, only those images that are of the desired modality are returned. We expect this to improve the precision as only images of the desired modality will be included within the result set. However, there might be a loss in recall if the modality extraction and classification process is not accurate.
The system is linked to the UMLS Metathesaurus. The second configuration option allows the user to perform manual or automatic query expansion using synonyms from the Metathesarus. In the manual mode, a list of synonyms is presented to the user, which the user can choose to add to the query. In the automatic mode, all synonyms of the UMLS preferred term are added to the query.
Another configuration option is the “stem and star” option, in which all the terms in the query are first stemmed. A wildcard (*) is then appended to the word to allow the search of words containing the desired root.
The last option allows the user to only send unique terms to the search engine. This can be useful when using the UMLS option, as many of the synonyms have a lot of overlap in the preferred terms.
In addition to user-selectable search engine configuration options described above, our system provides users with other interactive features. Once a user has submitted a query using the above-described query parser, they have the option to improve the precision of their results by using an interactive re-ordering system. In this year’s 3 http://www.mathworks.com 4 http://ferret.davebalmain.com ! system, users select what they feel to be a visually representative image from their search’s results. The system then attempts to re-rank the search results according to their degree of visual similarity with the “probe image” that the user selected. If the user is not satisfied with the re-ordering produced by their choice if image, they may repeat the process by selecting different probe images until they arrive at a satisfactory sorting.
To assess the visual similarity of the images within a result set, the system uses a relatively straightforward
approach derived from Latent Semantic Analysis [
Once the feature vectors have been assembled for the images in a result set, they are combined into an n " m matrix. In this matrix, n is equal to the number of images in the result set, and m is equal to the number of features that the user has selected. Depending on the combination of features, this could be in the hundreds or low thousands. We then follow the classical LSA process, beginning by taking the Singular Value Decomposition (SVD) of our large matrix. This transforms our single matrix into three matrices that may be trivially recomposed to approximate the original matrix. The elements of one of these matrices represents the eigenvalues of the original document/term matrix; by varying the number of these elements that we use when recomposing the matrices, we may vary the fidelity of the resulting approximation.
After carrying out the SVD, we retain the first r eigenvalues of the decomposed matrix, project the probe image’s m-dimensional feature vector into the new lower-dimensional space, and, finally, compute the vector distance between the probe image’s new representation and that of the images in the result set. In our system, the user is able to experiment with different values for r, and may pick the one that achieves the best performance for a given set of results. The user may also quickly and easily select different images to act as probe images, and can therefore evaluate many possible result sortings.
Obviously, this system’s utility is variable, and depends highly on the contents of the initial result set. In the case of a set where the desired images are simultaneously visually similar to one another and distinct from the rest of the images in the set, this visual re-sorting system works quite well. However, in the case where the desired images are visually different from one another, or where all of the results (including the non-relevant ones) are visually similar, this re-sorting system is not very useful.
For example, a result set consisting entirely of ultrasound images will not be improved very much by resorting. In fact, in this particular case, resorting the result set may hurt its precision, as any ordering imposed by our textual search engine will be lost. On the other hand, a result set in which most of the relevant images are ultrasounds and most of the non-relevant images are x-rays could benefit from being re-ordered based on visual similarity to a user-selected probe image.
Our present system requires the user to select a combination of features to use. This is clearly sub-optimal, and our future work could include improved feature selection methods. Similarly, the user is currently able to change the number of eigenvalues used by the algorithm. While this is a powerful tool for tuning the algorithm’s performance, it is also something that we would ultimately like to automate.
We submitted a total of 10 runs. The search options for the different runs are provided in table 1. These runs included textual and mixed, automatic and interactive options. Although the ImageCLEF2005-2007 collection with qrels and topics was available, we did not use any external training data.
Three automatic text-based runs were submitted with different custom parsing options including the use of UMLS term expansion. We also submitted four mixed, automatic runs. The modality classification based on the text parsing of the caption and the classification based on visual features was used to improve the precision of the search.
While the majority of our runs were automatic in nature, several of ours were interactive. In the first such run (ohsu_int_2), the user chose different combinations of options for each topic and added terms based on the list provided using the UMLS query expansion option. Two runs using the interactive result sorting system were submitted. The first such run, “ohsu_sdb_lsa”, used the result sorting system on every topic. The second run, “ohsu_sdb_full_interactive”, only used the result sorting system on topics where the user thought that it would be beneficial to the run’s precision. This second run also featured much more intervention on the part of the user, who took full advantage of our retrieval system’s interactive nature and enabled or disabled options and features as needed.
Table 2 contains a subset of the official performance metrics for the OHSU runs. We have also included the average of these metrics for all runs, the highest measure in each category as well as data from the best run (based on MAP) in the 2008 campaign. OHSU performed reasonably well, especially among the runs that did not use any external training data. All but two of our runs performed better than the average for all measures. As described in the previous section, our systems have been designed to improve precision, perhaps at the expense of recall. Our custom parsing improved the mean average precision as well as the early precision, as can be seen in the text runs. The use of modality parsing and detection improved the MAP as well as the early precision. All our mixed runs performed better than the corresponding text runs. OHSU-ohsu_mod_pars2_sp.txt had the highest early precision (up to P30) of all official runs. OHSU had submitted four of the top ten mixed runs, as sorted using the precision as 10. The use of term expansion with UMLS increased the recall. We had submitted runs after the creation of the pools. This penalizes the runs as potentially fewer of the images are judged. One of these runs had the highest bpref, a measure that is robust in the case of incomplete judgments.
The performance of the first LSA run (ohsu_sdb_lsa) was unsatisfactory: as described earlier, there are many situations in which the original result sorting provided by our textual search engine was adequate, and changing Acknowledgments
We acknowledge the support of NLM Training Grant 1T15 LM009461 and NSF Grant ITR-0325160. it by means of our interactive visual re-sorting system damaged a topic’s precision. The second LSA run, “ohsu_sdb_full_interactive”, performed much better. In fact, its p10 was greater than that of the overall competition winner’s (0.46 for “ohsu_sdb_full_interactive” vs 0.43 for “SINAI-sinai_CT_Mesh_Fire20”). The third interactive run, where the parsing mode and UMLS term expansion was performed interactively also performed quite well.
Our image retrieval system built using open-source tools is a flexible platform for evaluating various tools and techniques in image processing as well as natural language processing for medical image retrieval. The use of visual information to automatically extract the imaging modality is a promising approach for the ImageCLEFmed campaign. The use of UMLS term expansion, query parsing and modality detection all add value over the basic Ferret (Lucene) search engine. We will continue to improve our image retrieval system by adding more image tags using automatic visual feature extraction. Our next goal is to annotate the images with the their anatomical location and view attributes.