Over the past several years, our team has focused its e orts on improving retrieval precision performance by mixing visual and textual information. This year, we chose to explore ways in which we could use external data to enrich our retrieval system's data set; speci cally, we annotated each image in the test collection with a set of MeSH headings from two di erent sources: human-assigned MEDLINE index terms, and automatically-assigned MeSH headings (via the National Library of Medicine's MetaMap software). In addition to exploring these di erent data enrichment techniques, we also revamped the architecture of our retrieval system itself. In past years, we have used a two-tiered approach wherein the data is stored in a relational database (RDBMS), but the indexing and searching are done using Lucene-like system. This year, we took advantage of our RDBMS's full-text search capabilities and performed both storage and searching in the RDBMS. This turned out to have both positive and negative e ects at a practical level. On the one hand, using the database's built-in text retrieval subsystem resulted in improved retrieval speed and easier query analysis; however, these gains came at the cost of reduced exibility and increased code complexity. Our experiments investigated the e ects of using various combinations of human- and automatically-assigned MeSH terms, along with several of the techniques that have proved useful in previous years. We found that including automatically-assigned MeSH terms sometimes provided a small amount of improvement (in terms of bpref, MAP, and early precision) and sometimes hurt performance, whereas including the humanassigned MEDLINE index headings consistently yielded a sizable improvement in those same metrics.
As has been discussed at length in previous works[12, 11], medical image
retrieval represents a large and ever-growing problem. As the utilization rates of
diagnostic imaging increase[
The ImageCLEF series of evaluation campaigns provides a forum for researchers working on image retrieval problems to share ideas and compare their systems. One of the campaign's ongoing tracks is a medical image retrieval task, which is described in detail in [12, 11]. This year, the task's test collection was an expanded version of the collection used in 2008 and 2009, and included 77,495 images from 5,609 articles in the journals Radiology and Radiographics.
OHSU has participated in the medical track of ImageCLEF since 2006, and
our focus has consistently been on exploring ways to make use of both visual and
textual information during retrieval. Over the past several years, our system has
achieved good performance (particularly in terms of precision) by using image
modality information to adaptively determine which results are relevant to a
given query[
This year, however, we decided to try something slightly di erent. In the past, we had annotated documents in the test collection with automaticallyextracted modality labels and made use of external knowledge sources (such as the US National Library of Medicine's UMLS metathesaurus) to perform query expansion. This year, we attempted to make further use of external knowledge in the form of MeSH (Medical Subject Heading) keyword annotations.
Each record in the ImageCLEF medical test collection includes a \PMID," or PubMed identi er: essentially, a link back to the MEDLINE record (journal article) from which the image originally came. Each entry in MEDLINE is indexed by a professional indexer, and we added these index terms to our copy of the test collection. We therefore were able to make use of an average of 12 guaranteed-relevant keywords for each item in the collection. Furthermore, since a certain number of index terms for each MEDLINE entry are designated as \major headings" (i.e., particularly relevant keywords), we were able to be highly con dent of the relevance of at least a few index terms for each collection entry.
Of course, most medical data sets do not include human-curated index terms.
We therefore experimented with automatically-assigned index terms using the
National Library of Medicine's MetaMap software1[
As in years past, our retrieval system is written in the Ruby language3 and uses
the Ruby on Rails4 web framework as well as the open-source PostgreSQL
relational database system5. However, unlike our system from 2006{2009, this year's
system uses neither Lucene nor Ferret (a port of Lucene to Ruby)6 to perform
1 http://mmtx.nlm.nih.gov/
2 See [
In the past, our system had to maintain its full-text index of image captions, titles, etc. as a separate le, and relied on extra software libraries to perform retrieval. Our hope was that, by integrating the full-text searching with the database itself, our system would have fewer \moving parts." This turned out to be the case; integrating text search with the database did make certain parts of our system less cluttered, and this year's system's retrieval speed was de nitely improved over previous years' systems.
However, these improvements came at a cost. PostgreSQL's full-text query syntax is somewhat cumbersome, and modifying our query processor to translate user queries into PostgreSQL-compatible queries turned out to be more challenging than we had initially anticipated. That said, the fact that PostgreSQL's search subsystem uses a set of extensions to standard SQL syntax meant that it was extremely convenient and easy to develop and debug our query processor, particularly in comparison to our analogous experiences with Lucene, whose query system could be somewhat opaque.
Overall, though, the nal product ended up being somewhat more complex than its Lucene-based predecessor, and there were also certain convenience features that we missed. For example, the port of Lucene that we used made it very easy to add additional index elds that represented dynamically calculated values (for example, a concatenation of two other elds). Achieving a similar e ect using PostgreSQL involved adding a new index to our database itself. While this is certainly easy enough to do, it ultimately resulted in a very cluttered database schema.
Overall, integrating our search system with the database itself was probably a wash from a technical standpoint. From a performance standpoint, it ended up representing a small step backwards in certain ways. Lucene uses a vector-space retrieval model, whereas PostgreSQL's text search subsystem uses a boolean model. As such, we found this year's system to be much less robust when faced with topics with few relevant results, which a ected its recall.
Besides this architectural change, other aspects of our system were relatively
unchanged from the descriptions given in previous years' Working Notes
papers[
To compensate for this, we added a mode to our system wherein the nal
result set returned to the user will contain both ltered and un- ltered results,
7 http://www.postgresql.org/docs/8.4/static/textsearch.html
8 See [
In addition to this modi cation, as mentioned earlier, this year we also added externally-derived annotations to our database, in the form of MeSH headings. To actually make use of these for retrieval, we combined the image caption elds with a space-delimted list of MeSH headings to create two \meta- elds" (one that included the human-generated MEDLINE index terms, and another for the automatically-assigned headings). We then created full-text indices of these columns just as we would any others (e.g., caption, title, etc.), and used them accordingly.
As in previous years, our system can be used interactively (see gures 1 and 2). However, it can also be used in a more batch-oriented manner (see gure 3), and its results can be downloaded directly as trec-eval-compliant run les. Furthermore, one of the authors (SB) has written a script which submits queries and generates runs programmatically. We submitted a total of ten ad-hoc runs, all of which used some combination of system features (see table 1 for a breakdown of the runs). Our primary focus this year was on exploring the e ects of using di erent combinations of MeSH terms. As mentioned earlier, we had two types of MeSH term for each record in the collection: a set of human-assigned MEDLINE index terms, and a set of automatically-assigned terms. Of the MEDLINE terms, some were designated by the human indexers at the National Library of Medicine as \major topics" (i.e., particularly relevant key words). Our runs either did or did not include the automatically-assigned MeSH terms (\MetaMap"), and used either none of the MEDLINE terms, only the major topic terms, or all of the MEDLINE terms.
Another setting we varied from run to run was whether to use modality ltration, and, if so, whether to use modalities extracted from image titles, captions, a visual classi er (\jaykc"), or the union of all three. Additionally, some of our runs used image titles as well as captions; others only used captions.
For the purpose of comparison, we also produced a \control" run featuring all of our \classic" features (query expansion, modality ltration and reordering, and use of titles) but without any of the MeSH terms. We did not submit this run to ImageCLEF, but we did run it through trec eval with the same qrel le. As such, its results are directly comparable to those of our submitted runs. 4
OHSU's runs performed competitively, although we were not at the very top of the categories we competed in. Our results are summarized in table 2 and gure 4. In terms of both MAP and bpref, our runs follow a bimodal distribution, with run bpref map p5 p10 p20 p100 pm all all mod 0.344 0.3029 0.4875 0.4313 0.3344 0.1562 pm major all mod 0.3404 0.3004 0.5000 0.4375 0.3469 0.1519 all mh major all mod 0.3428 0.2983 0.4625 0.4188 0.3031 0.1494 all mh major jaykc mod 0.3428 0.2983 0.4625 0.4188 0.3031 0.1494 high recall with titles modality reorder 0.2754 0.2623 0.4375 0.3875 0.293 0.1644 high recall with titles 0.2714 0.2592 0.4375 0.3875 0.2937 0.1581 all mh major jaykc mod reorder 0.2533 0.256 0.4375 0.3813 0.275 0.1487 all mh major all mod reorder 0.2533 0.256 0.4375 0.3813 0.275 0.1487 mm all mod 0.2594 0.2476 0.4625 0.4125 0.3062 0.1444 high recall 0.2533 0.2386 0.4125 0.3625 0.2844 0.1544 control 0.2614 0.2397 0.4000 0.3625 0.2875 0.1581 runs a{d noticeably outperforming the remainder. In terms of early precision, our runs all performed quite well, although there was some notable between-run variation.
Regarding the performance of runs a{d, the question arises: what was different about those runs from the others? They all featured the combination of MEDLINE terms, modality ltration, query expansion, and no result reordering. Similarly-con gured runs that did use result reordering su ered a map penalty, as did runs without reordering but using MetaMap terms instead of MEDLINE terms (e.g., mm all mod, although it should be noted that while this particular run had a low map, it did quite well in terms of simple early precision). Using only \major subject" MEDLINE headings did not seem to yield any signi cant bene t over simply using all of the human-assigned index terms. 5
In conclusion, our idea of using external MeSH annotations to improve retrieval does seem to have promise; however, there is a great deal of experimentation yet to do before we can take full advantage of these annotations. Speci cally, we need to determine why the manually-assigned MEDLINE annotations seemed to be so much more bene cial than the automatically-assigned annotations. One possible explanation would be that the average quality of the automaticallyassigned annotations is lower than the average quality of the human-assigned annotations| i.e., that the MetaMap program is assigning erroneous or incomplete subject headings. Our initial examinations have actually shown the opposite to be true| given image captions, MetaMap seems to be doing a reasonable job at assigning relevant and speci c MeSH headings.
Another possibility is that the level of annotation detail is di erent between the human- and automatically-assigned MeSH headings, and that this di er
c d e f g h j k
p100 ence in detail is a ecting the ability of the terms to help retrieval. The human indexers are assigning subject headings for an entire article, whereas in our system MetaMap is working on individual captions. Therefore, our automaticallyassigned annotations sometimes seem to be very speci c, whereas the humanassigned annotations can seem very vague. For example, consider the case of gure 62141, from an article entitled \High-resolution CT and CT angiography of peripheral pulmonary vascular disorders." The caption text is as follows: Figure 8c. Parasitic pulmonary embolism. (a, b). CT scans demonstrate rupture of an Echinococcus cyst (E. granulosus ) (*) into the inferior vena cava (c, d). CT scans show peripheral pulmonary embolism of scolices (c) with subpleural calci ed daughter cysts (arrowheads in d). Massive central pulmonary arterial embolism can occur in hydatid disease or in ascariasis in association with acute pulmonary arterial thrombosis.
This represents a detailed description of a gure, including lots of helpful anatomical vocabulary. The MEDLINE terms for this article are: { Angiography { Humans { Lung Diseases { Peripheral Vascular Diseases { Tomography, X-Ray Computed { Rupture { Cysts { Thrombosis { Echinococcosis { Ascariasis { Pulmonary Artery { Tomography, X-Ray Computed { Vena Cava, Inferior { Pulmonary Embolism
The MetaMap-derived terms, on the other hand, are as follows: Clearly, these two sets of terms are operating at di erent levels of speci city. While we have not done an exhaustive study of our annotations, this pattern does seem to repeat itself with some regularity from record to record. Given this asymmetry in annotation detail between sources, it is not surprising that one source would prove more helpful than the other. However, the questions of which source would be most helpful, and, more importantly, why it would be so, remain open and will form the next step in this research.
Additional future steps include exploring ways to enrich our use of these annotations. Currently, we are using them in a very simplistic way, and are treating all annotations as equally important. Perhaps we should only make use of certain categories of annotation, and use only anatomical terms, for example. We may want to weight certain annotations more heavily than others, based perhaps on frequency of occurrence. Some very common annotations might be best ignored altogether. Hopefully, exploring these directions will enable us to improve our system's performance for next year's ImageCLEF.
We acknowledge the National Library of Medicine grant 3T15LM007088 for supporting this work, as well as 1K99LM009889. 11. Muller, H., Kalpathy-Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Jr., C.K., Hersh, W.: Overview of the medical retrieval task at imageclef 2009.
Working Notes of the CLEF 2009 workshop, Corfu, Greece (2009) 12. Muller, H., Kalpathy-Cramer, J., Kahn Jr., C.E., Hatt, W., Bedrick, S., Hersh, W.: Overview of the ImageCLEFmed 2008 medical image retrieval task. In: Peters, C., Giampiccol, D., Ferro, N., Petras, V., Gonzalo, J., Pen~as, A., Deselaers, T., Mandl, T., Jones, G., Kurimo, M. (eds.) Evaluating Systems for Multilingual and Multimodal Information Access { 9th Workshop of the Cross-Language Evaluation Forum. Lecture Notes in Computer Science, Aarhus, Denmark (Sep 2008 (printed in 2009)) 13. Radhouni, S., Kalpathy-Cramer, J., Bedrick, S., Bakke, B., Hersh, W.: Multimodal medical image retrieval improving precision at imageclef 2009. In: Peters, C. (ed.) Working Notes for the CLEF 2009 Workshop, Corfu, Greece (2009)