Living with the Semantic Gap: Experiences and Remedies in the Context of Medical Imaging Yannis Kalfoglou, Srinandan Dasmahapatra, David Dupplaw, Bo Hu, Paul Lewis, and Nigel Shadbolt Abstract—Semantic annotation of images is a key concern and a set of concepts for describing meta-image for the newly emerged applications of semantic multimedia. information. For instance, image analysts and/or Machine processable descriptions of images make it possible to radiographic technicians might focus more on low-level automate a variety of tasks from search and discovery to graphic features, e.g. shape, size and luminosity of an ROI composition and collage of image data bases. However, the ever occurring problem of the semantic gap between the low (Region Of Interest) while a radiologist might step away level descriptors and the high level interpretation of an image from the fine details and concentrate on the interpretation of poses new challenges and needs to be addressed before the full all ROIs within the context of the whole image. While potential of semantic multimedia can be realised. We explore looking into each individual imaging module, we define not the possibilities and lessons learnt with applied semantic only image descriptors but also image-capture related multimedia from our engagement with medical imaging where concepts. However, knowing what is on an image is we deployed ontologies and a novel distributed architecture to provide semantic annotation, decision support and methods for sometimes not sufficient for domain experts to give a proper tackling the semantic gap problem. interpretation of the image, as knowledge on how the image is produced is equally critical. In practice, deciding whether Index Terms—Semantics, Medical Imaging, Ontologies, such knowledge should be included or ignored is a trade-off Description Logics. between the complexity and the accuracy of the ontology. BCIO is trying to satisfy the requirements on both usability I. MEDICAL IMAGING DOMAIN and extensibility. BCIO provides handles for the A dvances in medical technology generate huge amounts of non-textual information, like images and other multimedia, along with more conventional media like text information pertaining to a particular case based on different aspects and different grain-sizes that are appropriate. This allows an expert to focus only on the facets relevant to her reports. Most of the existing systems focusing on extracting interest and/or expertise and makes available her interest visual cues with the aid of image analysis algorithms may BCIO is supported by a distributed architecture which experience problems when a rather abstract and ambiguous enables a number of web-based services that provide query is asked [1]. Because low level descriptors cannot be discrete and disparate functionality to a generic application uniquely associated with any other meaningful label unless base. We provide annotation support where users can explicitly declared or derived as the outcome of a annotate an ROI of their choice with the aid of a graphical classification procedure, retrieval based on knowledge level editor. We also support semantic querying on the BCIO constructs is a non-trivial task to achieve in general. Our concept and instance descriptions. BCIO is DL-based and domain of exploration is medical imaging, in particular we employ DL-based inference to provide automatic providing semantic support for the breast cancer screening classification of query constructs using the underlying processes. In the context of the MIAKT project (Medical BCIO ontology. Imaging and Advanced Knowledge Technologies 1 we built the Breast Cancer Imaging Ontology (BCIO). It consists of II. DEALING WITH THE SEMANTIC GAP several relatively independent modules at different levels of When we consider the semantic gap as described by Hare granularity with uniform interfaces to enable integration. and colleagues in [2]: "much of the interesting work which Separations are defined vertically and horizontally. Because is attempting to bridge the gap automatically is tackling the the patients are viewed through different apparatuses and gap between descriptors and the labels and not that between instrumentation whose results are overlaid and compiled to the labels and the full semantics"; the question is how this give the whole picture, a natural vertical separation of the gap can be narrowed, with which technology and under domain would be one module for each imaging method, X- which assumptions. We reformulate this question as a ray, MRI, Ultra-Sound, etc. Each imaging module is question of applying ontology mapping techniques to tackle composed of a set of image feature descriptors, a set of it. As long as we have available a codified representation of diagnosis descriptors capturing high-level abstract features the full semantics Hare and colleagues are referring to, then ontology mapping is a feasible approach with a lot of Y.Kalfoglou and N.Shadbolt are with the Advanced Knowledge potential applications. We first apply a technique which Technologies (AKT) group; email: {y.kalfoglou|nrs}@ecs.soton.ac.uk. allows us to wrap the different interpretations of an image Y.Kalfoglou is the corresponding author. into a single representation. But, there are situations where S.Dasmahapatra is with the Science and Engineering Natural Systems (SENSe) group. Email: sd@ecs.soton.ac.uk . we need to preserve these different interpretations and align D.Dupplaw, B.Hu, and P.Lewis are with the Intelligence Agents them for the sake of enabling interoperability. To do that, Multimedia (IAM) group. Email: {dpd|bh|phl}@ecs.soton.ac.uk. we employ semantic alignment. It is a subset of a bigger set All authors are with the School of Electronics and Computer Science of technologies that aim to find alignments of entire (ECS), University of Southampton, Southampton SO17 1BJ, UK. 1 ontologies, like for instance ontology mapping and More on: www.aktors.org/miakt alignment tools. To tackle the problem of finding question, but practitioners like to work with more and more alignments in order to narrow the semantic gap between expressive and detailed descriptions of an image (especially semantics of the multimedia object and its label, we deploy in specialist domains, like medical imaging). Finally, an the idea of semantic issue which is related to that of annotation, is the use and alignment which aims to use a subset of an ontology analysis of user queries to assist with annotation and mapping system. In particular, we use semantic metrics [4] tackling the semantic gap. In their brief survey of user based to discover alignments between ontological structures which queries analysis, Hare and colleagues point out a number of could range from concept name alignments to simple string techniques that aim to classify user queries according to a matching. We devised a modular architecture for deploying given organisation. As most of that work is focused on ontology mapping systems, most of which can provide us inferring the context of the query, thus assisting in finding with semantic alignments of the structures we are interested the most relevant image, we are interested to see whether in. The principle behind our modular architecture is that such an analysis could be used alongside other contextual there is a variety of alignment systems out there and we are information to help narrow the semantic gap between the keen to use them conjunctively for the benefit of a better object labels (image in question) and the full semantics informed alignment. For example, in the context of mapping (image description). a specialists medical vocabulary, like the Foundational Bridging the semantic gap in visual information retrieval Model of Anatomy (FMA) large OWL ontology, to a is a key enabler for deploying semantic multimedia generic medical model, like the OpenGALEN ontology, we applications. Our experience with the MIAKT project and found that we can improve the results of mapping by the breast cancer screening process support highlighted that deploying different ontology mapping systems; each of this phenomenon exists and is fuelled by the inevitably which provides a cutting edge in different alignment different interpretations of images by different domain algorithms [5]. The architecture is implemented as a experts. We presented an approach to tackle it based on a multi-stage and multi-strategy system comprising of four simple wrapping technique, and we are considering modules, namely, Feature Generation, Feature Selection deploying ontology mapping technology to cope with the and Processing, Aggregator and Evaluator. In this system, more semantically rich heterogeneous descriptions of those different features of the input ontologies are generated and images (if available). However, we also advocate that the selected to fire off different kinds of feature matchers, use of a flexible and distributed architecture with reasoning which are an integral part of many ontology mapping systems. The resultant similarity values are compiled by support, such as the one we presented earlier, is an multiple similarity aggregators running in parallel or important infrastructure that needs to be in place to support consecutive order. The overall similarity is then evaluated to semantic multimedia applications. initiate iterations that backtrack to different stages. ACKNOWLEDGMENT III. CONSIDERATIONS All the authors except Kalfoglou and Shadbolt are The MIAKT architecture was designed to provide general supported fully or partially by the OpenKnowledge and knowledge management in many domains, while allowing HealthAgents projects which are funded by the EU IST semantically marked-up services to be easily integrated into Framework 6 under Grant numbers IST-FP6-027253 and a knowledge management application. We applied three IST-FP6-027213. Kalfoglou and Shadbolt are fully guidelines when designing the architecture: (a) it must allow supported by the Advanced Knowledge Technologies institutions to retain control over their own data. In the (AKT) Interdisciplinary Research Collaboration (IRC) medical imaging domain, this means that a hospital's project which is sponsored by the UK EPSRC under Grant radiography unit retains control of the images that they number GR/N15764/01. The views and conclusions produce by having them stored on an institutional image contained herein are those of the authors and should not be server; (b) it must provide simple and fast extendibility. interpreted as necessarily representing official policies or This means that new services can be imported into the endorsements, either expressed or implied, of the EU IST architecture quickly and easily; (c) a major consideration FP6 or the UK EPSRC or any other member of the UK which proved a challenge in its implementation, is that the AKT IRC. architecture must provide enough flexibility to be able to be used in different application domains. Ensuring the REFERENCES distinction between domain and non-domain data, both [1] E. Di Sciascio, F.M.Donini and M.Mongelio. Structured Knowledge ontological and otherwise, means that a great deal of care Representation for Image Retrieval. Journal Of Artificial Intelligence Research, has to be taken in designing the interfaces between 16:209-257, 2002 [2] J.Hare, P.Lewis, P.Enser, and C.Sandom. Mind the gap: Another look at the components. Utilising ontologies for the description of the problem of the semantic gap in image retrieval. In Proceedings of Multimedia system as well as the domain has helped us achieve this Content Analysis, Management and Retrieval 2006 SPIE Vol.6073, pp.607309- 1, San Jose, California, USA., 2006 goal. Another consideration is that of annotation. Although [3] L.Hollink, G.Shreiber, B.Wielemaker, and B.Wielinga. Semantic annotations annotation of images with the use of a rich semantic for image collections. In Proceedings of the K-CAP’03 Workshop on substratum (ideally codified in an ontology) has been Knowledge Markup and Semantic Annotation, Banff, Canada, October 2004. [4] B.Hu, Y.Kalfoglou, H.Alani, D.Dupplaw, P.Lewis, and N.Shadbolt. Semantic explored in the literature (see, for example, [3]), we are Metrics. In Proceedings of the 15th International Conference on Knowledge looking forward to a more expressive annotation regime that Engineering and Knowledge Management (EKAW’06), Podebrady, Czech Republic, October 2006. goes beyond memorable object labels and attaches what [5] Y.Kalfoglou and B.Hu, CMS: CROSI Mapping System – Results of the 2005 Hare and colleagues refer to as "full semantics". There Ontology Alignment Contest. In Proceedings of the K-CAP’-5 Integrating might be practical problems with the attachment and Ontologies workshop, Banff, Canada, pages 77-84, October 2005. location and use of those codified semantics on the image in