Today, clinicians rely on medical images for screening, diagnosis, treatment planning and follow up, but still a generic and exible medical image understanding is missing. Although, there exist several approaches for semantic image annotation, those approaches do not make use of practical clinical knowledge, such as best practice solutions or clinical guidelines. Our nal goal is to enhance medical image annotations by integrating clinical knowledge, such as lymphoma staging systems. The contribution of this paper is to introduce a formal approach to the classi cation of patients in well-de ned categories. As rst step, we have developed an OWL DL ontology representing the Ann-Arbor Lymphoma staging system that is suitable for performing automatic patient classication. Our aim for the ontology design was to establish means for automatic staging that maps each patient to one staging degree.
The vision of MEDICO is to automatically extract the meaning from the medical images and to seamlessly integrate the extracted knowledge into medical processes, such as clinical decision making. In other words, the computer shall learn to nd, catalogue and interpret medical images. This requires the semantic representation of medical images' content and the preprocessing of semantic image annotations for seamless integration into clinical applications.
There exist several approaches for semantic image annotation, such as
automatic image parsing [
The remainder of the paper is organized as follows. In Section 2 we will introduce the knowledge resources our approach is based on. Section 3 details challenges we faced and design decisions we made when developing the ontology for lymphoma patient classi cation. In Section 4 we will discuss related approaches and Section 5 concludes this paper with an outlook on future work. 2
is the staging system for lymphomas. It was initially developed for Hodgkin's Lymphoma, but has some use in Non-Hodgkin lymphomas. The stage depends on two criteria. The rst criterion is the place where the malignant tissue is located. The location can be identi ed with located biopsy as well as with medical imaging methods, such as CT scanning and increasingly positron emission tomography. The second criterion are systemic symptoms, such as night sweats, weight loss of more than 10 percent or fevers, caused by the lymphoma. Those systemic systems are called \B symptoms".
The principal stage is determined by the location of the tumor and re ects the grade of expansion of lymphoma occurrences. Four di erent stages are recognized: Stage I indicates that the cancer is located in a single region, either an a ected lymph node or organ within the lymphatic system. In Stage II the cancer is located in two separated regions, an a ected lymph node or an a ected organ within the lymphatic system and a second a ected lymph node area. Moreover, the a ected areas are con ned to one side of the diaphragm - that is, both are above the diaphragm or, both are below the diaphragm. Stage III indicates that the cancer has spread to both sides of the diaphragm, including one extra lymphatic organ or site. Stage IV shows di use or disseminated involvement of one or more extra lymphatic organs.
Ontological Knowledge Resources For capturing the semantics to the
classication of lymphoma patients and for achieving re-usability and interoperability,
we required third party taxonomies or ontologies that cover all possible regions
of lymphatic occurrences. Two ontologies, the Foundational Model of Anatomy
(FMA)[
The FMA is developed and maintained by the School of Medicine of the
University of Washington and the US National Library of Medicine. Besides the
speci cation of an anatomy taxonomy, i.e. an inheritance hierarchy of
anatomical entities, the FMA provides de nitions for conceptual attributes, part-whole,
location, and other spatial associations of anatomical entities. FMA covers
approximately 70,000 distinct anatomical concepts and more than 1.5 million
relations instances from 170 relation types. It provides concepts that describe single
lymph nodes, such as 'axilliary lymph node', and concepts that describe multiple
lymph nodes, such as 'set of axilliary lymph node'. It contains 425 concepts
representing singular lymph nodes and 404 concepts describing sets of lymph nodes.
The FMA is freely available as a Protege 3.0 project or can be accessed via the
Foundational Model Explorer. There also exist conversions of the frame-based
Protege version of FMA to the OWL DL format [
RadLex is a terminology developed and maintained by the Radiological Society of North America (RSNA) for the purpose of uniform indexing and retrieval of radiology information, including medical images. RadLex contains over 8,000 anatomic and pathologic terms, also those about imaging techniques, di culties and diagnostic image qualities. Its purpose is to provide a standardized ter1 www.radlex.org minology for radiological practice. RadLex is available in English and German language and covers 144 concepts describing lymph node concepts.
As the use case scenario relies only on high-level concepts, both ontologies
- FMA and RadLex - are suitable in terms of coverage and both ontologies are
used within the MEDICO project for annotating medical images and radiology
reports. For this use case, we decided to use RadLex as primary third party
resource for labeling lymphatic occurrences. As we are working with radiology
reports in German language, the availability of a German translation was an
crucial argument for this decision. Due to the large size of Radlex, we needed
to establish an ontology fragment that is scalable and e cient for reasoning
application. Moreover, we transformed the ontology fragment into an appropriate
OWL DL format. The manually created ontology fragment covers all RadLex
concepts describing lymph nodes as well as a list of likely regions for extra nodal
lymphatic occurrences covering concepts such as liver, skin, or bone.
Semantic Image Annotation The MEDICO project is based on multiple
ways of generating semantic image annotations. For instance methods for
automated image parsing, such as [
Ontology-based Classi cation of Lymphoma Patients
Our aim is to establish an ontology that enables the automatic classi cation of
lymphoma patients by means of ontology-based reasoning services. For doing
so, we transformed the Ann Arbor Hodgkin Lymphoma classi cation [
In a rst step, the Ann Arbor Hodgkin Lymphoma Classi cation was translated into complex questions for the subsequent decomposition into more simple and manageable queries. Both, the complex questions as well as the simple questions, provided us valuable guidance for evaluating the ontological commitment that has been made. This was possible by systematically establishing test patient classes and respectively patient instances according to the simple and the complex questions. 3.1
As already mentioned, the stages of the Ann Harbor Staging System classify lymphoma patients in accordance to their number and location of lymph node occurrences and extra lymphatic organ or site involvement. For accessing the number of lymph node occurrences and respectively extra lymphatic organ or site involvement, the following queries, i.e. de ned OWL classes, were de ned: { The OWL de ned Classes N0, N1 and N2 identify all patients with zero, one or two and more involved lymph node regions. { The OWL de ned Classes E0, E1 and E2 identify all patients with zero, one or two and more involved extra lymphatic organ or site involvement. { The OWL de ned Classes N AllAboveD and N AllBelowD identify patients with the location of the lymph node regions either only above or only below the diaphragm. For accessing patients with occurrence of lymph node on both sides of the diaphragm, one has to make sure that the occurrences neither are located all above or located all below the diaphragm. This can be formulated by the following axiom:
: N AllAboveD t : N AllBelowD { The location of extra nodal occurrences are identi ed in an analogous manner, i.e. by establishing the de ned classes E AllAboveD and E AllBelowD, as well as the corresponding complex axiom for accessing patients with extra nodal occurrences on both sides of the diaphragm.
In a further step, the above established simple or auxiliary queries could be used for specifying de ned classes deducing the Ann Harbor Stages. Before detailing the staging classi cation, we will discuss the particular requirements towards the ontology design for realizing the auxiliary queries. 3.2
For establishing the ontological model basically three di erent challenges needed to be addressed: 1. How to ensure that the ontological model provides the basis for inferring the number of lymph node occurrences and the number of extra lymphatic involvements? 2. OWL DL is based on the Open World Assumption (OWA), thus, it is only applicable to a limited extent for checking the validity of closed sets of information. Moreover, the patient diagnosis deals with increasing set of information. How to represent the patient staging classi cation dealing with an open set of information in combination with the OWA paradigm? 3. How to identify the relative location { above, below or on both sides of the diaphragm { of the lymphatic occurrences? Counting Lymphatic Occurrences Although in most programming languages counting items is quite straight forward, this is not the case for OWL ontologies. Counting lymphatic occurrences requires some preparation to achieve the intended classi cation. A reasoner is only able to count concepts that are di erent. To provide the basis for counting the number of involved regions, the concepts for lymph nodes and for extra nodal lymphatic regions in the established RadLex fragment need to be labeled as disjoint. Subsumption in OWL means necessary implications, thus, classes and their subclasses can not be labeled as disjoint. Therefore, we decided to describe the parent-child relationship of RadLex by introducing a transitive is a relationship and by using the following type of axioms
Right Hilar Lymph Node v (Anatomical Structure u 9is a:Hilar Lymph Node Additional, we entered disjointness axioms indicating that the lymph nodes as well as the extra nodal lymphatic regions are not overlapping.
for the ontology design was to establish means for automatic staging that maps each patient to uniquely one staging degree. Due to open world reasoning, this could not be achieved directly. For instance, by establishing a de ned class with necessary and su cient conditions stating the existence of exactly one lymph node occurrence, e.g.
ExampleStage
Patient(x) ^ 9=1y hasLymphaticOccurrence(x; y) the inferred ontological model will never classify patient classes as subclasses of the ExampleStage. In open world reasoning, anything might be true unless it can be proven as false. In the context of counting lymphatic occurrences, this translates to: Any patient might have more than the indicated lymphatic occurrences unless it can be proven as false. For instance, a Patient A with one explicit indicated lymph node occurrence can possibly have two or more lymphatic occurrences unless explicitly stated di erently. Inferring Patient A as sub-class of a staging class consisting of patients with exactly one lymphatic occurrence would bear the risk of future inconsistencies. However, staging classes speci ed by a minimum number of occurrences, such as 9>=1y hasLymphaticOccurrence(x; y) are suitable for OWL DL reasoning. But this again will cause, for instance, that a patient with three indicated lymphatic occurrences on both side of the diaphragm is mapped to more than one staging class, i.e. stage I and stage III. Our goal is to achieve the unique mapping between patients and staging classes. This becomes possible by interpreting the highest staging class as the unique mapping result. Yet, this interpretation is not explicitly stated in the derived OWL DL model and has to be re ected when realizing clinical applications that make use of the automatically inferred classi cation results.
The Location of Lymphatic Occurrences The patient staging results distinguish between patient that have lymphatic occurrences only above, only below or on both sides of the diaphragm. Thus, the relative position of lymphatic occurrences to the diaphragm has to be expressed in the knowledge model. For achieving this, we see two alternatives. The information about the relative position of lymphatic occurrences can either be derived by the image segmentation algorithm or be directly derived from the underlying ontological model. In other words, this information either comes with the patient record information or with the integrated anatomy and radiology knowledge model. The computation of relative spatial positions of lymphatic occurrences by segmentation algorithms will be addressed within our future work. In the meantime, we have to rely on the anatomy and radiology ontology for deriving this knowledge. RadLex and FMA do not explicitly capture information about relative positions between lymphatic regions or organs and the diaphragm. Thus, we required to enhance the RadLex fragment accordingly. This could be achieved by extending each lymphatic region by an axiom indicating that the region is above, or respectively, below the diaphragm. The classi cation of lymphatic regions - and of patients with lymphatic occurrences - above, below or on both sides of the diaphragm was modeled as value partition. 3.3
The Ann-Arbor Staging classes are represented as de ned OWL DL classes. Each staging class is capturing the semantics as detailed in Subsection 2. Their formal representation makes use of auxiliary classes introduced in Subsection 3.1. Figure 2 summarizes the Ann-Arbor Staging Formalization. Thus, for instance, Stage-II-N gathers all patients with more that two lymph node region that are all on one side of the diaphragm and Stage-II-mixed all patients with one involved lymph node regions and one involved extra lymphatic organ or site on the same side of the diaphragm, and so on. 3.4
Evaluation is an ongoing topic. For evaluating the simple queries as well as the complex staging queries, by hand we systematically generated the corresponding test patient classes and respectively test patient instances. Each test patient is equipped with di erent numbers and kinds of lymphatic occurrences at di erent locations in the body. In this way, the evaluation (as well as the development) of the classi cation axioms was straight forward.
Moreover - in our ongoing work - we are using real radiology reports for challenging the capabilities of our ontology model. Our goal is to semantically annotate more than 250 radiology reports that provide the basis for translating the clinical ndings - in terms of numbers and types of lymphatic occurrences into the OWL representation. As input we are using the semantic annotations (see left side of Fig. 3) which is translated into OWL format (see right side of Fig. 3), i.e. the class based OWL DL representation of a real patient with a large number of lymph nodes who was classi ed as Stage-III.
One particular challenge we are facing is how to handle multiple lymph node occurrences at the same location. For deciding how to tackle this problem, a detailed analysis of the available patient record will be required.
There exist a wide range of di erent imaging technologies and modalities, such as 4D 64-slice Computer Tomography (CT), whole-body Magnet Resonance Imaging (MRI), 4D Ultrasound, and the fusion of Positron Emission Tomography and CT (PET/CT) providing detailed insight into human anatomy, function and disease associations. Moreover, advanced techniques for analyzing imaging data generating additional quantitative parameters paving the way for improved clinical practice and diagnosis. However, for advanced applications in Clinical Decision Support and Computer Aided Diagnoses the comparative exploration of similar patient information is required. The missing link is a exible and generic image understanding. Currently, the large amounts of heterogeneous image data are stored in distributed and autonomous image databases being indexed by keywords without capturing any semantics. The vision of the MEDICO project is to automatically extract the meaning from the medical images and to seamlessly integrate the extracted knowledge into medical processes, such as clinical decision making, and to improve clinical work ows. Within the MEDICO project, one of the selected use case scenarios aims for improved image search in the context of patients su ering of lymphoma in the neck area. Lymphoma, a type of cancer originating in lymphocytes, is a systematic disease with manifestations in multiple organs.
Generic medical image understanding is still a long-term agenda due to the
high complexity of the problem. Several challenging research questions need to
be addressed for tackling this vision. For determining the scope and level of
detail of the semantics of the domain, i.e. the relevant metadata for annotating
medical images, one needs to nd out what kind of knowledge the clinicians are
interested in. The scope of the constraint domain can be determined by the set of
derived query patterns [
There exist several approaches that analyzed to what extent tumor grading and
classi cation can be performed automatically using the OWL-DL description
logic language, such as [
[
OWL DL-based reasoning is also used in the context of other clinical use
cases. [
We are using the reasoning capabilities of OWL DL to provide means for the automatic classi cation of lymphoma patient. The enhanced patient annotation data can be used in a multitude of clinical applications, such as the recommendations for treatments, the search and visualization of similar patients, or clinical studies. In our future work, we are aiming to extend our knowledge model by the \B symptoms" of the Ann-Arbor classi cation system. Moreover, we will integrate our OWL DL based staging approach into the MEDICO system for enabling improved clinical applications.
Acknowledgements This research has been supported in part by the THESEUS Program in the MEDICO Project, which is funded by the German Federal Ministry of Economics and Technology under the grant number 01MQ07016. The responsibility for this publication lies with the authors.