=Paper=
{{Paper
|id=None
|storemode=property
|title=Towards the Ontology-based Classification of Lymphoma Patients using Semantic Image Annotations
|pdfUrl=https://ceur-ws.org/Vol-559/Paper2.pdf
|volume=Vol-559
|dblpUrl=https://dblp.org/rec/conf/swat4ls/Zillner09
}}
==Towards the Ontology-based Classification of Lymphoma Patients using Semantic Image Annotations==
https://ceur-ws.org/Vol-559/Paper2.pdf
Towards the Ontology-based Classification of
Lymphoma Patients using Semantic Image
Annotations
Sonja Zillner
Corporate Technology, Siemens AG
Munich, Germany
sonja.zillner@siemens.com
Abstract. Today, clinicians rely on medical images for screening, diag-
nosis, treatment planning and follow up, but still a generic and flexible
medical image understanding is missing. Although, there exist several ap-
proaches for semantic image annotation, those approaches do not make
use of practical clinical knowledge, such as best practice solutions or clin-
ical guidelines. Our final goal is to enhance medical image annotations
by integrating clinical knowledge, such as lymphoma staging systems.
The contribution of this paper is to introduce a formal approach to the
classification of patients in well-defined categories. As first step, we have
developed an OWL DL ontology representing the Ann-Arbor Lymphoma
staging system that is suitable for performing automatic patient classi-
fication. Our aim for the ontology design was to establish means for
automatic staging that maps each patient to one staging degree.
1 Introduction
The vision of MEDICO is to automatically extract the meaning from the med-
ical images and to seamlessly integrate the extracted knowledge into medical
processes, such as clinical decision making. In other words, the computer shall
learn to find, catalogue and interpret medical images. This requires the seman-
tic representation of medical images’ content and the preprocessing of semantic
image annotations for seamless integration into clinical applications.
There exist several approaches for semantic image annotation, such as au-
tomatic image parsing [1], manual image annotation [2][3], the extraction of
information from DICOM headers and DICOM structured reports [4], or the
automated extraction from radiology reports. Although those approaches pro-
vide the very important basis for semantic image annotation, they do yet not
make use of practical clinical knowledge, such as best practice solutions or clinical
guidelines for fine-tuning and customizing the established annotations to reflect
the particular requirements of a clinical application or work flow. Our final goal
is to enhance medical image annotations by integrating clinical knowledge, such
as lymphoma staging systems. The contribution of this paper is to introduce a
formal approach to the classification of patients in well-defined categories. Our
approach is based on external medical taxonomies and ontologies which promote
re-usability and interoperability. We have focused our study on the Ann-Arbor
staging classification of Hodgkin-lymphoma as defined in [5] as external clinical
knowledge resource. Our aim for the ontology design was to establish means for
an automatic staging system that maps each patient to uniquely one staging de-
gree. We, thus, can automatically generate additional patient annotation data,
that can be used for supporting the clinicians in their daily tasks. By integrating
the patient’s staging information, into other clinical applications, for instance,
the clinical work-flow can be optimized or the search and comparison of patients
be improved. We have developed an OWL ontology that represents the Ann Ar-
bor staging system together with lymphoma patient records and that is suitable
for performing automatic lymphoma patient classification.
The remainder of the paper is organized as follows. In Section 2 we will
introduce the knowledge resources our approach is based on. Section 3 details
challenges we faced and design decisions we made when developing the ontol-
ogy for lymphoma patient classification. In Section 4 we will discuss related
approaches and Section 5 concludes this paper with an outlook on future work.
2 Knowledge Resources
Ann Arbor Hodgkin Lymphoma classification Ann Arbor staging (Fig. 1)
is the staging system for lymphomas. It was initially developed for Hodgkin’s
Lymphoma, but has some use in Non-Hodgkin lymphomas. The stage depends
on two criteria. The first criterion is the place where the malignant tissue is
located. The location can be identified with located biopsy as well as with med-
ical imaging methods, such as CT scanning and increasingly positron emission
tomography. The second criterion are systemic symptoms, such as night sweats,
weight loss of more than 10 percent or fevers, caused by the lymphoma. Those
systemic systems are called “B symptoms”.
The principal stage is determined by the location of the tumor and reflects
the grade of expansion of lymphoma occurrences. Four different stages are rec-
ognized: Stage I indicates that the cancer is located in a single region, either an
affected lymph node or organ within the lymphatic system. In Stage II the cancer
is located in two separated regions, an affected lymph node or an affected organ
within the lymphatic system and a second affected lymph node area. Moreover,
the affected areas are confined to one side of the diaphragm - that is, both are
above the diaphragm or, both are below the diaphragm. Stage III indicates that
the cancer has spread to both sides of the diaphragm, including one extra lym-
phatic organ or site. Stage IV shows diffuse or disseminated involvement of one
or more extra lymphatic organs.
Ontological Knowledge Resources For capturing the semantics to the classi-
fication of lymphoma patients and for achieving re-usability and interoperability,
we required third party taxonomies or ontologies that cover all possible regions
of lymphatic occurrences. Two ontologies, the Foundational Model of Anatomy
Fig. 1. Ann Arbor Staging System (Source: http://training.seer.cancer.gov)
(FMA)[6] and the Radiology Lexicon (RadLex)1 , provide the required coverage
of anatomical concepts for the staging scenario.
The FMA is developed and maintained by the School of Medicine of the
University of Washington and the US National Library of Medicine. Besides the
specification of an anatomy taxonomy, i.e. an inheritance hierarchy of anatomi-
cal entities, the FMA provides definitions for conceptual attributes, part-whole,
location, and other spatial associations of anatomical entities. FMA covers ap-
proximately 70,000 distinct anatomical concepts and more than 1.5 million rela-
tions instances from 170 relation types. It provides concepts that describe single
lymph nodes, such as ’axilliary lymph node’, and concepts that describe multiple
lymph nodes, such as ’set of axilliary lymph node’. It contains 425 concepts rep-
resenting singular lymph nodes and 404 concepts describing sets of lymph nodes.
The FMA is freely available as a Protégé 3.0 project or can be accessed via the
Foundational Model Explorer. There also exist conversions of the frame-based
Protégé version of FMA to the OWL DL format [7][8].
RadLex is a terminology developed and maintained by the Radiological Soci-
ety of North America (RSNA) for the purpose of uniform indexing and retrieval
of radiology information, including medical images. RadLex contains over 8,000
anatomic and pathologic terms, also those about imaging techniques, difficulties
and diagnostic image qualities. Its purpose is to provide a standardized ter-
1
www.radlex.org
minology for radiological practice. RadLex is available in English and German
language and covers 144 concepts describing lymph node concepts.
As the use case scenario relies only on high-level concepts, both ontologies
- FMA and RadLex - are suitable in terms of coverage and both ontologies are
used within the MEDICO project for annotating medical images and radiology
reports. For this use case, we decided to use RadLex as primary third party
resource for labeling lymphatic occurrences. As we are working with radiology
reports in German language, the availability of a German translation was an
crucial argument for this decision. Due to the large size of Radlex, we needed
to establish an ontology fragment that is scalable and efficient for reasoning
application. Moreover, we transformed the ontology fragment into an appropriate
OWL DL format. The manually created ontology fragment covers all RadLex
concepts describing lymph nodes as well as a list of likely regions for extra nodal
lymphatic occurrences covering concepts such as liver, skin, or bone.
Semantic Image Annotation The MEDICO project is based on multiple
ways of generating semantic image annotations. For instance methods for auto-
mated image parsing, such as [1], allow hierarchical - in terms of starting with
the head and subsequently moving down the body - parsing whole body CT
images and efficiently segment multiple organs taking contextual information
into account. While automated image parsing remains incomplete, manual im-
age annotation remains an important complement. In addition, ongoing work in
MEDICO project is focusing on the semi-automatically identification of terms
and relations in radiology reports that are generated by clinicians in the pro-
cess of analyzing the patient’s disease patterns by investigating medical imaging
data. The extracted knowledge always relates to a particular medical image and
can be used for the generation of further image annotation data.
3 Ontology-based Classification of Lymphoma Patients
Our aim is to establish an ontology that enables the automatic classification of
lymphoma patients by means of ontology-based reasoning services. For doing
so, we transformed the Ann Arbor Hodgkin Lymphoma classification [5] into a
formal and ontology-based representation suitable for reasoning and classifica-
tion tasks. For capturing the requirements of the ontology design, we followed
the formal approach of [9]. We used OWL DL – MEDICO’s agreed semantic
representation language – for representing the knowledge model. Description
Logics, a family of formal representation languages for ontologies, are designed
for classification-based reasoning. We developed two different version of ontolo-
gies; the first ontology model represents all patients as classes and the second
all patients as instances.
In a first step, the Ann Arbor Hodgkin Lymphoma Classification was trans-
lated into complex questions for the subsequent decomposition into more simple
and manageable queries. Both, the complex questions as well as the simple ques-
tions, provided us valuable guidance for evaluating the ontological commitment
that has been made. This was possible by systematically establishing test pa-
tient classes and respectively patient instances according to the simple and the
complex questions.
3.1 Decomposition of Queries
As already mentioned, the stages of the Ann Harbor Staging System classify
lymphoma patients in accordance to their number and location of lymph node
occurrences and extra lymphatic organ or site involvement. For accessing the
number of lymph node occurrences and respectively extra lymphatic organ or
site involvement, the following queries, i.e. defined OWL classes, were defined:
– The OWL defined Classes N0, N1 and N2 identify all patients with zero, one
or two and more involved lymph node regions.
– The OWL defined Classes E0, E1 and E2 identify all patients with zero, one
or two and more involved extra lymphatic organ or site involvement.
– The OWL defined Classes N AllAboveD and N AllBelowD identify patients
with the location of the lymph node regions either only above or only below
the diaphragm. For accessing patients with occurrence of lymph node on
both sides of the diaphragm, one has to make sure that the occurrences
neither are located all above or located all below the diaphragm. This can
be formulated by the following axiom:
¬ N AllAboveD ⊔ ¬ N AllBelowD
– The location of extra nodal occurrences are identified in an analogous man-
ner, i.e. by establishing the defined classes E AllAboveD and E AllBelowD,
as well as the corresponding complex axiom for accessing patients with extra
nodal occurrences on both sides of the diaphragm.
In a further step, the above established simple or auxiliary queries could be used
for specifying defined classes deducing the Ann Harbor Stages. Before detailing
the staging classification, we will discuss the particular requirements towards the
ontology design for realizing the auxiliary queries.
3.2 Implications and Requirements for the Ontological Model
For establishing the ontological model basically three different challenges needed
to be addressed:
1. How to ensure that the ontological model provides the basis for inferring
the number of lymph node occurrences and the number of extra lymphatic
involvements?
2. OWL DL is based on the Open World Assumption (OWA), thus, it is only
applicable to a limited extent for checking the validity of closed sets of in-
formation. Moreover, the patient diagnosis deals with increasing set of infor-
mation. How to represent the patient staging classification dealing with an
open set of information in combination with the OWA paradigm?
3. How to identify the relative location – above, below or on both sides of the
diaphragm – of the lymphatic occurrences?
Counting Lymphatic Occurrences Although in most programming lan-
guages counting items is quite straight forward, this is not the case for OWL
ontologies. Counting lymphatic occurrences requires some preparation to achieve
the intended classification. A reasoner is only able to count concepts that are
different. To provide the basis for counting the number of involved regions, the
concepts for lymph nodes and for extra nodal lymphatic regions in the estab-
lished RadLex fragment need to be labeled as disjoint. Subsumption in OWL
means necessary implications, thus, classes and their subclasses can not be la-
beled as disjoint. Therefore, we decided to describe the parent-child relationship
of RadLex by introducing a transitive is a relationship and by using the following
type of axioms
Tr(is a) (𝑇 𝑟𝑎𝑛𝑠𝑖𝑡𝑖𝑣𝑒)
Right Hilar Lymph Node ⊑ (Anatomical Structure ⊓ ∃is a.Hilar Lymph Node
⊓∃hasNLocation.aboveDiaphragm)
..
.
Additional, we entered disjointness axioms indicating that the lymph nodes as
well as the extra nodal lymphatic regions are not overlapping.
Open World Reasoning with Increasing Patient Information Our aim
for the ontology design was to establish means for automatic staging that maps
each patient to uniquely one staging degree. Due to open world reasoning, this
could not be achieved directly. For instance, by establishing a defined class with
necessary and sufficient conditions stating the existence of exactly one lymph
node occurrence, e.g.
ExampleStage ≡ Patient(𝑥) ∧ ∃=1 𝑦 hasLymphaticOccurrence(𝑥, 𝑦)
the inferred ontological model will never classify patient classes as subclasses of
the ExampleStage. In open world reasoning, anything might be true unless it
can be proven as false. In the context of counting lymphatic occurrences, this
translates to: Any patient might have more than the indicated lymphatic occur-
rences unless it can be proven as false. For instance, a Patient A with one explicit
indicated lymph node occurrence can possibly have two or more lymphatic oc-
currences unless explicitly stated differently. Inferring Patient A as sub-class of a
staging class consisting of patients with exactly one lymphatic occurrence would
bear the risk of future inconsistencies. However, staging classes specified by a
minimum number of occurrences, such as ∃>=1 𝑦 hasLymphaticOccurrence(𝑥, 𝑦)
are suitable for OWL DL reasoning. But this again will cause, for instance, that
a patient with three indicated lymphatic occurrences on both side of the di-
aphragm is mapped to more than one staging class, i.e. stage I and stage III.
Our goal is to achieve the unique mapping between patients and staging classes.
This becomes possible by interpreting the highest staging class as the unique
mapping result. Yet, this interpretation is not explicitly stated in the derived
OWL DL model and has to be reflected when realizing clinical applications that
make use of the automatically inferred classification results.
The Location of Lymphatic Occurrences The patient staging results dis-
tinguish between patient that have lymphatic occurrences only above, only below
or on both sides of the diaphragm. Thus, the relative position of lymphatic oc-
currences to the diaphragm has to be expressed in the knowledge model. For
achieving this, we see two alternatives. The information about the relative posi-
tion of lymphatic occurrences can either be derived by the image segmentation
algorithm or be directly derived from the underlying ontological model. In other
words, this information either comes with the patient record information or with
the integrated anatomy and radiology knowledge model. The computation of rel-
ative spatial positions of lymphatic occurrences by segmentation algorithms will
be addressed within our future work. In the meantime, we have to rely on the
anatomy and radiology ontology for deriving this knowledge. RadLex and FMA
do not explicitly capture information about relative positions between lymphatic
regions or organs and the diaphragm. Thus, we required to enhance the RadLex
fragment accordingly. This could be achieved by extending each lymphatic re-
gion by an axiom indicating that the region is above, or respectively, below the
diaphragm. The classification of lymphatic regions - and of patients with lym-
phatic occurrences - above, below or on both sides of the diaphragm was modeled
as value partition.
3.3 Representation of the Ann-Arbor Staging Classes
The Ann-Arbor Staging classes are represented as defined OWL DL classes.
Each staging class is capturing the semantics as detailed in Subsection 2. Their
formal representation makes use of auxiliary classes introduced in Subsection 3.1.
Figure 2 summarizes the Ann-Arbor Staging Formalization. Thus, for instance,
Stage-II-N gathers all patients with more that two lymph node region that
are all on one side of the diaphragm and Stage-II-mixed all patients with one
involved lymph node regions and one involved extra lymphatic organ or site on
the same side of the diaphragm, and so on.
3.4 Evaluation
Evaluation is an ongoing topic. For evaluating the simple queries as well as the
complex staging queries, by hand we systematically generated the corresponding
test patient classes and respectively test patient instances. Each test patient is
equipped with different numbers and kinds of lymphatic occurrences at different
locations in the body. In this way, the evaluation (as well as the development)
of the classification axioms was straight forward.
Moreover - in our ongoing work - we are using real radiology reports for
challenging the capabilities of our ontology model. Our goal is to semantically
annotate more than 250 radiology reports that provide the basis for translating
Fig. 2. Formal Representation of Ann-Arbor Staging
the clinical findings - in terms of numbers and types of lymphatic occurrences -
into the OWL representation. As input we are using the semantic annotations
(see left side of Fig. 3) which is translated into OWL format (see right side of
Fig. 3), i.e. the class based OWL DL representation of a real patient with a large
number of lymph nodes who was classified as Stage-III.
One particular challenge we are facing is how to handle multiple lymph node
occurrences at the same location. For deciding how to tackle this problem, a
detailed analysis of the available patient record will be required.
Fig. 3. The semantic annotation of patient records by tool support (left) and the
corresponding OWL-based formal representation (right)
4 Related Work
4.1 Semantic Image Understanding
There exist a wide range of different imaging technologies and modalities, such as
4D 64-slice Computer Tomography (CT), whole-body Magnet Resonance Imag-
ing (MRI), 4D Ultrasound, and the fusion of Positron Emission Tomography
and CT (PET/CT) providing detailed insight into human anatomy, function
and disease associations. Moreover, advanced techniques for analyzing imaging
data generating additional quantitative parameters paving the way for improved
clinical practice and diagnosis. However, for advanced applications in Clinical De-
cision Support and Computer Aided Diagnoses the comparative exploration of
similar patient information is required. The missing link is a flexible and generic
image understanding. Currently, the large amounts of heterogeneous image data
are stored in distributed and autonomous image databases being indexed by key-
words without capturing any semantics. The vision of the MEDICO project is
to automatically extract the meaning from the medical images and to seamlessly
integrate the extracted knowledge into medical processes, such as clinical deci-
sion making, and to improve clinical workflows. Within the MEDICO project,
one of the selected use case scenarios aims for improved image search in the
context of patients suffering of lymphoma in the neck area. Lymphoma, a type
of cancer originating in lymphocytes, is a systematic disease with manifestations
in multiple organs.
Generic medical image understanding is still a long-term agenda due to the
high complexity of the problem. Several challenging research questions need to
be addressed for tackling this vision. For determining the scope and level of
detail of the semantics of the domain, i.e. the relevant metadata for annotating
medical images, one needs to find out what kind of knowledge the clinicians are
interested in. The scope of the constraint domain can be determined by the set of
derived query patterns [10][11] that provide guidance in identifying the relevant
(fragments of) ontologies [12]. Moreover, the low level features, segmentations
and quantitative measures derived from image processing need to be associated
with ontologies.
4.2 Formal approaches to the Classification of Patients
There exist several approaches that analyzed to what extent tumor grading and
classification can be performed automatically using the OWL-DL description
logic language, such as [13] aiming for the classification of lung tumors and
[14] for the classification of glioma tumors. Different kinds of tumors rely on
different kinds of staging systems. Whereas, lung and glioma tumors - similar to
the most tumor kinds - can be classified by the TNM classification, lymphoma
draws on a particular staging system. The so-called Ann-Arbor Classification
System for lymphoma depends on the number, type and location of lymphatic
occurrences. Thus, it raises different ontological design requirements, such as
counting lymphatic occurrences and determining the relative location.
[15] is similar to our approach, inasmuch it introduces an application that
provides support for the semantic annotation of medical images. Yet, the exter-
nal knowledge used for enhancing the semantic annotation and the application
focus are different. Our approach aims to integrate external clinical knowledge
for enhancing existing image annotation to optimize and improve clinical appli-
cations.
OWL DL-based reasoning is also used in the context of other clinical use
cases. [15], for instance, relies on the anatomy model and its regional relation-
ships for assisting the labeling of the MRI image content. Due to the facts ex-
tracted from MRI images, rather topological relations are capturing the required
knowledge. For representing and extracting the topological information, i.e. the
interdependencies of properties, reasoning with rules in combination with ontolo-
gies is required. [16] aims for improved and concise patient data visualization by
incorporation of medical ontological knowledge. The proposed solution uses an
OWL DL view of the patient database with external semantics allowing for the
patient record classification by a reasoner, from where the inferred hierarchy is
directly fed into an appropriate visualization tool.
5 Conclusion
We are using the reasoning capabilities of OWL DL to provide means for the
automatic classification of lymphoma patient. The enhanced patient annotation
data can be used in a multitude of clinical applications, such as the recommenda-
tions for treatments, the search and visualization of similar patients, or clinical
studies. In our future work, we are aiming to extend our knowledge model by
the “B symptoms” of the Ann-Arbor classification system. Moreover, we will
integrate our OWL DL based staging approach into the MEDICO system for
enabling improved clinical applications.
Acknowledgements This research has been supported in part by the THE-
SEUS Program in the MEDICO Project, which is funded by the German Federal
Ministry of Economics and Technology under the grant number 01MQ07016. The
responsibility for this publication lies with the authors.
References
1. Seifert, S., Barbu, A., Zhou, S., Liu, D., Feulner, J., Huber, M., Suehling, M.,
Cavallaro, A., Comaniciu, D.: Hierarchical parsing and semantic navigation of full
body ct data. In: SPIE Medical Imaging. (2009)
2. D. Channin, P. Mongkolwat, V.K.K.S.D.R.: The cabig annotation and image
markup project. In: Journal of Digital Imaging, online (2009)
3. Rubin, D., Mongkolwat, P., Kleper, V., Supekar, K., Channin, D.: Medical imaging
on the semantic web: Annotation and image markup. In: AAAI Spring Symposium
Series, Semantic Scientific Knowledge Integration, Stanford, USA (2008)
4. Möller, M., Regel, S., Sintek, M.: Radsem: Semantic annotation and retrieval
for medical images. In: of The 6th Annual European Semantic Web Conference
(ESWC). (2009)
5. Wittekind, C., Meyer, H., Bootz, F.: TNM Klassifikation maligner Tumoren.
Springer Medizin Verlag, Heidelberg (2005)
6. Rosse, C., Mejino, J.J.: A reference ontology for biomedical informatics: the foun-
dational model of anatomy. J. of Biomedical Informatics 36 (2003)
7. Golbreich, C., Zhang, S., Bodenreider, O.: The foundational model of anatomy in
owl: Experience and perspectives. Web Semantics: Science, Services and Agents
on the World Wide Web 4 (2006)
8. Noy, N., Rubin, D.: Translating the foundational model of anatomy into owl. Web
Semantics: Science, Services and Agents on the World Wide Web 6 (2008)
9. Uschold, M., Gruninger, M.: Ontologies: Principles, methods and applications.
Knowledge Engineering Review 11 (1996)
10. Wennerberg, P., Buitelaar, P., Zillner, S.: Towards a human anatomy data set
for query pattern mining based on wikipedia and domain semantic resources. In:
Proceedings of a Workshop on Building and Evaluating Resources for Biomedical
Text Mining (LREC), Marrakech, Marocco (2008)
11. Buitelaar, P., Wennerberg, P., Zillner, S.: Statistical term profiling for query pat-
tern mining. In: Proceedings of ACL 2008 BioNLP Workshop, Columbus, Ohio,
USA (2008)
12. Wennerberg, P., Zillner, S.: Towards context driven modularization of large
biomedical ontologies. In: Proceedings of the International Conference of Biomed-
ical Ontology (ICBO), Buffalo, New York, US (2009)
13. Dameron, O., Roques, E., Rubin, D., Marquet, G., Burgun, A.: Grading lung
tumors using owl-dl based reasoning. In: Proceedings of 9th International Protégé
Conference. (2006)
14. Marquet, G., Dameron, O., Saikali, S., Mosser, J., Burgun, A.: Grading glioma
tumors using owl-dl and nci-thesaurus. In: Proceedings of the American Medical
Informatics Association Conference AMIA. (2007)
15. Golbreich, C., Dameron, O., Bierlaire, O., Gibaud, B.: What reasoning support for
ontology and rules? the brain anatomy case study. In: Proceedings of the Workshop
on OWL Experiences and Directions, Irlande (2005)
16. Zillner, S., Hauer, T., Rogulin, D., Tsymbal, A., Huber, M., Solomonides, T.:
Semantic visualization of patient information. In: Proceedings of the 21th IEEE
International Symposium on Computer-Based Medical Systems (CBMS), Jyvskyl,
Finland (2008)