Ontological Modelling of Situational
        Awareness in Surgical Interventions
Sebastian SIEMOLEIT a,1, Alexandr UCITELI a, Richard Bieckb and Heinrich HERRE a
 a
   Institute of Medical Informatics, Statistics and Epidemiology, University of Leipzig
        b
          Innovation Center for Computer Assisted Surgery, University of Leipzig


             Abstract. Optical navigation systems are the means of choice to overcome spatial
             association problems of the endoscopic imaging in minimally invasive surgery.
             Using optical markers, the patient's real position, his medical imaging data and
             surgical tool locations are mapped into the same workspace. Such visual-based
             assistance systems however, suffer from their technical requirements. The
             BIOPASS project aims to develop a navigation system based on a novel marker less
             localization method that uses only the current surgical situation and the procedure’s
             history to identify the present anatomy. The ontology, presented in this paper, plays
             an integral part in this system as it translates the situational information of a surgical
             procedure into an internal machine-readable representation. This representation
             combines multimodal sensor data, e.g. endoscopic images, endoscope movement or
             surgical work steps, to allow a classification of the apparent situation and provide
             navigation support based on identified anatomical landmarks and work steps.
             Furthermore, it is a foundation of situational awareness based on spatiotemporal
             reasoning.

             Keywords. Data streams, Endoscopic surgery, Formal ontology, Minimally
             invasive surgery, Situational Awareness



Introduction

Optical surgical navigation systems significantly reduce the cut-seam-time leading to
improved post-operative results [1]. However, training and experience is needed for the
registration process, and the overall optical marker setup is time-consuming and limiting
the nasal access path [2,3]. Furthermore, navigation systems are not a replacement for
surgical skills and anatomical knowledge. The BIOPASS project, therefore, develops a
novel localization approach for marker less navigation systems, to potentially reduce the
navigation hardware while assisting the surgeon's cognition with self-learning and
adaptive assistance [4]. The approach uses process and image databases of learnt surgical
procedures to intra-operatively identify anatomical landmarks. Novel sensors developed
in this project provide additional information, which further enrich classifier data. Thus,
the system creates multimodal data streams that we had to integrate into a unified view,
which allows interpretations grounding a situational decision support. An overall
description of the project's intention and architecture is given in detail in [5].
     The BIOPASS Situation Ontology (BISON) functions as data model that unifies the
apparent endoscope location and the current work step [6] in the context of an executed

1
 Institute of Medical Informatics, Statistics and Epidemiology, University of Leipzig, Härtelstrae 16, 04103
Leipzig, Germany; E-mail: sebastian.siemoleit@imise.uni-leipzig.de
surgical intervention based on traversable anatomical landmarks and corresponding
procedural data. BISON used the Foundational Model of Anatomy [7] as its domain
ontology according to the three ontology method [8]; according to which BISON is a
conceptual schema. Moreover, it holds the implementation of an axiom set, which leads
to situational awareness as needed by the domain experts, for which the system has been
tailored, as well as by the system itself to ensure its data integrity. Figure 1 outlines the
process of data stream classification and the subsequent reasoning tasks after which the
situations are saved into a situation database that extends the formerly mentioned process
database. The needed anatomical and procedural concepts have been implemented
prototypically for the use case of functional endoscopic sinus surgery (FESS). This work
was supported by the BMBF sponsored project BIOPASS (FK: 16SV7254K).




                Figure 1. Design of the system that has been developed in BIOPASS.



1. Methods

1.1. General Formal Ontology

The modelling of situational knowledge is carried out within the framework of the
General Formal Ontology (GFO) being developed at the University of Leipzig [9], the
basic features of which are summarized in the following. GFO provides an elementary
classification of the entities of the world and explicates primary relations between them.
The basic ontological distinction in GFO is between categories and individuals.
      Concepts are a special type of categories that have a close relation to language;
predicate forms, being expressions of a natural or formal language, describe them.
𝐶𝑜𝑛𝑡𝑖𝑛𝑢𝑎𝑛𝑡, 𝑃𝑟𝑒𝑠𝑒𝑛𝑡𝑖𝑎𝑙 and 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 are their categorization of individuals. A continuant
persists through time and has a lifetime, whereas a Process happens in time and is said
to have a temporal extension. A continuant’s lifetime is a process, thus, we consider
continuant as well as processes as being processual individuals. At any time point of this
lifetime, a continuant 𝐞𝐱𝐡𝐢𝐛𝐢𝐭𝐬 a uniquely determined entity, called presential, which is
wholly present at this time point.
      There is a basic classification of processes with respect to their structural
constitution in GFO. At two coinciding process boundaries, which are described in [10],
a 𝐷𝑖𝑠𝑐𝑟𝑒𝑡𝑒_𝐶ℎ𝑎𝑛𝑔𝑒 occurs within a process such that two properties instantiating the
same attributive are exhibited with different property values. A 𝐷𝑖𝑠𝑐𝑟𝑒𝑡𝑒_ 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 is
composed of discrete changes and states, which are processes without any change. A
𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑢𝑎𝑙_𝑟𝑜𝑙𝑒 is 𝐫𝐨𝐥𝐞_𝐨𝐟 some process and played by some individuals via 𝐩𝐥𝐚𝐲𝐬_𝐫𝐨𝐥𝐞,
for details see [11].
      A 𝑆𝑖𝑡𝑢𝑜𝑖𝑑 is a temporally extended part of the world, which can be understood as a
whole. A 𝑆𝑖𝑡𝑢𝑎𝑡𝑖𝑜𝑛 can be understood as the 𝐫𝐞𝐬𝐭𝐫𝐢𝐜𝐭𝐢𝐨𝐧_𝐨𝐟 a situoid to a timepoint. A
𝐜𝐨𝐧𝐬𝐭𝐢𝐭𝐮𝐞𝐧𝐭_𝐩𝐚𝐫𝐭_𝐨𝐟 a situoid (resp. situation) is an object involved in it. These notions
rely partially on the situation theory in [12].
2. Spatiotemporal Classification

An ontology, which is adequate for the given use case, must find a way to describe
spatiotemporally changing entities. To achieve such a representation, we utilized layers
as shown in figure 2. Since BISON has been implemented in OWL [13], we have to
distinguish between the instantiation of concepts according to GFO and the instantiation
of classes according to OWL. To avoid ambiguities, the first relation will be denoted by
the term 𝐢𝐧𝐬𝐭𝐚𝐧𝐜𝐞_𝐨𝐟, as it is defined in GFO, and the latter by the term 𝐞𝐥𝐞𝐦𝐞𝐧𝐭_𝐨𝐟.




             Figure 2. The layer structure of BISON and the relations between these layers.



     Knowledge shared between all components of the BIOPASS system is encoded in
the conceptual layer. The elements of the class 𝑆𝑢𝑟𝑔𝑖𝑐𝑎𝑙_𝑐𝑜𝑛𝑐𝑒𝑝𝑡 reflect process models
that specify the workflow of specific surgical interventions. A constituent 𝑎 of a surgical
concept 𝑏 is an element of the class 𝐴𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑐𝑜𝑛𝑐𝑒𝑝𝑡 and 𝑏’ s instances can occur
during an instance of 𝑎. The object property 𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐚𝐥_𝐬𝐩𝐚𝐭𝐢𝐚𝐥_𝐟𝐨𝐥𝐥𝐨𝐰𝐞𝐫_𝐨𝐟 relates an
anatomical concept 𝑎 to an anatomical concept 𝑏 if it is expected that all instances of 𝑎
are following some instance of 𝑏. Thus, this relation yields representations of surgical
process models as graphs. The descriptions of specific FESS interventions were analysed
to generate such conceptual graph, which, combined with BISON, is a task ontology.
     The processual layer holds all entities that are processual as defined earlier. The
elements of the class 𝑆𝑢𝑟𝑔𝑖𝑐𝑎𝑙_𝑖𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛 are situoids and specific to a patient on
which they are executed. If a surgical intervention 𝑎 is instance of a surgical concept 𝑏;
𝑎 and 𝑎′s parts are generated automatically according to 𝑏′s conceptual graph when a
surgeon has chosen to execute this kind of surgical intervention. Thus, surgical concepts
define templates for surgical interventions in general. The generated parts of a surgical
intervention are elements of the class 𝐴𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑜𝑏𝑗𝑒𝑐𝑡 being a subclass of 𝐶𝑜𝑛𝑡𝑖𝑛𝑢𝑎𝑛𝑡
and instances of anatomical concepts. The conceptual order of the corresponding
anatomical concepts is reflected via the object property 𝐬𝐩𝐚𝐭𝐢𝐚𝐥𝐥𝐲_𝐟𝐨𝐥𝐥𝐨𝐰𝐬.
     The presentic layer represents the content of endoscopical images and sensor data
provided by the system. Each element of the class 𝑆𝑢𝑟𝑔𝑖𝑐𝑎𝑙_𝑠𝑖𝑡𝑢𝑎𝑡𝑖𝑜𝑛 is a partially
reconstructed physical situation based on this data and it 𝐭𝐞𝐦𝐩𝐨𝐫𝐚𝐥𝐥𝐲_𝐟𝐨𝐥𝐥𝐨𝐰𝐬 a possibly
existing predecessor. A new situation is created if: (a) landmark changes have been
detected by the image processors, (b) the sensors detected a movement of the endoscope.
In case of (a), elements of the class 𝐴𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒 are generated and asserted to
be constituent parts of the surgical situation as wells as exhibited by an anatomical object
that is part of the executed surgical intervention. In case of (b), the direction in which the
endoscope has been moved is asserted, i.e. the situation 𝐩𝐚𝐫𝐭𝐢𝐜𝐢𝐩𝐚𝐭𝐞𝐬_𝐢𝐧 a discrete
process according to GFO, i.e. a 𝐹𝑜𝑟𝑤𝑎𝑟𝑑_𝑚𝑜𝑣𝑒𝑚𝑒𝑛𝑡 resp. 𝐵𝑎𝑐𝑘𝑤𝑎𝑟𝑑_𝑚𝑜𝑣𝑒𝑚𝑒𝑛𝑡.
3. Ontological Reasoning

During a surgical intervention, it is necessary to rule out incorrectly detected anatomical
structures. This function is implemented by the inference of movements presented in [14].
There, we presented conditions able to determine if a situation happened during a
forward resp. backward movement. We developed a specialized ontology design pattern
for temporally changing entities based on BISON and an axiomatization to express the
notions of forward and backward movement. Moreover, we introduced the OWL classes
𝑂𝑐𝑐𝑢𝑟𝑖𝑛𝑔_𝑎𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑜𝑏𝑗𝑒𝑐𝑡 and 𝑁𝑜𝑡_𝑜𝑐𝑐𝑢𝑟𝑖𝑛𝑔_𝑎𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑜𝑏𝑗𝑒𝑐𝑡 , the elements of
which are resp. are not constituent parts of the most current situation. Both are logical
concepts as defined in [15]. Thereby, BISON is able to reject incorrect information with
the help of the constraint that a situation can either participate in a forward movement or
in a backward movement. A detected anatomical structure that causes the dataset to be
inconsistent will not be a constituent part of the most current situation.
     The BIOPASS system will provide a decision support option that suggests the next
landmarks that have to be visited by the surgeon based on a classifier. However, it is
nearly impossible that statistical predictions will have full precision. Hence, BISON
infers all anatomical structures that can be visited in the following surgical situation to
enhance the classifiers’ precision further. An anatomical object having the disposition to
exhibit an anatomical structure that is a constituent part of the following surgical situation
is a 𝑁𝑒𝑎𝑟𝑏𝑦_𝑎𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑜𝑏𝑗𝑒𝑐𝑡, which is a logical concept and defined in Eq. (1).

 Nearby_anatomical_object ≡ Not_occuring_anatomical_object and spatially_follows
     some Occuring_anatomical_object                                                       (1)

     It is necessary to determine the role of anatomical structures that are visible in the
most current endoscopical image. The two most important roles in FESS are: (a)
landmark, which defines the need to be visited during a particular surgical intervention,
and (b) risk structure, which defines an easily damageable anatomical structure. If the
BIOPASS system can infer that an individual plays such processual role, it can display
them accordingly and show warning messages. We introduced the object properties
𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐚𝐥𝐥𝐲_𝐩𝐥𝐚𝐲𝐬_𝐫𝐨𝐥𝐞 and 𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐚𝐥_𝐫𝐨𝐥𝐞_𝐨𝐟 and used them to solve this problem of
role assignment.
     Assume an anatomical concept 𝑎 that categorially plays role 𝑏 , which is a
𝑅𝑜𝑙𝑒_𝑐𝑜𝑛𝑐𝑒𝑝𝑡 and categorial role of a surgical concept 𝑐. For each 𝑐 ′ that is an instance
of 𝑐, there will be role 𝑏′, which is an instance of 𝑏 and role of 𝑐′, and an anatomical object
𝑎′ , which is an instance of 𝑎 and plays 𝑏′ . By this definition, BISON can support
processual roles as part of the conceptual layer with corresponding subclasses of
𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑢𝑎𝑙_𝑟𝑜𝑙𝑒. Eventually, we could introduce the OWL class 𝐴𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑙𝑎𝑛𝑑𝑚𝑎𝑟𝑘
and 𝐴𝑛𝑎𝑡𝑜𝑚𝑖𝑐𝑎𝑙_𝑟𝑖𝑠𝑘_𝑠𝑡𝑟𝑢𝑐𝑡𝑢𝑟𝑒 as in Eq. (2) resp. analogous to Eq. (2). However, risk
structures are mostly not visible in an endoscopical image, i.e., if they are behind an
anatomical landmark. Thus, Eq. (3) is a secondary definition of this notion.

 Anatomical_landmark ≡ Anatomical_structure and exhibited_by some
     (Occuring_anatomical_object and plays_role some Landmark_object)                      (2)

 Anatomical_object and plays_role some Surgical_risk_object and spatially_follows
     some Occuring_anatomical_object
     ⊑ exhibits some Anatomical_risk_structure                                             (3)
4. Conclusion

In this paper, we introduced BISON, a situation ontology that implements a formalized
description of minimally-invasive surgical procedures based on situational information
extracted from endoscopic, procedural and sensory data. BISON utilizes ontological
layers, which are implicitly provided by GFO. These layers specify the conceptualization
of incorporated knowledge entities, their procedural characteristics, and, furthermore,
their situational manifestation. BISON was implemented in a web application module to
infer several process-related properties, e.g. anatomical landmark consecutiveness and
plausibility as well as endoscope movement. Subsequent work steps will include the
extension of situation concepts and their presentic manifestations as well as the
abstraction of reasoning output into a comprehensible, user-friendly verbalization of the
surgical situation. Furthermore, the elaboration of movements and direction is still in a
raw state and yields fundamental ontological problems that will be investigated in future
works.


5. References

[1]    Strauß G, Limpert E, Strauß M, Hofer M, Dittrich E, Nowatschin S, et al. Untersuchungen zur
       Effizienz eines Navigationssystems für die HNO-Chirurgie: Auswertungen von 300 Patienten,
       Laryngo-Rhino-Otologie. 2009;88(12):776–81.
[2]    Irugu DVK, Stammberger HR. A note on the technical aspects and evaluation of the role of navigation
       system in endoscopic endonasal surgeries, Indian Journal of Otolaryngology and Head & Neck
       Surgery. 2014;66(1):307–13.
[3]    Caversaccio M, Freysinger W. Computer assistance for intraoperative navigation in ENT surgery,
       Minimally Invasive Therapy & Allied Technologies. 2003;12(1-2):36–51.
[4]    ICCAS UL. BIOPASS: Bild-, Ontologie- und Prozessgestützte Assistenz für die minimal-invasive
       endoskopische Chirurgie. Available from: www.iccas.de/biopass.
[5]    Bieck R, Heuermann K, Schmidt M, Schmitgen A, Arnold S, Dietz A, et al. Towards and Information
       Presentation Model of a Situation-Aware Navigation System in Functional Endoscopic Sinus Surgery.
       In: Ansó J, Gerber K, Gerber N, Schwalbe M, Sznitman R, Weber S, Williamson TM, Wimmer W,
       Nabavi A, editors. 15. Jahrestagung der Deutschen Gesellschaft für Computer- und Roboterassistierte
       Chirurgie, September 29 - October 1, 2016, Bern, Switzerland. Der Andere Verlag; 2016. pp. 27–32.
[6]    O‘Grady MJ, O‘Hare GMP, Donaghey C. Delivering adaptivity through context-awareness, Journal of
       Network and Computer Applications. 2007;30(3):1007–33.
[7]    Rosse C, Mejino Jr JLV. The foundational model of anatomy ontology. In: Anatomy Ontologies for
       Bioinformatics. Springer; 2008. pp. 59–117.
[8]    Hoehndorf R, Ngonga Ngomo A-C, Herre H. Developing Consistent and Modular Software Models
       with Ontologies. In: The 8th International Conference on Software Methodologies, Tools and
       Techniques; 2009.
[9]    Herre H, Heller B, Burek P, Hoehndorf R, Loebe F, Michalek H. General Formal Ontology (GFO): A
       Foundational Ontology Integrating Objects and Processes. Part I: Basic Principles (Version 1.0).
       Onto-Med Report; 2006.
[10]   Baumann R, Loebe F, Herre H. Axiomatic theories of the ontology of time in GFO, Applied
       Ontology. 2014;9:171–215.
[11]   Loebe F. Abstract vs. social roles - Towards a general theoretical account of roles, Applied Ontology.
       2007;2(2):127–58.
[12]   Barwise J, Perry J. Situations and Attitudes. Mit Press; 1983.
[13]   Hitzler P, Krötzsch M, Parsia B, Patel-Schneider PF, Rudolph S, editors. OWL 2 Web Ontology
       Language: Primer. W3C Recommendation; 2009.
[14]   Siemoleit S, Uciteli A, Bieck R, Herre H. Processual Reasoning over Sequences of Situations in
       Endoscopic Surgery, Studies in Health Technology and Informatics. 2017;243:222–6.
[15]   Herre H. Formal Ontology and the Foundation of Knowledge Organization, Knowledge Organization.
       2013;40/5:332–9.