<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Risk Identification Ontology (RIO): An ontology for specification and identification of perioperative risks</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>GMC Systems mbH</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Institute for Medical Informatics, Statistics and Epidemiology (IMISE), University of Leipzig</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff3">
          <label>3</label>
          <institution>Jena University Hospital</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff4">
          <label>4</label>
          <institution>SurgiTAIX AG</institution>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Medical personnel in hospitals often works under great physical and mental strain. In medical decision making, errors can never be completely ruled out. Studies exposed that between 50 and 60 percent of adverse events could have been avoided through better organization, more attention or more effective security procedures. Critical situations especially arise during interdisciplinary collaboration and the use of complex medical technology, for example during surgical interventions and in perioperative settings. In this paper we present an ontology and an ontology-based software system which can identify risks across medical processes and which supports the avoidance of errors in the perioperative setting in particular.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>INTRODUCTION</title>
      <p>
        Patient safety is a quality target and an important factor of the
quality of treatment in hospitals in general
        <xref ref-type="bibr" rid="ref11">(“Empfehlung Zur
Einführung von CIRS Im Krankenhaus” 2007)</xref>
        . Prevention of
medical errors and risks is a significant method to improve patient safety.
Medical personnel often works under great physical and mental
strain. In medical decision making, errors can never be completely
ruled out
        <xref ref-type="bibr" rid="ref26">(Mahajan 2010)</xref>
        . In 2000, the report "To Err is Human"
        <xref ref-type="bibr" rid="ref24">(Kohn 2008)</xref>
        was published by the Institute of Medicine of the US
National Academy of Sciences (IOM). This attracted great
international attention and moved the topics of medical risks, errors and
patient safety into the focus of the scientific interest. The IOM
concluded in the report that from 2.9 to 3.7 percent of all patients
admitted to hospitals in the USA suffer an adverse event. In 70 percent
of these cases, the patient retains no or only minor damage, 7 percent
lead to permanent damage and 14 percent cause the patient's death.
The study also exposed that between 50 and 60 percent of these
adverse events could have been avoided through better organization,
more attention or more effective security procedures. Even for
Germany, analyses show that the number of medical errors is not
negligible. According to a report by the Robert Koch Institute
        <xref ref-type="bibr" rid="ref13">(Hansis et
al. 2007)</xref>
        , the incidence of suspected medical errors is approximately
40,000 cases across Germany per year. The error recognition rate of
about 30%, corresponds well to approximately 12,000 recognized
medical errors.
      </p>
      <p>
        Since the publication of “To Err Is Human”, risk management and
patient safety has consistently remained a topic of interest for
scientific studies as well as for suggestions of goals for improvements
        <xref ref-type="bibr" rid="ref9">(Bunting et al. 2016)</xref>
        . Critical situations arise especially during
interdisciplinary collaboration and the use of complex medical
technology, for example during surgical interventions and in
perioperative settings. Especially the oversight of medically relevant
treatment data or an incomplete medical history may cause an incorrect
treatment
        <xref ref-type="bibr" rid="ref2">(“Aus Fehlern Lernen” 2008)</xref>
        .
      </p>
      <p>
        We present an ontology and a conception for an ontology-based
software tool which can identify and analyze risks across medical
processes. Furthermore, the tool supports the avoidance of errors in
the perioperative setting. The results of the risk analysis are
transmitted to the medical personnel in form of context sensitive hints
and alerts. The software architecture is designed to respond not only
to risks within a single treatment step, but it also takes into account
the patient’s entire stay in the hospital. For a practical
implementation in the clinical environment, the cochlear implantation (CI) has
been selected as a surgical use case at Jena University Hospital.
Here, medical and technical treatment risks were analyzed, and
medical guidelines and standards were taken into account. In addition,
data and information sources have been defined on the basis of an
anonymized CI patient record. Further sources of critical events
were collected by undertaking of qualitative interviews with
technical, nursing and medical personnel participating in a CI. On this
basis, risk situations were defined and integrated into ontological
models. This work is a part of the BMBF-supported project
OntoMedRisk
        <xref ref-type="bibr" rid="ref29">(“OntoMedRisk” 2016)</xref>
        .
2
2.1
      </p>
    </sec>
    <sec id="sec-2">
      <title>METHODS</title>
      <sec id="sec-2-1">
        <title>Introduction in General Formal Ontology (GFO)</title>
        <p>
          The development of the intended ontologies and of the needed
ontological analyses are carried out within the top-level ontology GFO
          <xref ref-type="bibr" rid="ref14 ref15">(Herre 2010; Herre et al. 2006)</xref>
          . In GFO the entities of the world are
classified into categories and individuals. Categories can be
instantiated, individuals are not instantiable. GFO allows for categories of
higher order, i.e., there are categories whose instances are
themselves categories, for example the category “species”.
Spatio-temporal individuals are classified along of two axes, the first one
explicates the individuals’ relation to time and space, and the second
one describes the individuals’ degree of existential independence.
        </p>
        <p>
          Spatio-temporal individuals are classified into continuants,
presentials and processes. Continuants persist through time and have a
lifetime; they correspond to ordinary objects, as cars, balls, trees etc.
The lifetime of a continuant is presented by a time interval of
nonzero duration; such time intervals are called chronoids in GFO
          <xref ref-type="bibr" rid="ref5">(Baumann et al. 2014)</xref>
          . Continuants are individuals which may change,
for example, an individual cat C crossing the street. Then, at every
time point t of crossing C exhibits a snapshot C(t); these snapshots
differ with respect to their properties. Further, the cat C may lose
parts while crossing, though, remaining the same entity. The entities
C(t) are individuals of their own, called presentials; they are wholly
present at a particular time point, being a time boundary. Presentials
cannot change, because any change needs an extended time interval
or two coinciding time boundaries.
        </p>
        <p>
          Processes are temporally extended entities that happen in time, for
example a run; they can never be wholly present at a time point.
Processes have temporal parts, being themselves processes. If a
process P is temporally restricted to a time point then it yields a
presential M, which is called a process boundary of P. Hence, presentials
have two different origins, they may be snapshots of continuants or
process boundaries. There is a duality between processes and
presentials, the latter are wholly present at a time point whereas this is
never true for processes. The corresponding classes/sets of
individuals, denoted by the predicates Cont(x), Pres(x), and Proc(x), are
assumed to be pair-wise disjoint. Processes present the most
important kind of entity, whereas presentials and continuants are
derived from them. There are several basic relations which canonically
connect processes, presentials, and continuants
          <xref ref-type="bibr" rid="ref14 ref15">(Herre 2010; Herre
et al. 2006)</xref>
          .
        </p>
        <p>
          Spatio-temporal individuals, according to the second axis, are
classified with respect to their complexity and their degree of
existential independency. Attributives depend on bearers which can be
objects (continuants, presentials) and processes. Situations are parts
of reality which can be comprehended as a coherent whole
          <xref ref-type="bibr" rid="ref3">(Barwise
et al. 1983)</xref>
          . They are complex presentials and boundaries of
situoids, being processes which satisfy certain principles of coherence,
comprehensibility, and continuity. A surgical intervention is an
example of a process or a situoid. A snapshot of this situoid at a certain
time point is a surgical situation, which has spatial location and
includes various entities such that a coherent whole is established.
        </p>
        <p>There is a variety of types of attributives, among them, qualities,
roles, functions, dispositions, and structural features. Categories the
instances of which are attributives are called properties throughout
this paper. According to the different types of attributives (relational
roles, qualities, structural features, individual functions,
dispositions, factual, etc.) we distinguish quality properties (or intrinsic
properties) and role properties (extrinsic properties), and the role
properties are classified into relational role properties (abr. relational
properties) as well as social role properties (social properties).
2.2</p>
      </sec>
      <sec id="sec-2-2">
        <title>Ontological Definition of the Risk Notion</title>
        <p>The solution of all philosophical problems, related to the notion
of risk, is out of scope of this paper. Instead, we focus on a
practicable definition of the risk notion, which can be easily understood by
the medical staff and is usable for the software tools. Based on this
definition, it should be possible for the medical staff to specify the
relevant risk types, and for the software to identify and to analyze
the risk in a particular treatment situation.</p>
        <p>
          There are various definitions of the notion of risk. One of the most
known/popular definitions is that by
          <xref ref-type="bibr" rid="ref23">(Kaplan et al. 1981)</xref>
          . These
authors divide the notion of risk into three components which are
associated to the following questions:
1. What can happen, i.e., what can go wrong? (scenario)
2. How likely is it that that will happen? (probability of the
scenario)
3. If it does happen, what are the consequences? (consequence
of the scenario)
        </p>
        <p>
          A risk, then, is a triple which consists of a scenario, the probability
of that scenario, and consequence of that scenario. Furthermore,
there are several standards investigating the notion of risk. The
ISO/IEC 27005
          <xref ref-type="bibr" rid="ref20">(“Information Technology -- Security Techniques
- Information Security Risk Management” 2008)</xref>
          defines the notion
of risk as “a potential that a given treat will exploit vulnerabilities of
an asset or group of assets and thereby cause harm to the
organizations.”; the OHSAS 18001 (“OHSAS 18001 (Occupation Health and
Safety Assessment Series)” 2007) - as a “combination of the
likelihood of an occurrence of a hazardous event or exposure(s) and the
severity of injury or ill health that can be caused by the event or
exposure(s)”; and the ISO 31000 (Risk management)
          <xref ref-type="bibr" rid="ref30">(Purdy 2010)</xref>
          - as an “effect of uncertainty on objectives”. The common ground of
all these definitions is that all of them consider a risk as a possibility
for the occurrence of a particular event or situation. Most of these
definitions consider such events as adverse ones, whereas in the
standard ISO 31000 both adverse and positive events are admitted.
        </p>
        <p>
          The ontological analysis of risk is carried out within the
framework of GFO and takes into account the available definitions of risk.
The analysis is built upon the ontology of situations and situation
types, which partly uses ideas presented in
          <xref ref-type="bibr" rid="ref3 ref35">(Barwise et al. 1983;
Stalnaker 1986)</xref>
          . Situations which contain adverse events, being
related to a risk, are called adverse situations. In this paper we use the
notion of adverse event/situation not only in the sense of “Any
untoward occurrence that may present during treatment with a
pharmaceutical product but which does not necessarily have a causal
relation to the treatment”
          <xref ref-type="bibr" rid="ref10">(Edwards et al. 2000)</xref>
          , but we also include
events/situations that are not related to medical interventions. A
situation S is said to be a risk situation if it satisfies certain conditions
which imply that one of the possible succeeding situations of S is an
adverse situation.
        </p>
        <p>
          The notion of possible situation is established within the
framework of a particular actualist representationism, which postulates
that possible objects are abstract entities, the existence of which is
consistent with the currently available knowledge about the actual
world. This view is partly influenced by
          <xref ref-type="bibr" rid="ref1 ref33 ref38">(Adams 1974; Roper 1982;
Zalta 1993)</xref>
          .
        </p>
        <p>
          We hold that a risk exists in a situation, that it depends on it, and,
hence, that it can be considered as a situation’s property. We
distinguish between single (in sense of gfo:Property
          <xref ref-type="bibr" rid="ref14">(Herre 2010)</xref>
          ) and
composite properties, the latter being composed of single ones and
which can be disassembled by the relation gfo:has_part.
        </p>
        <p>Definition 1. A composite property CP is a property that has as
parts several single properties SP1, ..., SPn.</p>
        <p>Definition 2. A risk for an adverse situation of type AST is a
composite property CP such that every situation S possessing the
property CP has a possible succeeding situation of type AST which can
be realized with a certain probability.</p>
        <p>Definition 3. A risk is a composite property CP for which there
exists an adverse situation AST such that CP is a risk for the adverse
situation AST (as defined by 2).</p>
        <p>Definition 4. A risk situation is a situation having at least one risk
(Fig. 1).</p>
        <p>
          Example 1. The risk of a bacterial infection during cochlear
implantation in infants depends on various parameters, such as the
infants’ age, the corresponding bone thickness of the skull and the
inner ear structure. If the child is younger than 5 months, the bone
thickness mostly remains below 2 mm. Thus, the risk of penetrating
the skull and injuring the dura mater during surgery increases so that
the bacterial dura mater infection risk (meningitis) increases as well.
The ground-truth probability for the adverse event of dura mater
infection during CI is about 5-9%
          <xref ref-type="bibr" rid="ref31">(Reefhuis et al. 2003)</xref>
          . For
meningitis prevention the patient has to be vaccinated against
pneumococcus, meningococcus and haemophilus influenzae type b several
weeks before the surgery (indication phase). In addition, an
antibiotic prevention should be performed right before the surgery.
According to our definition an increased risk for acquiring meningitis
can be represented as a composite property, consisting of three
single properties, namely, the young age (&lt; 5 month), the absence of a
meningitis vaccination, as well as of an antibiotic prevention. This
example is used in this paper for further explanations.
        </p>
        <p>We developed a risk identification ontology (RIO, Fig. 2) which
is built upon the ontological model of the notion of risk. This
ontology is used for the specification and the identification of
perioperative risks. The ontology RIO is embedded into the GFO. As starting
point we consider the treatment process, which may possess various
treatment phases (gfo:has_part). The complete treatment as well as
the phases are complex processes (gfo:Situoid). The treatment has a
particular temporal extension, called the treatment time
(gfo:Chronoid). According to GFO processes are projected (gfo:projects_to)
onto its time intervals. For every time point (gfo:Time_boundary) of
the treatment exists (gfo:exists_at) exactly one treatment situation
(gfo:Situation). A treatment time point is according to GFO a
boundary of the treatment time (gfo:boundary_of), whereas the
corresponding treatment situation is a boundary of the treatment itself.</p>
        <p>For each treatment phase particular time points, called risk
detection time points (RDTP), can be defined. The treatment situations,
existing at these time points, are analyzed with respect to the
existence of risks. Such situations are called potential risk situations
(PRS), because they do not necessarily contain risks. Situations and
in particular treatment situations possess various properties
(gfo:Property). These properties may belong to the situation, but
also to the participants, as, for example physicians (doctors),
medical instruments, and, most important, to the patients. We consider
these properties also as properties of the current treatment situation
(gfo:has_property). Properties of the potential risk situations that are
relevant for the estimation of the risk are called in this paper KPIs
(Key Performance Indicators). According to Definitions 1-4 a
particular combination of a subset of the KPIs of a PRS (for example,
age of patient = 3 months, menginitis vaccination = false) represents
a risk if the PRS may lead to a later time point to an adverse situation
(rio:possible_succeeding_situation).</p>
        <p>A PRS may contain various risks, and risks of the same type may
occur in distinct PRS and may lead to distinct adverse situations
(rio:risk_for_adverse_situation). Each KPI is associated with
potential risk situations, whereas the risk situations additionally possess
the composite risk properties. Furthermore, the risks can be related
to those treatment phases for which they are relevant
(rio:risk_in_phase). Adverse situations may exhibit various degrees
of severity and risks may possess various probabilities for the
occurrence of adverse situations.</p>
        <p>With help of the RIO the risks in a current potential risk situation
are identified by the software component OntoRiDe, and, hence the
situation can be classified either as a risk or as a non-risk situation.</p>
        <sec id="sec-2-2-1">
          <title>3.2.1 Perioperative risk assessment</title>
          <p>
            For the development of a perioperative risk identification
ontology the recognition and assessment of potential medical, technical,
organizational and human risk factors are an essential prerequisite.
Therefore, an extensive risk assessment has been performed for an
otorhinolaryngological use case. The insertion of cochlear implants
(CI) was chosen in order to demonstrate the features and benefits of
the ontology-based risk identification system. The perioperative
medical and technical risk factors, procedure related complications
and their complication rates as well as prevention strategies were
extracted from peer-reviewed publications and evidence-based
bestpractice guidelines of the German Society of
Oto-Rhino-Laryngology, Head and Neck Surgery
            <xref ref-type="bibr" rid="ref25">(Lenarz et al. 2012)</xref>
            . In addition,
entries of the Critical Incident Reporting System (CIRS) of the
University Hospital Jena (Germany) and an example of an anonymized
patient record have been analyzed for organization and
humanrelated risk assessment. The derived risk characteristics, potential
following adverse situations and their causes were used to describe
relevant perioperative and cross-process risks factors.
          </p>
        </sec>
        <sec id="sec-2-2-2">
          <title>3.2.2 Perioperative process modeling</title>
          <p>The information of risk factors and of potentially adverse events
has to be provided to the responsible medical personnel in the right
time by offering appropriate context-sensitive hints and alerts.
Therefore, the medical and organizational processes have to be taken
into account. The general perioperative workflow of the CI
treatment was modeled and visualized in a process diagram, as
eventdriven process chain (EPC). In the following, both generalized and
use-case specific treatment phases have been defined in the formal
process model. The generalized treatment phases are depicted in Fig.
3. Besides the CI treatment process, the defined phases are suitable
for representing various elective surgeries and interventions.</p>
          <p>The treatment process was modeled by representing the sequence
of clinical activities, treatment decisions, parallel processes and
possible events, the involved persons as well as resources, like data and
documents, medical devices or IT systems. In addition, the
identified risk factors, complications and prevention activities were
integrated in the process model.</p>
          <p>By mapping the identified risk factors to the dedicated activities
and treatment phases, the process model has then been used
subsequently for further risk assessment and perioperative risk modeling.
This enabled over 120 potential perioperative risks to be identified
and also mapped to their related process step in the process model.</p>
        </sec>
        <sec id="sec-2-2-3">
          <title>3.2.3 Perioperative risks modeling</title>
          <p>In the next step the identified potential risk factors, adverse
situations and critical incidents, which are related to cochlear
implantation interventions, were examined in an extensive risk analysis.
Thereof, a risk classification for formal risk specification was
derived. The identified risk factors were subsequently classified into
different categories of medical, organizational, technical or
humanrelated risks. Thus, the treatment phases were categorized into risk
detection phases, in which the corresponding risk is relevant and
could potentially lead to an adverse situation. Additionally, there is
a category for cross-process risks, which could lead anytime to an
adverse situation, e.g. the risk of dizziness and falls or the high
bleeding risk during surgery due to anticoagulant medication.</p>
          <p>For each treatment phase different KPIs have been defined, which
allow the identification of specific perioperative risks. The KPIs are
linked with operators and a certain data type value to a conditional
expression of a possible risk factor (e.g., c1: Age_in_months IN [0,
5), c4: Vaccination_status == “no”, Fig. 4, Example 1). The KPI
data type values could be for instance a Boolean value, text, date or
number. A combination of these conditional expressions is
formalized as a risk specification rule. If the risk specification rule becomes
true, due to the values of their conditions and KPIs, there is a high
occurrence probability of adverse situations, which have to be also
specified for each risk. In addition, for each adverse situation an
occurrence probability and a severity (on a separate sheet) have been
defined. In the risk specification, the KPIs were described along with
their possible acquisition sources. Therefore, the risk specification
defines both the required measurement phases and the measurement
sources, like patient-related data and sensor data, e.g. data from the
digital patient record, the hospital information system, checklists or
situations in actual process execution. In Fig. 4 a risk specification
based on Example 1 is presented.</p>
          <p>The tool RIOGen, developed within the project, generates
ontological entities from the risk specification and inserts it into RIO.
For every risk condition, for example, a subclass of the
corresponding KPI is inserted. Here the class names are automatically
generated according to certain rules. For every condition class an
anonymous equivalent class is created as property restriction, based on the
property has_data_value (Fig. 5). Then, for the risk a subclass of
rio:Risk is defined, which is named as the risk. For the risk subclass
also an equivalent anonymous class is defined which is based on the
has_part property and on the corresponding condition classes; this
anonymous class represents the risk specification rule (Fig. 6).
Furthermore the treatment phases are created and connected with those
KPIs and risks which are relevant for them. Finally, we define the
connection between risks and those adverse situations, which
possibly evolve from them (incl. probability and severity as data property
restrictions).
3.3</p>
        </sec>
      </sec>
      <sec id="sec-2-3">
        <title>Ontology-based Risk Detector (OntoRiDe)</title>
        <p>We developed an ontology-based software module, called
Ontology-based Risk Detector (OntoRiDe), which allows the
identification of the ontologically specified risks. This tool receives the KPIs
of the current potential risk situation as input parameter, and carries
out the risk specification rule, which is contained in the ontology;
then it classifies the current situation as risk or non-risk situation and
returns the results. If the current KPIs satisfy one of the rules (i.e.,
at least one risk is recognized) then the considered situation is a risk
situation, otherwise it is a non-risk situation.</p>
        <p>Further information, which the tool returns to the user, includes
the description of the existing risks, the treatment phases, in which
the risks are relevant, but also the adverse situations which may
evolve from them (with probability of occurrence and degree of
severity). A particular position is the possibility to recognize the risks,
but, furthermore, to determine and provide for every recognized risk
all possible combinations of current KPIs which are responsible for
every recognized risk. Using this information the user is able to
eliminate all of the risks‘ causes.</p>
        <p>In the following we briefly sketch the functionalities of the
OntoRiDe. For every risk class the corresponding risk specification rule,
which is specified as an anonymous equivalent class (Fig. 6), is
interpreted and transformed into a disjunctive normal form (by
stepwise execution of the de Morgan rules and of the law of
distributivity). Any of the conjunctions presents a possible explanation for the
risk (e.g., c1 AND c4 AND c6, Fig. 4). Then, the single conditions
(Fig. 5) are checked, i.e., it is determined whether the current KPI
value is included in the specified value range. If all conditions of the
conjunction are satisfied, then the corresponding KPIs and further
information are provided for the user as explanation.</p>
        <p>
          We decided not to use a standard reasoner. Firstly, we want to
apply rules of types which cannot be easily interpreted by standard
reasoners, especially rules which contain mathematical expressions
or predefined constants. Such special types of rules are implemented
by the OntoRiDe. Secondly, standard reasoners carry out various
tasks (checking consistency, classification, and realization), not all
of them are relevant for risk identification, but which reduce the
efficiency of the overall system. Finally, OntoRiDe must provide the
user with all possible explanations about the existence of a risk in
the current situation in an understandable way. The problem of
detection and exploration of all possible explanations and justifications
of an entailment is a well-known task, for the solution of which there
exists several methods and tools,
          <xref ref-type="bibr" rid="ref19 ref22 ref32">(Kalyanpur et al. 2007; Horridge
et al. 2012; Riguzzi et al. 2013)</xref>
          . Furthermore, there are various
investigations about the cognitive complexity and the understanding
of the considered justifications
          <xref ref-type="bibr" rid="ref17 ref18">(Horridge et al. 2013; Horridge et al.
2011)</xref>
          . In this context a justification of an entailment is understood
to be “the minimal set of axioms sufficient to produce an entailment“
          <xref ref-type="bibr" rid="ref22">(Kalyanpur et al. 2007)</xref>
          . In the case of RIO and OntoRiDe the
solution is rather simple. The OntoRiDe translates the risk specification
rules into a disjunctive normal form and checks all conditions of the
respective conjunctions. By this procedure all KPI-combinations,
verified by the rule as true, and the corresponding conditions (value
ranges), can be provided for the user in form of understandable
explanations (e.g., age &lt; 5 month and vaccination = “no” and antibiotic
prevention = false).
3.4
        </p>
      </sec>
      <sec id="sec-2-4">
        <title>Agent System</title>
        <p>
          An agent system was developed to get access to the distributed
data in various systems in hospital needed to derive elementary
information for the risk detection. The KPIs mainly determine the data
which has to be captured by the agent system, respectively the
parameters which have to be monitored. Throughout the entire
perioperative treatment process the agent-based system retrieves
risk-relevant data from different data sources and provides these data for
further risk analyses in a centralized fashion. The results of such an
analysis will be transferred to medical experts as context-sensitive
hints and alerts. In doing so, continuous patient-specific risk
monitoring is facilitated for each treatment phase of the perioperative
treatment process. The OntoRiDe is an important component of the
agent system, because it determines the KPIs which have to be
monitored and it identifies the risks which have to be analyzed. This
reduces the risk of adverse situations and complications through early
and adequate interventions. The software-based agent system has
been implemented using the Java Agent Development Framework
(JADE), which embodies a framework, a platform and the
middleware for a FIPA-standardized development of multiagent systems
(MAS). The main functions of a JADE-based agent system can be
categorized into agent behavior and agent communication. The
agents communicate in an asynchronous, message-based fashion,
using the Agent Communication Language (ACL)
          <xref ref-type="bibr" rid="ref21 ref36">(“Jade: Java
Agent DEvelopment Framework” 2016; “The Foundation for
Intelligent Physical Agents” 2016)</xref>
          . The architecture of the agent system
consists of the OntoRiDe, a Blackboard, a Risk Analysis Unit and
various agents. The functionality of the agent system can be
separated into data acquisition and risk communication (Fig. 7). The
internal data storage of the agent system is based upon the
HL7-FHIRSpezification. Therefore, the data is represented as FHIR-Resources
          <xref ref-type="bibr" rid="ref12">(“FHIR: Fast Healthcare Interoperability Resources” 2016)</xref>
          .
4
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>RELATED WORK</title>
      <p>Several approaches towards the formal representation of risks and
adverse events through ontologies are described in the literature. We
analyzed these existing ontologies for their potential to detect
perioperative risks in hospitals, but we concluded that none of these
ontologies and tools could be applied to our project.</p>
      <p>
        Bouamrane et al.
        <xref ref-type="bibr" rid="ref6 ref6 ref7 ref7 ref8">(Bouamrane et al. 2010; Bouamrane et al.
2009a; Bouamrane et al. 2009b)</xref>
        report on the development of an
ontology-based system to support clinical decision making. The
support is provided in a two-step process. First, the developed system
calculates risk scores using numerical formulas. In this step, the
system does not use the developed ontology but computes numeric
values using an open-source Java-based rule engine (JBoss Rules).
After calculating the relevant risk scores, the DL reasoner (Pellet)
classifies the patient into a number of predefined categories for risks,
recommended tests and precaution protocols, using the OWL-DL
representation of the patient medical history profile and the decision
support ontology. The decision support ontology is divided into
three domains: a risk assessment ontology, a recommended test
ontology and a precaution protocol ontology. The aim of the risk
assessment ontology is to detect potential risks of intra-operative and
post-operative complications in a given formal representation of a
patient medical profile.
      </p>
      <p>Similar to the Bouamrane system, our approach also provides two
components of decision support namely OntoRiDe and Risk
Analysis Unit (Fig. 7). They can perform similar tasks as those of
Bouamrane’s system. In addition, OntoRiDe will also use the
selfdeveloped RIO for risk identification similarly to the usage of the
risk assessment ontology. However, there are also important
differences between the two ontologies and systems. The risk assessment
ontology focuses only on the patients risk related to intra-operative
and post-operative complications such as cardio-vascular and
respiratory risks, whereas RIO covers various risk types such as special
and general treatment risks, technical risks, organizational risks etc.
The second significant difference is that our approach integrates the
treatment process, its steps and situations in the risk
conceptualization. In this way, it is possible to analyze and identify cross process
risks or risk situations so that errors especially in the perioperative
field could be avoided.</p>
      <p>
        In
        <xref ref-type="bibr" rid="ref37">(Third et al. 2015)</xref>
        the authors describe a model for representing
scientific knowledge of risk factors in medicine. This model enables
the clinical experts to encode the risk associations between
biological, demographic, lifestyle and environmental elements and clinical
outcomes in accordance with evidence from the clinical literature.
The major advantage of our approach in comparison with the model
developed by Third is the formal representation of cross process
risks that can lead to potential adverse situations during different
treatment phases. Another added value of our approach is that it can
also cover risks related to human and environmental factors such as
technical or organizational risks. These types of risks are not
considered in Third’s model.
      </p>
      <p>
        <xref ref-type="bibr" rid="ref34">(Sigwarth et al. 2015)</xref>
        present an ontology of the Open Process
Task Model (OPT-Model). This ontology is primary intended as a
generic knowledge base, which implements the various influences
of processes and their relations in medical environments, for a
prospective risk analysis. The advantage of RIO over the
OPT-modelontology is that it provides an accurate risk analysis. By using RIO,
OntoRiDe is able to perform risks classification according to the risk
occurrence time. This process allows us to identify the time point
and treatment phase on which a risk arise. Another further benefit of
RIO is the implicitly embedded risk specification, which meets the
spirit of evidence-based medicine. This implicit domain knowledge
is encoded in OWL rules and can be inferred automatically using
ontological reasoning to assess current perioperative risk situations.
      </p>
      <p>
        <xref ref-type="bibr" rid="ref4">(Bau et al. 2014)</xref>
        report a clinical decision support system (CDSS)
for undergoing surgery based on domain ontology and rules
reasoning in the setting of hospitalized diabetic patients. Similar to our
approach this system uses logical rules to complement the domain
knowledge with implicitly embedded risk specification and clinical
domain knowledge. The important upside of our approach is that it
does not make restrictions based on certain diseases such as diabetes
mellitus, whereas CDSD focuses only on glycemic management of
diabetic patients undergoing surgery.
      </p>
      <p>
        The Ontology of Adverse Events (OAE)
        <xref ref-type="bibr" rid="ref16">(He et al. 2014)</xref>
        and the
Ontology of Vaccine Adverse Events (OVAE)
        <xref ref-type="bibr" rid="ref27">(Marcos et al. 2013)</xref>
        <xref ref-type="bibr" rid="ref27">(Marcos, Zhao, and He 2013)</xref>
        , which was developed based on OAE,
describe data relating to adverse events. The OAE was designed to
standardize and integrate data relating to adverse events that occur
after medical intervention. The OVAE is used for representing and
analyzing adverse events associated with US-licensed human
vaccines. In OAE the notion adverse event is defined as a pathological
bodily process that occurs after a medical intervention (e.g.,
following a vaccination), while a risk is represented by a factor associated
with the occurrence of an adverse event. The work presented here
focuses instead on the risk situations and proposes a generic model
for the risk specification in the perioperative area. Thus, we don’t
restrict ourselves to risks that are causally and exclusively related to
medical interventions. Contrary to OAE, our approach also
considers other risk types such as technical and organizational risks.
Moreover, we use the term “adverse situation” in order to avoid excluding
situations that are not related to medical interventions.
      </p>
      <p>None of the presented approaches can answer competency
questions such as “Which treatment situation could be a potential risk
situation?”, “Which properties or KPIs are responsible for an actual
risk situation?” and “Which risk situation belongs to which
treatment phase?”. The aim of RIO and OntoRiDe is to solve this issue.
5</p>
    </sec>
    <sec id="sec-4">
      <title>CONCLUSION AND FUTURE WORK</title>
      <p>We elaborated an ontological foundation of the notion of risk,
upon which we developed a risk identification ontology (RIO). With
help of RIO perioperative risks can be specified, whereas OntoRiDe
can be used to identify risks in a current treatment situation. This
allows the recognition of risk situations and supports the avoidance
of possible adverse situations. Furthermore, we conceptualized an
agent system which is currently implemented. This agent system
gathers during the whole perioperative treatment process
risk-relevant data from various sources and provides it for the risk
identification resp. the risk analysis in a centralized fashion. The results of
such an analysis are transmitted to the medical personnel in form of
context sensitive hints and alerts.</p>
      <p>We are currently working on the specification of risks. About 20
risks relating to cochlear implantation have already been specified,
and on this basis the functionality of RIO, RIOGen and OntoRiDe
successfully tested.</p>
      <p>Future work includes the conception of mathematical evaluation
methods and algorithms for the assignment of a risk to the current
process status and determination of the probability of occurrence.
The agent system will include risk communication features. In
particular, a Risk Analysis Unit for risk assessment (based on
probability and severity) and Cockpit component should be developed.
These components implement a role-based visualization of risk
information and of context-sensitive hints for the medical experts. In
the further development, this visualization should also be displayed
role-based on mobile devices. Furthermore, it is intended to expand
and to optimize the application of this agent system to other use
cases.</p>
    </sec>
    <sec id="sec-5">
      <title>ACKNOWLEDGEMENT</title>
      <p>This work was supported by the BMBF sponsored project
OntoMedRisk (FK: 01IS14022).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <surname>Adams</surname>
            ,
            <given-names>R. M.</given-names>
          </string-name>
          (
          <year>1974</year>
          ).
          <source>“Theories of Actuality.” Noûs</source>
          <volume>8</volume>
          (
          <issue>3</issue>
          ):
          <fpage>211</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          “Aus Fehlern Lernen.” (
          <year>2008</year>
          ). Aktionsbündnis Patientensicherheit e.V. http://www.aps-ev.de/fileadmin/fuerRedakteur/PDFs/Broschueren/Aus_Fehlern_lernen_0.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <surname>Barwise</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Perry</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>1983</year>
          ).
          <source>Situations and Attitudes</source>
          . Cambridge Mass.: MIT Press.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <surname>Bau</surname>
          </string-name>
          , C.-T.,
          <string-name>
            <surname>Chen</surname>
          </string-name>
          , R.-C., and
          <string-name>
            <surname>Huang</surname>
          </string-name>
          , C.-Y. (
          <year>2014</year>
          ).
          <article-title>“Construction of a Clinical Decision Support System for Undergoing Surgery Based on Domain Ontology and Rules Reasoning</article-title>
          .”
          <source>Telemedicine Journal and E-Health: The Official Journal of the American Telemedicine Association</source>
          <volume>20</volume>
          (
          <issue>5</issue>
          ):
          <fpage>460</fpage>
          -
          <lpage>72</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>Baumann</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Loebe</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Herre</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>“Axiomatic Theories of the Ontology of Time in GFO</article-title>
          .
          <source>” Appl. Ontology</source>
          <volume>9</volume>
          (
          <issue>3</issue>
          -4):
          <fpage>171</fpage>
          -
          <lpage>215</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <surname>Bouamrane</surname>
            ,
            <given-names>M.-M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rector</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Hurrell</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2009a</year>
          ).
          <article-title>“Development of an Ontology for a Preoperative Risk Assessment Clinical Decision Support System</article-title>
          .”
          <source>In 22nd IEEE International Symposium on Computer-Based Medical Systems</source>
          ,
          <year>2009</year>
          .
          <source>CBMS</source>
          <year>2009</year>
          ,
          <volume>1</volume>
          -
          <fpage>6</fpage>
          . Albuquerque, NM: IEEE.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <surname>Bouamrane</surname>
            ,
            <given-names>M.-M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rector</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Hurrell</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2009b</year>
          ).
          <article-title>“A Hybrid Architecture for a Preoperative Decision Support System Using a Rule Engine and a Reasoner on a Clinical Ontology.” In Web Reasoning and Rule Systems, edited by Polleres, A. and</article-title>
          <string-name>
            <surname>Swift</surname>
          </string-name>
          , T.,
          <volume>242</volume>
          -
          <fpage>53</fpage>
          . Lecture Notes in Computer Science 5837. Springer Berlin Heidelberg.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <surname>Bouamrane</surname>
            ,
            <given-names>M.-M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rector</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Hurrell</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>“Using OWL Ontologies for Adaptive Patient Information Modelling and Preoperative Clinical Decision Support</article-title>
          .
          <source>” Knowledge and Information Systems</source>
          <volume>29</volume>
          (
          <issue>2</issue>
          ):
          <fpage>405</fpage>
          -
          <lpage>18</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <surname>Bunting</surname>
            ,
            <given-names>R. F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Groszkruger</surname>
            ,
            <given-names>D. P.</given-names>
          </string-name>
          (
          <year>2016</year>
          ).
          <article-title>“From To Err Is Human to Improving Diagnosis in Health Care: The Risk Management Perspective</article-title>
          .
          <source>” Journal of Healthcare Risk Management</source>
          <volume>35</volume>
          (
          <issue>3</issue>
          ):
          <fpage>10</fpage>
          -
          <lpage>23</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <surname>Edwards</surname>
            ,
            <given-names>I. R.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Aronson</surname>
            ,
            <given-names>J. K.</given-names>
          </string-name>
          (
          <year>2000</year>
          ). “Adverse Drug Reactions: Definitions, Diagnosis, and Management.”
          <source>The Lancet</source>
          <volume>356</volume>
          (
          <issue>9237</issue>
          ):
          <fpage>1255</fpage>
          -
          <lpage>59</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <article-title>“Empfehlung Zur Einführung von CIRS Im Krankenhaus</article-title>
          .” (
          <year>2007</year>
          ). Aktionsbündnis Patientensicherheit e.V. http://www.apsev.de/fileadmin/fuerRedakteur/PDFs/AGs/07-07-25-CIRSHandlungsempfehlung.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <source>“FHIR: Fast Healthcare Interoperability Resources.”</source>
          (
          <year>2016</year>
          ). http://hl7.org/implement/standards/fhir/index.html.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <surname>Hansis</surname>
            ,
            <given-names>M. L.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Hart</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          (
          <year>2007</year>
          ).
          <article-title>Medizinische Behandlungsfehler in Deutschland. unveränd</article-title>
          . Nachdr. Gesundheitsberichterstattung des Bundes,
          <string-name>
            <surname>H.</surname>
          </string-name>
          <year>2007</year>
          ,
          <volume>05</volume>
          . Berlin: Robert-Koch-Inst.
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <string-name>
            <surname>Herre</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2010</year>
          ). “
          <article-title>General Formal Ontology (GFO): A Foundational Ontology for Conceptual Modelling.” In Theory and Applications of Ontology: Computer Applications</article-title>
          , edited by Poli, R.,
          <string-name>
            <surname>Healy</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Kameas</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <volume>297</volume>
          -
          <fpage>345</fpage>
          . Netherlands: Springer.
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <surname>Herre</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heller</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Burek</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoehndorf</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Loebe</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Michalek</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2006</year>
          ). “
          <article-title>General Formal Ontology (GFO): A Foundational Ontology Integrating Objects and Processes. Part I: Basic Principles (Version 1</article-title>
          .0).
          <source>” Onto-Med Report 8</source>
          . Research Group Ontologies in
          <source>Medicine (Onto-Med)</source>
          , University of Leipzig.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <surname>He</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sarntivijai</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Xiang</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jagannathan</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Toldo</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tao</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Smith</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>“OAE: The Ontology of Adverse Events</article-title>
          .
          <source>” Journal of Biomedical Semantics</source>
          <volume>5</volume>
          (
          <issue>1</issue>
          ):
          <fpage>29</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bail</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <source>“The Cognitive Complexity of OWL Justifications.” In The Semantic Web - ISWC</source>
          <year>2011</year>
          , edited by Aroyo,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Welty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Alani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            ,
            <surname>Taylor</surname>
          </string-name>
          , J.,
          <string-name>
            <surname>Bernstein</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kagal</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noy</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Blomqvist</surname>
          </string-name>
          , E.,
          <volume>241</volume>
          -
          <fpage>56</fpage>
          . Lecture Notes in Computer Science 7031. Springer Berlin Heidelberg.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bail</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>“Toward Cognitive Support for OWL Justifications.” Knowledge-Based Systems 53</article-title>
          (November):
          <fpage>66</fpage>
          -
          <lpage>79</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          (
          <year>2012</year>
          ).
          <article-title>“Extracting Justifications from BioPortal Ontologies</article-title>
          .”
          <source>In The Semantic Web - ISWC</source>
          <year>2012</year>
          , edited by Cudré-Mauroux,
          <string-name>
            <given-names>P.</given-names>
            ,
            <surname>Heflin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Sirin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Tudorache</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Euzenat</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Hauswirth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Parreira</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. X.</surname>
          </string-name>
          , et al.,
          <volume>287</volume>
          -
          <fpage>99</fpage>
          . Lecture Notes in Computer Science 7650. Springer Berlin Heidelberg.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <surname>“Information Technology -- Security Techniques -- Information</surname>
          </string-name>
          Security Risk Management.” (
          <year>2008</year>
          ). http://www.pqmonline.com/assets/files/lib/std/iso_iec_
          <fpage>27005</fpage>
          -
          <lpage>2008</lpage>
          .pdf.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <surname>“Jade: Java Agent DEvelopment Framework.”</surname>
          </string-name>
          (
          <year>2016</year>
          ). http://jade.tilab.com/.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <surname>Kalyanpur</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sirin</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          (
          <year>2007</year>
          ).
          <article-title>“Finding All Justifications of OWL DL Entailments.” In The Semantic Web</article-title>
          , edited by Aberer,
          <string-name>
            <given-names>K.</given-names>
            ,
            <surname>Choi</surname>
          </string-name>
          , K.-S.,
          <string-name>
            <surname>Noy</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Allemang</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>K.-I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nixon</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Golbeck</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , et al.,
          <volume>267</volume>
          -
          <fpage>80</fpage>
          . Lecture Notes in Computer Science 4825. Springer Berlin Heidelberg.
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          <string-name>
            <surname>Kaplan</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Garrick</surname>
            ,
            <given-names>B. J.</given-names>
          </string-name>
          (
          <year>1981</year>
          ).
          <source>“On The Quantitative Definition of Risk.” Risk Analysis</source>
          <volume>1</volume>
          (
          <issue>1</issue>
          ):
          <fpage>11</fpage>
          -
          <lpage>27</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <string-name>
            <surname>Kohn</surname>
          </string-name>
          , L. T., ed. (
          <year>2008</year>
          ). To Err Is Human:
          <article-title>Building a Safer Health System. 7. print</article-title>
          . Washington, DC: National Acad. Press.
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          <string-name>
            <surname>Lenarz</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Laszig</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2012</year>
          ). “
          <string-name>
            <surname>Cochlea-Implantat-Versorgung Und</surname>
          </string-name>
          Zentral-Auditorische Implantate.
          <article-title>” Clinical Practice Guideline. Bonn: German Society of Oto-Rhino-</article-title>
          <string-name>
            <surname>Laryngology</surname>
          </string-name>
          , Head and Neck Surgery.
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          <string-name>
            <surname>Mahajan</surname>
            ,
            <given-names>R. P.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>“Critical Incident Reporting and Learning</article-title>
          .”
          <source>British Journal of Anaesthesia</source>
          <volume>105</volume>
          (
          <issue>1</issue>
          ):
          <fpage>69</fpage>
          -
          <lpage>75</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          <string-name>
            <surname>Marcos</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhao</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>He</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>“The Ontology of Vaccine Adverse Events (OVAE) and Its Usage in Representing and Analyzing Adverse Events Associated with US-Licensed Human Vaccines</article-title>
          .
          <source>” Journal of Biomedical Semantics</source>
          <volume>4</volume>
          (
          <issue>1</issue>
          ):
          <fpage>40</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          <source>“OHSAS 18001 (Occupation Health and Safety Assessment Series)</source>
          .” (
          <year>2007</year>
          ). http://www.ircaglobal.com/EBMS_Demo_Published/OHSAS18 001_View/SE___18001_Terms_Definitions.htm.
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          “OntoMedRisk.” (
          <year>2016</year>
          ). http://www.ontomedrisk.de/.
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          <string-name>
            <surname>Purdy</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <source>“ISO</source>
          <volume>31000</volume>
          :
          <fpage>2009</fpage>
          -
          <article-title>-Setting a New Standard for Risk Management</article-title>
          .”
          <source>Risk Analysis: An Official Publication of the Society for Risk Analysis</source>
          <volume>30</volume>
          (
          <issue>6</issue>
          ):
          <fpage>881</fpage>
          -
          <lpage>86</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          <string-name>
            <surname>Reefhuis</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Honein</surname>
            ,
            <given-names>M. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Whitney</surname>
            ,
            <given-names>C. G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chamany</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mann</surname>
            ,
            <given-names>E. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Biernath</surname>
            ,
            <given-names>K. R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Broder</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          , et al. (
          <year>2003</year>
          ).
          <article-title>“Risk of Bacterial Meningitis in Children with Cochlear Implants</article-title>
          .
          <source>” New England Journal of Medicine</source>
          <volume>349</volume>
          (
          <issue>5</issue>
          ):
          <fpage>435</fpage>
          -
          <lpage>45</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          <string-name>
            <surname>Riguzzi</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bellodi</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lamma</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Zese</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>“Computing Instantiated Explanations in OWL DL.</article-title>
          ” In
          <source>AI*IA 2013: Advances in Artificial Intelligence</source>
          , edited by Baldoni,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Baroglio</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Boella</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            , and
            <surname>Micalizio</surname>
          </string-name>
          , R.,
          <volume>397</volume>
          -
          <fpage>408</fpage>
          . Lecture Notes in Computer Science 8249. Springer International Publishing.
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          <string-name>
            <surname>Roper</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          (
          <year>1982</year>
          ).
          <article-title>“Towards an Eliminative Reduction of Possible Worlds</article-title>
          .”
          <source>Philosophical Quarterly</source>
          <volume>32</volume>
          (
          <issue>126</issue>
          ):
          <fpage>45</fpage>
          -
          <lpage>59</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          <string-name>
            <surname>Sigwarth</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Loewe</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Beck</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pelchen</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Schrader</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>“Conceptual Ontology of Prospective Risk Analysis in Medical Environments - the OPT-Model-Ontology.</article-title>
          ”
          <source>In 2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS)</source>
          ,
          <fpage>88</fpage>
          -
          <lpage>93</lpage>
          . Cairo: IEEE.
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          <string-name>
            <surname>Stalnaker</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>1986</year>
          ).
          <article-title>“Possible Worlds and Situations</article-title>
          .
          <source>” Journal of Philosophical Logic</source>
          <volume>15</volume>
          (
          <issue>1</issue>
          ):
          <fpage>109</fpage>
          -
          <lpage>23</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          <article-title>“The Foundation for Intelligent Physical Agents</article-title>
          .” (
          <year>2016</year>
          ). http://www.fipa.org/.
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          <string-name>
            <surname>Third</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kaldoudi</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gkotsis</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Roumeliotis</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pafili</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Domingue</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>“Capturing Scientific Knowledge on Medical Risk Factors</article-title>
          .” In . Palisades,
          <string-name>
            <surname>NY</surname>
          </string-name>
          , USA.
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          <string-name>
            <surname>Zalta</surname>
            ,
            <given-names>E. N.</given-names>
          </string-name>
          (
          <year>1993</year>
          ). “
          <string-name>
            <surname>Twenty-Five Basic</surname>
          </string-name>
          Theorems in Situation and
          <source>World Theory.” Journal of Philosophical Logic</source>
          <volume>22</volume>
          (
          <issue>4</issue>
          ):
          <fpage>385</fpage>
          -
          <lpage>428</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>