The growing complexity of modern world and inherent limitations of human cognition create the need to offload the situation assessment and decision making to intelligent systems. The achievement of situation awareness in such systems presents a major challenge because it should include the patterns recognition and reasoning using knowledge about previous situations. The representation of knowledge using ontologies is widely used for reasoning implementation in intelligent systems. However, in systems with situational awareness the complexity of existing ontologies and the limitations on expressivity precludes their efficient usage, because decisions should be done in real time. In this article another approach, formulated within the well-known JDL model of situational awareness process is proposed. It is based on the usage of dynamically changing and small contextual, situational, task ontologies and contextual graphs. The data transformations occurring on each level of JDL model is described. The dependencies between different ontology types used on different levels of JDL model are specified. The obtained results can be used as a basis for situational awareness process.
1. Introduction • • • system;
We live in dynamic world where the problems arise spontaneously and unpredictably and are requiring from us to make quick decisions. Situational awareness and decision making based on this awareness are the important parts of human cognition. However, the growing complexity of modern world and inherent limitations of humans in making fast and correct decisions in complex situations, create a driving force to offload the part of situation assessment and decision making to intelligent, computer-based systems. The implementation of situation awareness in such systems has become a major challenge in the area of artificial intelligence.
The major factors, contributing to this challenge are: reasoning about them using conceptual knowledge; the need to combine both the perception and recognition of patterns in environment with the need to implement focusing and selection in perceptive part and in reasoning part of the policies and used knowledge models; current context; the decision making should be done in real-time. the highly dynamic nature of environment leads to the need of constantly updating focusing taking in consideration the contextual nature of knowledge, we need to quickly identify
2021 Copyright for this paper by its authors.
Moreover, an intelligent system able to function in real world environment need to manifest the full situation awareness process, including the perception on the world, interpreting its results, identifying situation, making decisions, acting and correcting knowledge based on feedback.
According to definition [
The concept of situational awareness was mentioned first during the First World War by the pilot
and military tactician Oswald Boelke, who argued the importance to realize the awareness of the
enemy before the enemy gets similar knowledge "[
The first studies on situational awareness as a part of decision-making systems were conducted for
military, aviation and other complex human-machine systems to support the activities of operators
[
In the process of situational awareness study multiple models were developed. They can be
classified as process, functional and formal models. The most known process model is John Boyd’s
Observe-Orient-Decide-Act (OODA) loop [
JDL model considers five levels in situational awareness process: • level 0. Signal/Feature assessment. On this level the signals from various sensors are gathered and interpreted as input data, corresponding to attributes of measured entities; The confidences of measured data are assessed; • level 1. Entity assessment. The data obtained are interpreted as attributes of entities from knowledge base; • level 2. Situation assessment. The entities involved in the current context and their relationships are analyzed in order to build the model of situation; • level 3. Impact assessment. Planning actions according to detected situations. Analyzing the impact of the situation; • level 4. Performance assessment. Evaluating the correspondence between current situation and system goals, performance analysis.
JDL model is constantly evolving and is revised. For example in [
However, while JDL model describes the process of situational awareness, it does not reveal how different types of knowledge are used and how they are processed in different levels of this model.
The accepted way to formally represent conceptual domain knowledge is to use ontologies – formal specification of conceptualizations.
Ontologies are widely used to represent knowledge in JDL model [
The assessing of the situation and making decisions are often required to be done in real time. Therefore, it is important to simplify the process and decrease the resources usage by adding constraints and reducing the number of concepts and relations from real world which should be taken into consideration.
The attempts to formalize domain conceptualization in form of an ontology usually produce large
ontologies. For example, Cyc ontology has hundreds of thousands concepts and more than
million rules [
The different aspects of ontology complexity are addressed in current research separately. One of
the important components of ontology complexity is structural complexity. In [
The early idea of counteracting the structural complexity of ontology was to split the general
ontology into parts according to generality/granularity of used concepts level. The reusable part
became top (upper ontology) and specialized part domain ontology. This created additional problem
of integrating top and domain ontologies [
One of the possible solutions for subjectivity in ontology creation is the use of well-researched sets of foundational ontology components. The effort of revealing and formalizing the most basic concepts and relations which can be reused consistently across multiple ontologies resulted in creation of foundational ontologies SUMO, UFO, and GFO [15, 16] and libraries of objects and patterns such as [17].
Another solution to the complexity problem is modularization and partitioning of ontologies. In [18] the large ontology is split into smaller modules using taxonomy structure. The authors of [19] propose extracting a module by specifying the part of larger ontology based on specific set of interrelated concepts, which form the signature of the module. The module defines the subset of knowledge, which can be used separately from original ontology for specified tasks.
The modern research in the domain of ontologies tend to express ontologies as set of smaller patterns and patterns groups [20], paving a way to the development of pattern languages [21].
Foundational ontology GFO [16] introduces the first class ontology object types to describe situations. Those classes are Situation, defined as “a special configuration which can be comprehended as a whole and satisfies certain conditions of unity, which are imposed by relations and categories associated with the situation” and Situoid, defined as “the processes with the boundaries – situations”. Situoid is a part of the world that is coherent whole and does not need other entities to exist. Every situoid is framed by object types specifying time characteristics (cronoid) and spatial data (topoid).
Another dimension, adding to the complexity of situational awareness implementation is the need to take into consideration different perspectives, goals and intents of agent and also context – as environment state influencing understanding and decision making. The same concept can have different conceptualizations depending on the situation, context and agent perspective, dictated by his intents.
The problem of formalizing contexts and reasoning within them has been widely researched. In [22] are specified the requirements that context modelling and reasoning techniques should meet, including the modelling of a variety of context information types and their relationships, of situations as abstractions of context information facts, of histories of context information, and of uncertainty of contextual information.
The article [23] considers the idea of recognizing contextual situations using process mining techniques. The authors propose to use fuzzy matching techniques for situation identification.
In [24] a Context-aware Decision Support (CaDS) system is described, which consists of a situation model for shared situation awareness modelling and a group of entity agents, one for each individual user, for focused and customized decision support. By incorporating a rule-based inference engine, the entity agents provide functions including event classification, action recommendation, and proactive decision making.
The analysis of existing research in the domain of situational awareness shows that, we cannot use a single ontology in situation awareness process. It would be very cumbersome, complex to handle and inflexible. We would rather use a network of simpler, dynamically constructed ontologies, reflecting the situation, context and the intent of agent in this situation. To correctly specify the situational knowledge and allow to make correct decisions, those ontologies should have elements with meanings adequately describing objects and conditions related to the situation.
This article intends to investigate how different forms of ontologies and knowledge models are used in the situational awareness process represented by JDL model. We will clarify the definitions and establish dependencies of different smaller ontological knowledge models such as contextual, situational, task ontologies, contextual graphs and where they are used in the situational awareness process.
First, let’s delineate the main premises of our research:
• As a basis we use the modified JDL model [
On the zero level of JDL model intelligent agent (Ag) is obtaining data from the world. This data is captured by sensors ( = 1 ÷ ) and relayed to situational awareness system for further processing. The amount of data read is constrained by agent’s intent and its properties (intrinsic limitations). The agent can focus on the part of the sensors, prioritize some sensors over others, controlling the amount of data read from specific sensor or sensor groups. This control can be also produced by feedback loops coming from the next levels of situation awareness process, shifting the focus of observation depending on current situation (Fig.1).
observes
has constrains has constrains
The Context (Con) is the part of the possible world (W) observed by agent. It can be understood as the environment perceived by the agent in the current moment. The context depends on agent’s location, intents, and state.
In the first stage of JDL model saw system obtains data from sensors and interprets them as attributes/parameters values of concepts from Contextual ontology. This ontology is a formal conceptualization of the part of the world corresponding to context, observed by agent. Contextual ontology is constructed as a result of combination (or extraction) from other domain ontologies, available to the agent, where perceived context and agent’s intent work as a selection filter defining what will be included in contextual ontology (Fig.2).
On this level the system receives a vector ̅ = ( 1, 2, … , ) of values read from sensors. As the result of processing those data a change in the fact information base ′ is generated. This change is comprised with changed facts, which include both concept and relation facts .
′ = ( 1, 2, … , ).
Each changed fact belongs to one or more classes (concepts) of Contextual ontology: ∀ : ( ) = { 1, 2, … , }, > 0, ∀ : ∈ ⊆ , (1) (2) where is a contextual ontology and is ontology (or set of ontologies) used for construction of . The ontology is a source of conceptual knowledge used to update Contextual ontology over time when the current context changes.
The data processing (mapping) performed on the first stage of JDL model is not straightforward. Currently to implement it we often use the neural networks algorithms to capture and recognize shapes and objects of specific classes from contextual ontology in the real world (such as automobile, a tree, a pedestrian, a cyclist etc) in the raw data flow produced by sensors. So we assume that the concepts instances are recognized first and their parameters in information base are changed second.
As a result of the first level entity assessment a conceptual model of current context (a scene) is formed using concepts and relations from the contextual ontology.
The context is constantly changing with the flow of time. Therefore, the contextual ontology and conceptual model of context is changing too adding new and discarding non relevant concepts and relations, updating facts identified in previous time moments. The task of identification of entities is simplified by reusing the entities identified in previous time moments, because in practice there are only small changes introduced in context from one moment to another. For example, if an automobile was identified in the near lane in the previous moment of time, there’s no need to identify it again, we can only change its parameters.
The conceptual model built in previous time moments can help to identify and reason about new objects introduced into Context, using the affinity of elements belonging to the current scene to simplify the detection of new elements. For example, when on the road we expect to see other vehicles, road signs, pedestrians, cyclists etc. While being in kitchen we are likely to see tables, kitchen utensils, an owen.
Therefore, the conceptual model of Context and corresponding contextual ontology is changing over time. However, for the long periods of time the differences between conceptual models in adjacent moments are minimal. Based on similarities between contextual ontologies we can discern typical Contexts. (environments) in which Agent is placed. In each of those Contexts different contextual ontologies are viable, different sets of situations can be detected, and different actions taken.
In the second level of JDL model a situation assessment is provided. We will understand a situation as a state of the world which presents some pragmatic (depending on agent’s values and goals) interest to agent (and therefore being worth to be identified). information base produced (Fig.3).
As an input on this level the current conceptual model of context is used. As a result, the set of identified situations
, = 1 ÷ } are stored as conceptual models – patterns of situations - in a separate repository. Each conceptual model of situation , is described by specific situational ontology
, , with classes, relations and axioms belonging to this situational pattern. Situations occur in current context. In order to be correctly identified there should be a mapping between subsets of elements from contextual and situational ontology: ∀ ∃1 : ( ) → ( , ).
Therefore, contextual and situational ontologies partly overlap and contextual ontology provide factual values for some elements of situational ontology. However, some elements from contextual ontology can be not present in situational ontology, being not relevant to the situation. On the other hand, situational ontology may contain references to data, not available in contextual ontology, such as historical, geographical, normative data. Those data should be obtained additionally from external sources in order to fully identify the situation.
Current context
limits the number of possible situations, which can occur in this context ,
.
One way to identify the set of possibly relevant situational patterns is to use the similarity between contextual and situational ontologies. However in practice, the look up into the list of possible situations is not efficient, especially if situation identification should be done in real time. The more efficient method is looking for cues, where cue is the combination of specific parameters values (or ranges of values) from the objects in contextual ontology, which can point to the probable situation (like symptoms point into possible disease).
= ( ,
( , )), where is a set of conditions, defined on the values of elements from contextual ontology, which should be true for the specified subset of situations ( , ) to occur.
While many different situations can be identified in current context it is important to prioritize situations as early as possible in the process of identification and focus resources on most important situations. Giving the large number of concepts, relations and their attributes in contextual ontology it is important to detect important cues and focus attention on them as early as possible.
In order to detect the important situation earlier the cues could be organized in hierarchies (or networks), where each parent node will encompass cue for multiple possible situations, child nodes will contain the specialization and elaboration of cues for more specific situations and arcs, leading to (3) (4) child node will be weighted with relative importance for corresponding situations, so they will be visited first in the process of analysis.
There could be different approaches to triggering the process of detecting cues in the process of constant monitoring of data state in information base: • based on anomalies, when compared to baseline values; • based on the known lists of situations to be checked in the current context; • using neural networks trained on real life cases to detect cues pointing to specific situations or groups of situations.
The baseline using approach monitors a set of important attributes and detects any deviation from baseline values. This detection can either point to specific situation pattern in the repository, or in initiate additional probing of parameters, and reasoning aiming to get more details in order to identify a situation.
The second approach is like running through checklist and is a standard operational procedure, for example, for surgery teams or airline pilots. Each typical context has associated checklists. Each checklist contains the list of conditions, specified using the values of attributes from contextual ontology. The deviations from those conditions points to specific situation or requires and additional investigation.
In the approach when situations are detected using neural network, this network is trained with real life situations examples to incorporate the cues and their importance into the fabric of neural network. The neural network approach is used in artificial intelligence applications. 3.4.
On the third level of JDL model, based on identified situation we have chosen to act upon, we make decision about required actions, specify those actions in more detail, and assess the impact of those actions’ execution.
As an input on this level the conceptual model of selected situation , situational ontology and the content of information base is used. As a result, the contextual graph = () as structure of linked context evaluation nodes and action nodes is produced (Fig.4).
Next, the Agent should make decision about the course of actions to undertake given the identified situation. Basically, this requires answering two questions: a) what change in the world is desirable given situation identified? b) how to introduce this change?
In most cases, once the situation pattern is identified, the set of possible decisions/actions is already known. This is the basis of Natural Decision Making (NDM) approach. In case, if there are several different alternative decisions available, we can use the methods of decision theory, such as analytical hierarchy analysis, to select the decision preferable according to specified criteria from several alternatives.
The goal of decision implementation can be specified as desired change of the state of the world, defined on the elements of situational ontology.
Let’s be the state of information base, which describes the state of the world, as the Agent sees it. We introduce the algebraic type of Goal with instances specifying the set of states of information base, where goal conditions are met.
( ) = ( ( )) = ̂ .
(5)
The goal function ( ) allow to find whether in specific state of information base the goal is met: ` ( ) = {
One of possible ways to define the goal function ( ) is to represent it as a list of assertions ( ( )) formulated on the state of information base , and which should be true and cover all required conditions for the goal: ( ) = (
( )).
The definition of goal function is also helpful on the later level of assessment whether the goal was reached as the result of actions performed.
The process of implementing the change according to specific goal depends on factual values of attributes in the conceptual model of situation. It can be formalized using contextual graph [26].
The model of context graphs was developed for the analysis of variants of processing of abnormal situations by subway operators [25]. The authors of this model note that usually the list of possible abnormal situations is well known and formalized in the company's contingency documents. However, the specific ways to resolve the abnormal situation depends on the conditions of its occurrence (context) and are determined by the human operator individually, based on experience. This method of solving the problem in [26] is called a practice. The task of context graphs is to explicitly define contextual data that influence the choice of a practice and present a sequence of possible practices for a particular situation indicating what information is used from the conceptual model of situation for each practice.
A context graph is an acyclic graph (Fig. 5) with one vertex-input and one vertex-output. The input vertex is the identified situation (with associated goal) that needs to be processed. A context graph has two types of vertices: context elements and actions. Contextual elements reflect contextual knowledge - that is, knowledge relevant to a given configuration of values from the instance of conceptual model of situation. The structure of this knowledge is specified by contextual element ontology − ⊆ which is a fragment of situational ontology. More precisely, the context element reflects relevant data, information and knowledge. The arcs of the graph emanating from the vertices of the context elements have marks that specify which part of the contextual knowledge is used by this or that practice (called procedural context). In addition to the context vertices that branch, reflecting different options for solving the problem, there are joining vertices, in which the different path of the process merge again.
After the course of actions is selected, those actions should be performed and results are evaluated against the goal, defined in previous level.
The processing on this level receives as an input the contextual graph , the ontology of situation , current state of information base . The actions, specified in contextual graphs are executed and resulting state of information base (after updating it with new facts) ′ is evaluated against the goal (Fig. 6).
The actions in contextual graph are specific tasks models. Each task corresponds to an elementary action, which is considered as a whole and not decomposed into smaller actions. With each a task’s goal is associated, which allows to check whether task execution was successful or not. The task knowledge is reused in multiple situations and presents a body of knowledge separate from the knowledge about contexts or situations. This knowledge is stored in the separate repository of tasks models.
The interest in formalizing task knowledge structure dates from the early ages of information technologies development, on the first work on this topic being [27]. This research started with investigation of task structures performed by humans, with a purpose to build better human-machine interfaces. Today the research on tasks knowledge shifts in direction of implementing task executing robots, and use of ontologies for task knowledge specification.
Each task type , is described by conceptual model of task , which, in turn is formulated based on the task ontology , .
In order to be instantiated and executed the conceptual model of task should receive parameter values from the instance of contextual element in contextual graph it is linked to. Therefore, there should be a mapping between the contextual element ontology and task ontology.
During the execution of contextual graph tasks, the constantly repeating action pattern is to evaluate the success of task execution by checking whether task goal was reached. This is done by collecting the information from the environment (Context) into the information base (Level 1 and 2) of the process and checking the state of information base against the goal conditions. The success/fail of task execution defines the changes the course of actions in contextual graph. For example, in data transmission failed, the system can retry transmission or renounce from it altogether. Such verification is also done for the contextual graph as a whole, after last action is successfully performed.
The results of actions from contextual graph may be not immediately manifested. Therefore, each goal can have a specified time delay after which the success of situation processing should be evaluated.
The evaluation of results is performed as a separate process, loosely connected with a history of previous situation identifications, and provides information for analysis and leads to eventual update of situation or task knowledge.
The implementation of situational awareness is a major challenge in the area of intelligent information technologies. To do this, we need to achieve the fusion of data, obtained from sensors and analog reasoning, such as neural networks with conceptual reasoning based on logic and models. The conceptualizations are formalized and processed using methods of ontology engineering. Due to large size of domain ontologies, the reasoning and decision making with them is quite limited and resource intensive. However, acquiring situational awareness should be done in real- time, because decisions and corresponding actions, depending on this awareness also should done quickly. In this article I argue, that situational awareness process should be based on small ontologies, relevant to current context and situations, which can occur in this context. The ontology- based situational awareness process is based on JDL model. The process uses contextual, situational ontologies and contextual graphs. The transformation, provided on each level of JDL model, using those knowledge structures is described.
The usage of smaller contextual ontologies allows reducing the size of model, used for decisions. From the analysis of elements included in contextual model, we can deduce the set of possible situations, and identify situations, based on cues, depending on the values of attributes of entities included in contextual model.
Since the context in real world keeps constantly changing, the contextual model and the set of possible situations also keep changing from one moment to another. However, the large part of context from previous moment tends to be preserved in next one. The research on dynamically changing context and correspondingly updates on contextual ontology present a promising area for future research.
Another promising trajectory of development is aimed at analysis, formalization, and development of implementation methods for changes based on feedback, coming from higher levels of JDL model to lower levels. Thus, we would be able, for example to update the selection of data from sensors, focusing the attention of system on important objects found during situation assessment level.
[15] G. Guizzardi, G. Wagner,. Using the unified foundational ontology (UFO) as a foundation for general conceptual modeling languages. In: Theory and applications of ontology: computer applications Springer, Dordrecht, 2010, pp. 175-196. [16] H. Herre.. General Formal Ontology (GFO): A foundational ontology for conceptual modelling.
In: Theory and applications of ontology: computer applications. Springer, Dordrecht, 2010, pp. 297-345. [17] C. Masolo, S. Borgo, A. Gangemi,N. Guarino, A. Oltramari, L. Schneider, The wonderweb library of foundational ontologies, 2002. [18] H. Stuckenschmidt, M. Klein, Structure-based partitioning of large concept hierarchies. In:
International semantic web conference. Springer, Berlin, Heidelberg 2004, pp. 289-303 [19] C.D. Vescovo, B. Parsia, U. Sattler, T. Schneider, The Modular Structure of an Ontology:
Atomic Decomposition. IJCAI, 2011. [20] G. Guizzardi, Ontological patterns, anti-patterns and pattern languages for next-generation conceptual modeling. In International Conference on Conceptual Modeling. Springer, Cham, 2014, pp. 13-27. [21] F. B. Ruy, R.A. Falbo, M.P. Barcellos, G. Guizzardi, G. K. Quirino, “An ISO-based software process ontology pattern language and its application for harmonizing standards”. ACM SIGAPP Applied Computing Review, 15(2), (2015): 27-40. [22] C. Bettini et al. "A survey of context modelling and reasoning techniques." Pervasive and Mobile
Computing 6.2 (2010): 161-180. [23] Z. Jaroucheh, X. Liu, S. Smith. “Recognize contextual situation in pervasive environments using process mining techniques”. Journal of Ambient Intelligence and Humanized Computing, 2(1),. (2011): 53-69. [24] Yu H. Feng, T.H.Teng, A.T. Hwee. “Modelling Situation Awareness for Context-Aware
Decision Support.” Expert Systems with Applications 36, no. 1 (2009): 455–63. [25] P. Brézillon. Task-realization models in contextual graphs Modeling and Using Context, 2005 pp. 1-8. [26] P. Brézillon. Elaboration of the Contextual Graphs representation: From a conceptual framework to an operational software. 2017. [27] P. Johnson, H. Johnson, R. Waddington, A. Shouls.. “Task-related knowledge structures: analysis, modelling and application”. BCS HCI (1988) 35-62.