An important trend in the development of information technologies today is researching and creating artificial situation aware systems using the recent results from the field of artificial intelligence and knowledge-based systems.
Intelligent agents, capable of making autonomous decision-making, should be able to continually collect information about the environment, assess and model it according to their internal knowledge base and make quality decisions about actions to accomplish to further their goals. Internal knowledge base is based on conceptualization of the world, which is constantly tested, updated, and replenished using feedbacks from the assessment of results. This conceptualization of knowledge is formalized as an ontology.
However, the implementation of such autonomous intelligent agents has several substantial challenges to overcome. Firstly, to produce timely actions, situation assessment should be done in real time. However, taking in consideration that modern ontologies are very complex, reasoning and modeling using them quickly becomes computationally prohibitive.
Secondly, all knowledge is contextual, that is valid only in specific circumstances. The ability to recognize contexts, find and process context-related knowledge adds a new dimension to the complexity of the situational awareness implementation problem.
Finally, the dynamic nature of the environment requires constantly adapting the knowledge and checking its validity and the consistency.
Taking in consideration that a major part of agent's knowledge base is irrelevant to the current situation, the promising solution to these challenges could be the implementation of filtering, focusing mechanism into the process of situation awareness. It will select only a small, relevant to current context and situations part of ontology and build using it the conceptual models, which could be used in the process of making decisions.
This article explores the process of such attentional mechanism implementation using the approach of adaptive ontologies and weighted graph ontology representation. We also highlight the importance of prioritizing conceptual models used in higher levels of JDL/DFIG model and the resource monitoring process which controls the attention levels on multiple levels of situational awareness process.
The article has an introduction and six sections. In the background research section, we introduce the problem of situational awareness implementation in intelligent agents and models used to structure and understand the situational awareness process. Much of the knowledge used in situational awareness exist in the form of conceptual models, which are formulated using ontologies. We stress the problem of high ontology complexity, which leads to the high computing resources usage in one hand and the requirement for situational awareness system to make decisions and react to changes in real time in the other. Hence, we infer the need to introduce attentional mechanism in situational awareness system prioritizing the use of resources. Next, we look for insight into neurology, psychology and machine learning which have studied attentional mechanisms for a long time.
In the second section we describe how knowledge is prioritized on the first two levels of JDL/DFIG model and how contextual ontology is built. In the next section we describe attentional mechanisms on the higher levels of JDL/DFIG model. The performance monitoring section describes feedbacks provided to all lower levels of situational awareness system controlling the attention levels throughout the system. Finally, in the conclusion and discussion session we discuss possible ways of implementing the attentional mechanisms, the advantages of using prioritizing in conceptual modeling and reasoning.
According to the definition [1] situational awareness is the "conscious knowledge of the immediate environment and the events that are occurring in it. Situation awareness involves perception of the elements in the environment, comprehension of what they mean and how they relate to one another, and projection of their future states". Situational awareness research is filed under the more general topic of data fusion [2].
The process of situation awareness encompasses various kinds of operations, associated with being intelligent, such as selective perception of the environment, pattern and object recognition, situation identification based on previous experiences, reasoning, decision making and acting upon those decisions, assessing the success of actions, adapting knowledge and processes. The ultimate goal of situation aware system is to make decisions and adapt intelligent agent's behavior to the changing environment according to the goals of this agent.
Several models were developed to represent situational awareness process. They can be classified as process, functional and formal models. The early process models, such as John Boyd's Observe-Orient-Decide-Act (OODA) loop [3,4] or Predict-Match-Extract-Search loop [5] were developed as a generalization of real-world situation awareness processes in complex environments, such as battlefield. Functional models are represented by Endsley model [6], JDL (Joint Directors of Laboratories/ DFIG (Data fusion information group) [7][8][9] models. There are also the inquiries exploring different perspectives in situational awareness process using different formal frameworks such as Category theory, generalized information theory, interpreted systems, ontologies and specification languages. However, the most widely accepted framework of conceptualizing situational awareness process is functional JDL/DFIG model [10]. This model, as many other situational awareness process models follows the structure of human cognition process [11].
JDL/DFIG model splits the situation awareness process in five levels [10]: Evaluating the correspondence between current situation and system goals, performance analysis, forming feedbacks to lower levels of information fusion, updating models. JDL/DFIG model is constantly evolving and revised. Recently it is considered as a part of data fusion process -the process of integrating multiple data sources to produce more consistent, accurate and useful information [12].
The development of situational awareness systems requires the ability to build conceptual models of tasks, contexts, and situations. The basic vocabulary for such models is provided by ontologiesformal specifications of conceptualizations. Such conceptualizations aim to capture the knowledge about relationships between concepts in the domain.
Ontologies are routinely used to represent knowledge in JDL model [13,14]. The research about the application of ontologies in situational awareness often focuses on representing different approaches to reasoning, supporting situation identification and decision making in the context of JDL/DFIG model. For example, [14] proposes the enhancement of JDL model with processing complex events structures, building actionable abstractions from event streams and dependencies.
The attempts to formalize domain conceptualization in form of an ontology usually produce large ontologies. For example, Cyc ontology has hundreds of thousands of concepts and more than million rules [16], the number of concepts and WordNet has more than 117 thousand of synsets (synonym groups) [17].
Ontology complexity had different aspects which are addressed in current research separately. The important part of ontology complexity is structural complexity. In [18] an ontology is considered as a kind of a complex, compositional system. The author specifies the two main causes of such systems complexity. The first one is the nonlinear increase of the dependencies between elements, when the number of elements grows. The second one roots in the sensitivity of the whole system even to the minor change. Every such change in one element propagates to others, dependent objects and thus requires the review of the whole ontology.
The early idea of counteracting the structural complexity of ontology was splitting the general ontology into parts according to specificity of used concepts level. The reusable across domains part became top (upper ontology) and specialized part -domain ontology. This created additional problem of integrating top and domain ontologies [19]. Moreover, domain ontologies remained too structurally complex and redundant when resolving specific practical problems or used in specific business environments.
The effort of revealing and formalizing the most basic concepts and relations which can be reused consistently across multiple ontologies resulted in creation of foundational ontologies SUMO, UFO, and GFO [20,21] and libraries of objects and patterns such as [22].
Another solution to the complexity problem is modularization and partitioning of ontologies. In [23] the initial ontology is split into smaller parts using taxonomy structure. The authors of [24] propose extracting a module by specifying the part of larger ontology based on specific set of interrelated concepts, which form the signature of the module. The module defines the subset of knowledge, which can be used separately from original ontology for specified tasks.
The modern research in the domain of ontologies tend to represent ontologies as set of smaller patterns and patterns groups [25], paving a way to the development of pattern languages [26].
The complexity of ontologies makes the extensive reasoning using them computationally prohibitive [27]. That's why currently we are limited to simpler forms of logical reasoning with ontologies, based on less expressive dialects of description logic [28].
On the other hand, the assessing of the current context and making decisions are often required to be done in real time. Therefore, it is important to support dynamic changes in the ontology structure as well as to simplify this process and decrease the resources usage by adding constraints and reducing the number of concepts and relations from real world which should be taken into consideration.
There are several approaches, which could be useful in handling the changes in the ontology. First one is dynamic ontologies [29]. The promising approach in handling the changes in ontology is presented in Palantir platform [30] which allows to use weighted ontology components to reason on different specificity levels, reflect the importance of ontology components and integrate ontologies presented in different formats.
However, the dynamic ontologies consider the problem of ontology revisions in the process of its evolution and knowledge acquisition, which is quite different problem compared to situational awareness, where the importance of specific components of ontology is highly contextual.
Another one is adaptive ontologies [31], where concepts and relations used in the ontology reflect their importance in the real-life usage. Such approach can be used, for example, for deriving ontology from texts. In [32] the method for selection of important concepts and relations is proposed based on the automated weighting of concepts and relations during ontology learning. However, in the situational awareness systems the importance of ontology elements dynamically changes depending on the changes in the environment, agent's goals, and reasoning results. It is constantly updated via feedbacks from higher levels of JDL/DFIG model. It also depends on the current system load level and the importance of tasks processed by system currently. Taking in consideration the limited computing resources available to intelligent agent, the development of attentional mechanism, dynamically highlighting and prioritizing the ontology elements and larger knowledge aggregates, such as models and patterns is needed for efficient functioning of intelligent agent.
The valuable insights for creation of attentional mechanism in situational aware systems can be obtained from psychology and neurology, where attention was studied for a long time [33]. The purpose of attentional mechanism in both natural and artificial system is to prioritize information and tasks in face of data processing constraints.
Literature defines attention as "the concentration of awareness on some phenomenon to the exclusion of other stimuli" [34]. Another, verbose explanation states: "The term attention captures the cognitive functions which are responsible for filtering out unwanted information and bringing to consciousness what is relevant for the organism. ... Closely related to this aspect of selectivity is the assumption that the available quantity of attention is finite".
The most researched was the visual attention process, which is similar in structure to JDL/DFIG model, having the sensory part, object attention, reasoning, and executive control [35].
The author in [34] states, that despite the many attempts to precisely define and quantify the attentional process while also identifying the underlying mental and neural architectures that give rise to it, no one still knows what attention is. However, the modeling of human attention has recurrent findings, which can be reused in the creation of attentional mechanism in artificial situational aware systems. The useful insight, for example is provided by saliency maps used in visual attention modeling, allowing to detect the part of the picture which stand out from the background. Another one is that attention manifests in increasing the firing rates of neurons for attended features and suppressing rates for the rest.
Recently, the attentional mechanisms were proposed in Machine learning [35]. They mostly are implemented as the dynamic calculation of the weights of the nodes in artificial neural networks depending on the input data and previously calculated weights. Such mechanisms have given the system flexibility and efficiency in the condition of limited resources [35].
This article proposes the outline of attentional mechanism for situational aware system based on JDL/DFIG model and demonstrates how resulting attention is formed from feedbacks, coming from multiple levels of this model.
The main assumptions we make in our research are as follows.
1. We use JDL/DFIG model as a representation of situational awareness process.
2. This process is implemented by intellectual agent, which actively perceives the environment, has its own set of conceptual knowledge (represented as ontology), knowledge base and information base storing facts and knowledge in the form of rules, algorithms, and models.
3. An agent reasons about the world using conceptual models, which are built from the agent's own ontology. 4. Each agent is constantly learning using feedbacks coming from the assessment of results of its actions. This feedback is coming from higher levels activities in JDL/DFIG model. Thus, the knowledge base and ontology of an agent is constantly updated.
In the first stage of JDL model situational aware system obtains data from sensors and interprets them as attributes/parameters values of concepts from agent's ontology ππ. The objects and their relationships as observed by agent, form the conceptual model of the environment πΆπ πππ£ , described using the elements from ππ. The elements from πΆπ πππ£ form a part of the agent's domain ontology which can be represented as a smaller ontology ππ πππ£ β ππ which can be extracted from the ππ.
However, the sphere of agent's attention is not limited to observed world. It should also consider important objects and relationships deduced by reasoning processes, coming from agent's intents, goals and values, previous experiences. Therefore, contextual ontology ππ πππ , which includes all those relevant to current context components is larger than the ontology of perceived environment: ππ πππ£ β ππ πππ β ππ .
We will model contextual ontology as a part of agent's ontology selected by attentional mechanisms. With this purpose we will use the graph-based representation of ontology structure as a graph of concept-vertices π πππ and relationsedges πΈ πππ .
πΊ ππ = (π πππ , πΈ πππ ) We omit from this structural model the last component of ontology definition-axioms because we are not using them to model attentional processes.
To differentiate the importance of specific ontology components in the current context a weight functions assigns weight to every concept π£ πππ π The values of weights are constantly updated based on feedback signals coming from higher levels of JDL/DFIG model, changes in environmentnew objects perceived and recognized by system.
The attentional mechanism monitors the resources available and decides to increase or decrease the value of the threshold of attention πβ πππ . All ontology elements having weights below the threshold are ignored and system focuses on the elements being above the threshold. This is used to establish priority in reading information from corresponding sensors and getting additional information from them. This is also used in prioritizing access to reasoning, modeling operations, getting additional information from external knowledge bases.
On the second level of JDL/DFIG model a conceptual knowledge is processed, tasks executed, situations detected. To do this, system uses the knowledge about similar situations stored in the knowledge base, performs reasoning to get information for decision making, happening on the next level. Such activity uses a substantial amount of computing resources. Hence, the prioritization of tasks and attentional mechanism is required for efficient situational awareness system functioning on this level.
We will assume that only one conceptual model is processed in any given time. The parallel processing of conceptual models requires additional resources for coordination of model execution and adds another level of complexity. Thus, system is constantly switching between models belonging to the active set of models ππΆπ. Each model πΆπ π is assigned a weight π€ ππ π . Only models that have a weight above the model attention threshold πβ ππ are included in the working set:
βπΆπ π β ππΆπ: π€ ππ π > πβ ππ The models not included in working set are not processed. The models in working set obtain processor proportionally to their weight.
What kind of conceptual models are included in the working set? We will differentiate between: β’ Task models. An agent is performing various tasks. Tasks are chosen taking in consideration agent's intentions and goals (which can be represented as another model). Different agents, even being in the same environment, can perform different tasks. For example, a car driver attends to driving a car, while his passenger can attend to admiring the scenic route.
Situation processing models. If some specific situation was detected, agent activates model processing it.
Model monitoring conceptual model of context πΆπ πππ for cues and events which points to the situations, requiring actions. For the sake of simplicity, we omit other important processes, such as planning, goal setting etc. We assume that they are performed outside of the scope of situational awareness process.
All models take information from the information base and contextual model, populated with data coming from sensors and interpreted as objects, belonging to ontology types. Models also use knowledge coming from experience and accumulated in the knowledge base. They can also form queries to external knowledge bases, if needed information is missing in agent's knowledge base.
An agent rarely meets the current context for the first time. Typically, he has been in a similar condition before. The knowledge about similar contexts is stored in the knowledge base as prototypical context models. This knowledge comes from the previous experiences of the agent or is formed as a result of learning. According to the prototype theory [36,37] humans form the mental representation of concept as a fuzzy set of typical objects, belonging to the concept. This set has some central, core object and several exceptions, describing the deviations from the central object. Different concepts, formed in this way can be overlapping. Prototypical contexts are formed in the same way, as groups of similar contexts, where similar intents are followed in the similar environments. Prototypical contexts are stored in the knowledge base of the agent as a set of conceptual models ππΆπ π‘π¦π = {πΆπ π‘π¦π π } . With each prototypical context is associated the knowledge about tasks, practices, techniques which can be used in the current context and situations which can happen. On level 2 of JDL/DFIG model an agent connects to the relevant knowledge from its knowledge base by finding similarity between current conceptual model of context and prototypical context models. The identification of relevant prototypical context is a pattern recognition problem, which uses a similarity function πΉ π ππ (πΆπ π , πΆπ π ) which could be interpreted as a distance between two conceptual models. Therefore, this problem can be formulated as finding the prototypical model πΆπ π‘π¦π π which minimizes the value of similarity function:
The identification of prototypical modes reduces the computational load, because it limits the number of situations and cues to monitor or tasks to execute to the subset of models associated with identified prototypical model.
Once the relevant prototypical context model is identified, the conceptual context model πΆπ πππ is updated with knowledge from this prototypical model πΆπ π‘π¦π π :
β’ new concepts and relations are added. Those concepts may be not observed directly, but they are important for reasoning and decision making.
if needed, the values of newly added objects are populated by querying the agent's knowledge base or external information sources.
β’ the weights of components in πΆπ πππ is updated, reflecting the knowledge about importance of objects β’ a mapping between components of πΆπ πππ and πΆπ π‘π¦π π is established.
When the prototypical context is identified, the agent obtains access to the specification of situations which can occur in this context. Knowing the prototypical context reduces the number of situations to consider and analyze. Let's denote such set of situation specifications as πππ‘.
The knowledge about situation ππ‘ π , stored in knowledge base has such parts:
β’ conceptual model of situation πΆπ π ππ‘ specifying the relevant objects and their relationships, involved in the detection, analysis, making decision and projecting its impact. All elements of πΆπ π ππ‘ belong to the agent's ontology ππ.
Diagnostic information, containing conditions and procedures needed to detect the situation. An important part of this information is a set of cues ππΆπ’ πconditions to monitor in the environment which, if present, can point to the possibility of having a situation. This part of specification also contains possible variations (scenarios) in situation development, requiring additional analysis and different courses of actions. This section also contains the assessment of possible risks for different variants of situation.
Decision making and impact assessment procedures which process the results of situation analysis. This section contains information used on the next, third level of JDL/DFIG model.
Actionable information about what should be done, once the decision is made, task models, which describe the task execution, success/fail conditions, expected results, feedback actions depending on the result of task execution. The review of situation specifications can add additional elements to conceptual model of context, especially if there's high risk, or high impact on agent's goals associated with this situation. Such elements are included in cues. For example, the car driving system can choose to monitor information about data traffic jams or closed roads ahead.
Because of complexity in situation definitions and processing procedures it is viable to monitor the context and corresponding conceptual model only for cues pointing for situations and assign resources to processing the situation only when this situation was detected.
The process monitoring the contextual model πΆπ πππ for cues takes the list of cues πΏπΆπ’ = (πΆπ’ π ) ordered according to their weights π€ ππ’ π . The weights are assigned to cues depending on the likelihood of corresponding situation occurrence and risk associated with it. The attentional mechanism only considers cues with a weight above corresponding threshold πβ ππ’ .
The agent's environment and corresponding conceptual model are constantly changing, because of external events occurring in it. Therefore, the list of situations is updated too, taking in consideration the situations involving new objects and removing situations with objects which disappeared from the conceptual model.
Attentional mechanism on level 2 dynamically reassigns weights to models, belonging to the working set. The values of weights for all models in working set are normalized, so if one model obtains an increase in weight, other models' weights are decreased. As a result, some low priority tasks in working set could be temporarily excluded from processing to conserve resources.
The prioritization of tasks in working set on the level 2 of JDL/DFIG model affects the weights of elements included in conceptual model πΆπ πππ considered on the level 1. Thus, the model components, included in the working set models with higher weight also obtain the weight higher, than the objects only included in the models with lower weights. The elements not included in models belonging to the working set, or cues monitored can be ignored.
In turn, the weight of components in conceptual model πΆπ πππ influences the policies of data collection from the environment. There is no point to collect data about the ignored entities, especially in condition of severe lack of resources and high load.
The next, third level of JDL/DFIG uses sets of models associated with each detected situation ππ‘ π . There's a mapping between detected situation and corresponding decision-making and assessment models: πΉ πππ : ππ‘ π β (πΆπ πππ π , πΆπ πππ π ) Using those models, situation aware agent makes decisions and proposes actions to execute taking in consideration the data from the actual state as described by context model πΆπ πππ . The impact and possible consequences assessment model projects the results of proposed actions on agent's goals and current context model. If modeled impact is acceptable, then corresponding actions are initiated. After the completion of action πΆπ πππ checks the success or fail of actions and updates the information for situation model πΆπ π ππ‘ .
The πΆπ πππ π , πΆπ πππ π models are prioritized according to priorities of related situation models πΆπ π ππ‘ π . So, the most important situations get processed first. If situation is assessed as not important, then corresponding decision making is also deprioritized.
Decision making and impact assessment models execution results influence the weight of corresponding situation or task models. Thus, if decision impact was assessed as negligible, then corresponding situation model is deprioritized. Conversely, if the consequences even for minor situation in current context are assessed as grave, then this situation model gets a massive boost in weight.
The fourth level of JDL/DFIG model is often described as a level, where the overall performance of the situation awareness system is monitored and analyzed. Depending on the results of analysis the decisions are made, system parameters are changed, providing feedback for lower levels of situational awareness system. The situational awareness system on fourth level could be considered as situation aware system on its own right, implementing the functions of all other levels of JDL model, but aiming to reach not the awareness about environment, but self-awareness, that is the awareness of the state of awareness detection system itself.
This system can use the sophisticated models describing the dependencies between system parameters. However, the analysis of functions for such a system is outside the scope of this article. The operation of self-awareness subsystem will require additional computing resources.
The attention management mechanism is implemented on level 4. It takes as an input the information from the specific group of sensors, measuring the use of resources and interpreted as processors load, communication channels load, queues lengths. As output, it makes decision to lower or upper the attention thresholds πβ πππ , πβ ππ , πβ ππ’ .
This mechanism uses its own conceptual model πΆπ ππ‘π‘ , linking the values of load parameters, their dependencies to form the decision of how much the attention threshold should be changed.
This change is communicated to all levels of situational awareness system and influences the processing of conceptual models on those levels.
The summary of attention management feedback loops is shown on figure 1.
Figure1. Attention management feedback loops in situational awareness system
The level 3 subsystem does not have an attentional threshold, because its model set is dependent on models, processed on level 2 of JDL/DFIG model. However, the feedback from level 4 can take a form of indication, a hint of system being low on resources. For level 3 subsystem this will influence its preferences to select simpler models, probably with larger margin of error, when such model choice is available.
For the models on Level 2 attentional feedback just lowers or uppers the threshold of attention, influencing models and cues processed. As a result, some models may fall below the threshold of attention. The importance of components, included in the ignored model is recalculatedtheir weight is changed. These objects can be ignored in the Level 1 and data selection polices of the Level 0 also changed.
The implementation of attentional mechanisms in situational awareness system allows to quickly adapt to the everchanging contextual environment in which an intelligent agent operates. It helps to manage the limited computing resources, trying to dynamically specify the relative importance of ontology components and processed knowledge models. The use of normalized weights across this attentional mechanism presents a simple, and thus computationally not-taxing solution to the problem of dynamic allocation of resources.
As alternatives, the contextual ontology could be extracted from main agent's ontology with every change in context. However, ontology extraction is itself a complex procedure because this extracted ontology should be semantically complete logical theory [38] and continuously ascertaining this imposes additional computational load to the system.
The promising approach to provide flexibility to agent's ontology is the use of approach followed in Palantir platform, where only high-level ontology objects are mandatory as they are used to construct knowledge structures dynamically, depending on context [30].
The efficiency of assigning and processing relative weights to ontology components could be enhanced, if we would assign weights not to specific concepts and relations in the ontology, but entire patterns, clusters of related concepts used together [26].
Another unresolved problem is how the actual weights of ontology components or conceptual models could be defined taking in consideration the real-world tasks and considerations. This is complex problem, because there are so many unknown dependencies between objects which influence the relative importance of conceptual models and ontology components. To resolve this problem, we suggest the use of Machine learning approach, where on different levels of JDL model an artificial neuron network is used as an intermediary for the definition of weights of model and ontology components. Such a network will take as input the information coming from the result of conceptual modeling from different models and map it to the weights, reflecting the relative importance of those models. Such networks should be trained using real-world data.
Overall, the implementation of attentional mechanism in situational awareness systems is aimed to the efficient usage of limiting computing resources, allowing the intelligent agent to focus on important aspects of current context.