=Paper=
{{Paper
|id=Vol-3171/paper78
|storemode=property
|title=The Introduction of Attentional Mechanism in the Situational Awareness Process
|pdfUrl=https://ceur-ws.org/Vol-3171/paper78.pdf
|volume=Vol-3171
|authors=Yevhen Burov
|dblpUrl=https://dblp.org/rec/conf/colins/Burov22
}}
==The Introduction of Attentional Mechanism in the Situational Awareness Process==
https://ceur-ws.org/Vol-3171/paper78.pdf
The Introduction of Attentional Mechanism in the Situational
Awareness Process
Yevhen Burov 1
1
Lviv polytechnic National University, 12 Bandera str, Lviv,79013, Ukraine
Abstract
The important trend in the development of information technologies today is the focus on the
intelligent systems capable of situational awareness. Such systems provide the continuous
monitoring of the environment, they build conceptual models and reason about tasks and
situations, make decisions, and assess their impact. Situational awareness process should be
done in real time. However, this is hard to achieve because of limited computational
resources available. The promising solution of this problem is the implementation of dynamic
prioritization of the elements of conceptualization as well as knowledge models processed.
The article describes such attentional mechanism based on JDL/DFIG model of situational
awareness process. The specifics of attention management on each model level as well as
feedback mechanisms influencing the importance of specific knowledge components are
presented. To prioritize knowledge components, the weighted graph ontology representation
is chosen. The implementation of attentional mechanisms in situational awareness system
allows to quickly adapt to the changing environment in which an intelligent agent operates
having limited computing resources.
Keywords 1
Situational awareness, attention, JDL/DFIG model, intelligent agent, ontology, context
1. Introduction
An important trend in the development of information technologies today is researching and
creating artificial situation aware systems using the recent results from the field of artificial
intelligence and knowledge-based systems.
Intelligent agents, capable of making autonomous decision-making, should be able to continually
collect information about the environment, assess and model it according to their internal knowledge
base and make quality decisions about actions to accomplish to further their goals. Internal knowledge
base is based on conceptualization of the world, which is constantly tested, updated, and replenished
using feedbacks from the assessment of results. This conceptualization of knowledge is formalized as
an ontology.
However, the implementation of such autonomous intelligent agents has several substantial
challenges to overcome. Firstly, to produce timely actions, situation assessment should be done in real
time. However, taking in consideration that modern ontologies are very complex, reasoning and
modeling using them quickly becomes computationally prohibitive.
Secondly, all knowledge is contextual, that is valid only in specific circumstances. The ability to
recognize contexts, find and process context-related knowledge adds a new dimension to the
complexity of the situational awareness implementation problem.
Finally, the dynamic nature of the environment requires constantly adapting the knowledge and
checking its validity and the consistency.
COLINS-2022: 6th International Conference on Computational Linguistics and Intelligent Systems, May 12–13, 2022, Gliwice, Poland
EMAIL: yevhen.v.burob@lpnu.ua
ORCID: 0000-0001-8653-1520 (Y. Burov);
©️ 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Taking in consideration that a major part of agent’s knowledge base is irrelevant to the current
situation, the promising solution to these challenges could be the implementation of filtering, focusing
mechanism into the process of situation awareness. It will select only a small, relevant to current
context and situations part of ontology and build using it the conceptual models, which could be used
in the process of making decisions.
This article explores the process of such attentional mechanism implementation using the approach
of adaptive ontologies and weighted graph ontology representation. We also highlight the importance
of prioritizing conceptual models used in higher levels of JDL/DFIG model and the resource
monitoring process which controls the attention levels on multiple levels of situational awareness
process.
The article has an introduction and six sections. In the background research section, we introduce
the problem of situational awareness implementation in intelligent agents and models used to
structure and understand the situational awareness process. Much of the knowledge used in situational
awareness exist in the form of conceptual models, which are formulated using ontologies. We stress
the problem of high ontology complexity, which leads to the high computing resources usage in one
hand and the requirement for situational awareness system to make decisions and react to changes in
real time in the other. Hence, we infer the need to introduce attentional mechanism in situational
awareness system prioritizing the use of resources. Next, we look for insight into neurology,
psychology and machine learning which have studied attentional mechanisms for a long time.
In the second section we describe how knowledge is prioritized on the first two levels of
JDL/DFIG model and how contextual ontology is built. In the next section we describe attentional
mechanisms on the higher levels of JDL/DFIG model. The performance monitoring section describes
feedbacks provided to all lower levels of situational awareness system controlling the attention levels
throughout the system. Finally, in the conclusion and discussion session we discuss possible ways of
implementing the attentional mechanisms, the advantages of using prioritizing in conceptual modeling
and reasoning.
2. Background research
2.1. Modeling situational awareness systems
According to the definition [1] situational awareness is the “conscious knowledge of the
immediate environment and the events that are occurring in it. Situation awareness involves
perception of the elements in the environment, comprehension of what they mean and how they relate
to one another, and projection of their future states”. Situational awareness research is filed under the
more general topic of data fusion [2].
The process of situation awareness encompasses various kinds of operations, associated with being
intelligent, such as selective perception of the environment, pattern and object recognition, situation
identification based on previous experiences, reasoning, decision making and acting upon those
decisions, assessing the success of actions, adapting knowledge and processes. The ultimate goal of
situation aware system is to make decisions and adapt intelligent agent’s behavior to the changing
environment according to the goals of this agent.
Several models were developed to represent situational awareness process. They can be classified
as process, functional and formal models. The early process models, such as John Boyd’s Observe-
Orient-Decide-Act (OODA) loop [3,4] or Predict-Match- Extract-Search loop [5] were developed as a
generalization of real-world situation awareness processes in complex environments, such as
battlefield. Functional models are represented by Endsley model [6], JDL (Joint Directors of
Laboratories/ DFIG (Data fusion information group) [7-9] models. There are also the inquiries
exploring different perspectives in situational awareness process using different formal frameworks
such as Category theory, generalized information theory, interpreted systems, ontologies and
specification languages. However, the most widely accepted framework of conceptualizing situational
awareness process is functional JDL/DFIG model [10]. This model, as many other situational
awareness process models follows the structure of human cognition process [11].
JDL/DFIG model splits the situation awareness process in five levels [10]:
• level 0. Signal/Feature assessment. On this level the signals from various sensors are gathered
and interpreted as input data, corresponding to attributes of measured entities; signals are
processed, the errors in measured data are assessed
• level 1. Entity assessment. The data obtained are interpreted as attributes of entities from the
ontology
• level 2. Situation assessment. The entities involved in the current context and their
relationships are analyzed to build the model of context and detect situations in this context. This
level works with conceptual models of environment, context, and situations
• level 3. Impact assessment. Planning actions according to detected situations. Making
decisions. Analyzing the consequences of the decisions made.
• level 4. Performance assessment. Evaluating the correspondence between current situation
and system goals, performance analysis, forming feedbacks to lower levels of information fusion,
updating models.
JDL/DFIG model is constantly evolving and revised. Recently it is considered as a part of data
fusion process - the process of integrating multiple data sources to produce more consistent, accurate
and useful information [12].
2.2. Using ontologies for conceptual modeling in situational awareness
systems
The development of situational awareness systems requires the ability to build conceptual models
of tasks, contexts, and situations. The basic vocabulary for such models is provided by ontologies –
formal specifications of conceptualizations. Such conceptualizations aim to capture the knowledge
about relationships between concepts in the domain.
Ontologies are routinely used to represent knowledge in JDL model [13,14]. The research about
the application of ontologies in situational awareness often focuses on representing different
approaches to reasoning, supporting situation identification and decision making in the context of
JDL/DFIG model. For example, [14] proposes the enhancement of JDL model with processing
complex events structures, building actionable abstractions from event streams and dependencies.
The attempts to formalize domain conceptualization in form of an ontology usually produce large
ontologies. For example, Cyc ontology has hundreds of thousands of concepts and more than million
rules [16], the number of concepts and WordNet has more than 117 thousand of synsets (synonym
groups) [17].
Ontology complexity had different aspects which are addressed in current research separately. The
important part of ontology complexity is structural complexity. In [18] an ontology is considered as a
kind of a complex, compositional system. The author specifies the two main causes of such systems
complexity. The first one is the nonlinear increase of the dependencies between elements, when the
number of elements grows. The second one roots in the sensitivity of the whole system even to the
minor change. Every such change in one element propagates to others, dependent objects and thus
requires the review of the whole ontology.
The early idea of counteracting the structural complexity of ontology was splitting the general
ontology into parts according to specificity of used concepts level. The reusable across domains part
became top (upper ontology) and specialized part - domain ontology. This created additional problem
of integrating top and domain ontologies [19]. Moreover, domain ontologies remained too structurally
complex and redundant when resolving specific practical problems or used in specific business
environments.
The effort of revealing and formalizing the most basic concepts and relations which can be
reused consistently across multiple ontologies resulted in creation of foundational ontologies SUMO,
UFO, and GFO [20, 21] and libraries of objects and patterns such as [22].
Another solution to the complexity problem is modularization and partitioning of ontologies. In
[23] the initial ontology is split into smaller parts using taxonomy structure. The authors of [24]
propose extracting a module by specifying the part of larger ontology based on specific set of
interrelated concepts, which form the signature of the module. The module defines the subset of
knowledge, which can be used separately from original ontology for specified tasks.
The modern research in the domain of ontologies tend to represent ontologies as set of smaller
patterns and patterns groups [25], paving a way to the development of pattern languages [26].
The complexity of ontologies makes the extensive reasoning using them computationally
prohibitive [27]. That’s why currently we are limited to simpler forms of logical reasoning with
ontologies, based on less expressive dialects of description logic [28].
On the other hand, the assessing of the current context and making decisions are often required to
be done in real time. Therefore, it is important to support dynamic changes in the ontology structure
as well as to simplify this process and decrease the resources usage by adding constraints and
reducing the number of concepts and relations from real world which should be taken into
consideration.
There are several approaches, which could be useful in handling the changes in the ontology. First
one is dynamic ontologies [29]. The promising approach in handling the changes in ontology is
presented in Palantir platform [30] which allows to use weighted ontology components to reason on
different specificity levels, reflect the importance of ontology components and integrate ontologies
presented in different formats.
However, the dynamic ontologies consider the problem of ontology revisions in the process of its
evolution and knowledge acquisition, which is quite different problem compared to situational
awareness, where the importance of specific components of ontology is highly contextual.
Another one is adaptive ontologies [31], where concepts and relations used in the ontology reflect
their importance in the real-life usage. Such approach can be used, for example, for deriving ontology
from texts. In [32] the method for selection of important concepts and relations is proposed based on
the automated weighting of concepts and relations during ontology learning. However, in the
situational awareness systems the importance of ontology elements dynamically changes depending
on the changes in the environment, agent’s goals, and reasoning results. It is constantly updated via
feedbacks from higher levels of JDL/DFIG model. It also depends on the current system load level
and the importance of tasks processed by system currently. Taking in consideration the limited
computing resources available to intelligent agent, the development of attentional mechanism,
dynamically highlighting and prioritizing the ontology elements and larger knowledge aggregates,
such as models and patterns is needed for efficient functioning of intelligent agent.
2.3. Attention research in neurology, psychology, and machine learning
The valuable insights for creation of attentional mechanism in situational aware systems can be
obtained from psychology and neurology, where attention was studied for a long time [33]. The
purpose of attentional mechanism in both natural and artificial system is to prioritize information and
tasks in face of data processing constraints.
Literature defines attention as “the concentration of awareness on some phenomenon to the
exclusion of other stimuli” [34]. Another, verbose explanation states: “The term attention captures the
cognitive functions which are responsible for filtering out unwanted information and bringing to
consciousness what is relevant for the organism. ... Closely related to this aspect of selectivity is the
assumption that the available quantity of attention is finite”.
The most researched was the visual attention process, which is similar in structure to JDL/DFIG
model, having the sensory part, object attention, reasoning, and executive control [35].
The author in [34] states, that despite the many attempts to precisely define and quantify the
attentional process while also identifying the underlying mental and neural architectures that give rise
to it, no one still knows what attention is. However, the modeling of human attention has recurrent
findings, which can be reused in the creation of attentional mechanism in artificial situational aware
systems. The useful insight, for example is provided by saliency maps used in visual attention
modeling, allowing to detect the part of the picture which stand out from the background. Another one
is that attention manifests in increasing the firing rates of neurons for attended features and
suppressing rates for the rest.
Recently, the attentional mechanisms were proposed in Machine learning [35]. They mostly are
implemented as the dynamic calculation of the weights of the nodes in artificial neural networks
depending on the input data and previously calculated weights. Such mechanisms have given the
system flexibility and efficiency in the condition of limited resources [35].
This article proposes the outline of attentional mechanism for situational aware system based on
JDL/DFIG model and demonstrates how resulting attention is formed from feedbacks, coming from
multiple levels of this model.
The main assumptions we make in our research are as follows.
1. We use JDL/DFIG model as a representation of situational awareness process.
2. This process is implemented by intellectual agent, which actively perceives the environment,
has its own set of conceptual knowledge (represented as ontology), knowledge base and
information base storing facts and knowledge in the form of rules, algorithms, and models.
3. An agent reasons about the world using conceptual models, which are built from the agent’s
own ontology.
4. Each agent is constantly learning using feedbacks coming from the assessment of results of its
actions. This feedback is coming from higher levels activities in JDL/DFIG model. Thus, the
knowledge base and ontology of an agent is constantly updated.
3. Contextual ontology modeling as a part of attentional mechanism on the
first two levels of JDL/DFIG model
In the first stage of JDL model situational aware system obtains data from sensors and interprets
them as attributes/parameters values of concepts from agent’s ontology 𝑂𝑛. The objects and their
relationships as observed by agent, form the conceptual model of the environment 𝐶𝑚𝑒𝑛𝑣 , described
using the elements from 𝑂𝑛. The elements from 𝐶𝑚𝑒𝑛𝑣 form a part of the agent’s domain ontology
which can be represented as a smaller ontology 𝑂𝑛𝑒𝑛𝑣 ⊆ 𝑂𝑛 which can be extracted from the 𝑂𝑛.
However, the sphere of agent’s attention is not limited to observed world. It should also consider
important objects and relationships deduced by reasoning processes, coming from agent’s intents,
goals and values, previous experiences. Therefore, contextual ontology 𝑂𝑛𝑐𝑜𝑛 , which includes all
those relevant to current context components is larger than the ontology of perceived environment:
𝑂𝑛𝑒𝑛𝑣 ⊆ 𝑂𝑛𝑐𝑜𝑛 ⊆ 𝑂𝑛 .
We will model contextual ontology as a part of agent’s ontology selected by attentional
mechanisms. With this purpose we will use the graph-based representation of ontology structure as a
graph of concept-vertices 𝑉𝑐𝑜𝑛 and relations – edges 𝐸𝑟𝑒𝑙 .
𝐺𝑜𝑛 = (𝑉𝑐𝑜𝑛 , 𝐸𝑟𝑒𝑙 )
We omit from this structural model the last component of ontology definition- axioms because we
are not using them to model attentional processes.
To differentiate the importance of specific ontology components in the current context a weight
𝑖 𝑗
functions assigns weight to every concept 𝑣𝑐𝑜𝑛 and relation 𝑒𝑐𝑜𝑛 in ontology.
𝑖
𝑊𝑣: 𝑣𝑐𝑜𝑛 →ℝ
𝑗
𝑊𝑒: 𝑒𝑟𝑒𝑙 → ℝ
𝑖 𝑖 𝑗 𝑗
For brevity we will use notation for weights 𝑊𝑣(𝑣𝑐𝑜𝑛 ) = 𝑤𝑐𝑜𝑛 ; 𝑊𝑒(𝑒𝑟𝑒𝑙 ) = 𝑤𝑟𝑒𝑙
The values of weights are constantly updated based on feedback signals coming from higher levels
of JDL/DFIG model, changes in environment – new objects perceived and recognized by system.
The attentional mechanism monitors the resources available and decides to increase or decrease
the value of the threshold of attention 𝑇ℎ𝑐𝑜𝑛 . All ontology elements having weights below the
threshold are ignored and system focuses on the elements being above the threshold.
This is used to establish priority in reading information from corresponding sensors and getting
additional information from them. This is also used in prioritizing access to reasoning, modeling
operations, getting additional information from external knowledge bases.
4. Attentional mechanisms on the situation assessment level
4.1. Conceptual models processed on the situation assessment level
On the second level of JDL/DFIG model a conceptual knowledge is processed, tasks
executed, situations detected. To do this, system uses the knowledge about similar situations stored in
the knowledge base, performs reasoning to get information for decision making, happening on the
next level. Such activity uses a substantial amount of computing resources. Hence, the prioritization
of tasks and attentional mechanism is required for efficient situational awareness system functioning
on this level.
We will assume that only one conceptual model is processed in any given time. The parallel
processing of conceptual models requires additional resources for coordination of model execution
and adds another level of complexity. Thus, system is constantly switching between models belonging
𝑖
to the active set of models 𝑆𝐶𝑚. Each model 𝐶𝑚𝑖 is assigned a weight 𝑤𝑚𝑑 . Only models that have a
weight above the model attention threshold 𝑇ℎ𝑚𝑑 are included in the working set:
𝑖
∀𝐶𝑚𝑖 ∈ 𝑆𝐶𝑚: 𝑤𝑐𝑚 > 𝑇ℎ𝑚𝑑
The models not included in working set are not processed. The models in working set obtain
processor resources proportionally to their weight.
What kind of conceptual models are included in the working set? We will differentiate between:
• Task models. An agent is performing various tasks. Tasks are chosen taking in consideration
agent’s intentions and goals (which can be represented as another model). Different agents, even
being in the same environment, can perform different tasks. For example, a car driver attends to
driving a car, while his passenger can attend to admiring the scenic route.
• Situation processing models. If some specific situation was detected, agent activates model
processing it.
• Model monitoring conceptual model of context 𝐶𝑚𝑐𝑜𝑛 for cues and events which points to
the situations, requiring actions.
For the sake of simplicity, we omit other important processes, such as planning, goal setting etc.
We assume that they are performed outside of the scope of situational awareness process.
All models take information from the information base and contextual model, populated with data
coming from sensors and interpreted as objects, belonging to ontology types. Models also use
knowledge coming from experience and accumulated in the knowledge base. They can also form
queries to external knowledge bases, if needed information is missing in agent’s knowledge base.
4.2. The use of prototypical contexts to reuse conceptual knowledge
An agent rarely meets the current context for the first time. Typically, he has been in a similar
condition before. The knowledge about similar contexts is stored in the knowledge base as
prototypical context models. This knowledge comes from the previous experiences of the agent or is
formed as a result of learning. According to the prototype theory [36,37] humans form the mental
representation of concept as a fuzzy set of typical objects, belonging to the concept. This set has some
central, core object and several exceptions, describing the deviations from the central object. Different
concepts, formed in this way can be overlapping. Prototypical contexts are formed in the same way,
as groups of similar contexts, where similar intents are followed in the similar environments.
Prototypical contexts are stored in the knowledge base of the agent as a set of conceptual models
𝑖
𝑆𝐶𝑚𝑡𝑦𝑝 = {𝐶𝑚𝑡𝑦𝑝 } . With each prototypical context is associated the knowledge about tasks,
practices, techniques which can be used in the current context and situations which can happen.
On level 2 of JDL/DFIG model an agent connects to the relevant knowledge from its knowledge
base by finding similarity between current conceptual model of context and prototypical context
models. The identification of relevant prototypical context is a pattern recognition problem, which
uses a similarity function 𝐹𝑠𝑖𝑚 (𝐶𝑚𝑖 , 𝐶𝑚𝑗 ) which could be interpreted as a distance between two
conceptual models. Therefore, this problem can be formulated as finding the prototypical model
𝑘
𝐶𝑚𝑡𝑦𝑝 which minimizes the value of similarity function:
𝑘
𝐹𝑠𝑖𝑚 (𝐶𝑚𝑐𝑜𝑛 , 𝐶𝑚𝑡𝑦𝑝 ) → 𝑚𝑖𝑛
The identification of prototypical modes reduces the computational load, because it limits the
number of situations and cues to monitor or tasks to execute to the subset of models associated with
identified prototypical model.
Once the relevant prototypical context model is identified, the conceptual context model 𝐶𝑚𝑐𝑜𝑛 is
𝑘
updated with knowledge from this prototypical model 𝐶𝑚𝑡𝑦𝑝 :
• new concepts and relations are added. Those concepts may be not observed directly, but they
are important for reasoning and decision making.
• if needed, the values of newly added objects are populated by querying the agent’s knowledge
base or external information sources.
• the weights of components in 𝐶𝑚𝑐𝑜𝑛 is updated, reflecting the knowledge about importance
of objects
• a mapping between components of 𝐶𝑚𝑐𝑜𝑛 and 𝐶𝑚𝑡𝑦𝑝 𝑘
is established.
4.3. Environment monitoring and situation detection
When the prototypical context is identified, the agent obtains access to the specification of
situations which can occur in this context. Knowing the prototypical context reduces the number of
situations to consider and analyze. Let’s denote such set of situation specifications as 𝑆𝑆𝑡.
The knowledge about situation 𝑆𝑡 𝑖 , stored in knowledge base has such parts:
• conceptual model of situation 𝐶𝑚𝑠𝑖𝑡 specifying the relevant objects and their relationships,
involved in the detection, analysis, making decision and projecting its impact. All elements of
𝐶𝑚𝑠𝑖𝑡 belong to the agent’s ontology 𝑂𝑛.
• Diagnostic information, containing conditions and procedures needed to detect the situation.
An important part of this information is a set of cues 𝑆𝐶𝑢𝑖 – conditions to monitor in the
environment which, if present, can point to the possibility of having a situation. This part of
specification also contains possible variations (scenarios) in situation development, requiring
additional analysis and different courses of actions. This section also contains the assessment of
possible risks for different variants of situation.
• Decision making and impact assessment procedures which process the results of situation
analysis. This section contains information used on the next, third level of JDL/DFIG model.
• Actionable information about what should be done, once the decision is made, task models,
which describe the task execution, success/fail conditions, expected results, feedback actions
depending on the result of task execution.
The review of situation specifications can add additional elements to conceptual model of context,
especially if there’s high risk, or high impact on agent’s goals associated with this situation. Such
elements are included in cues. For example, the car driving system can choose to monitor information
about data traffic jams or closed roads ahead.
Because of complexity in situation definitions and processing procedures it is viable to monitor the
context and corresponding conceptual model only for cues pointing for situations and assign resources
to processing the situation only when this situation was detected.
The process monitoring the contextual model 𝐶𝑚𝑐𝑜𝑛 for cues takes the list of cues 𝐿𝐶𝑢 = (𝐶𝑢𝑖 )
𝑖
ordered according to their weights 𝑤𝑐𝑢 . The weights are assigned to cues depending on the likelihood
of corresponding situation occurrence and risk associated with it. The attentional mechanism only
considers cues with a weight above corresponding threshold 𝑇ℎ𝑐𝑢 .
The agent’s environment and corresponding conceptual model are constantly changing, because of
external events occurring in it. Therefore, the list of situations is updated too, taking in consideration
the situations involving new objects and removing situations with objects which disappeared from the
conceptual model.
Attentional mechanism on level 2 dynamically reassigns weights to models, belonging to the
working set. The values of weights for all models in working set are normalized, so if one model
obtains an increase in weight, other models’ weights are decreased. As a result, some low priority
tasks in working set could be temporarily excluded from processing to conserve resources.
The prioritization of tasks in working set on the level 2 of JDL/DFIG model affects the weights of
elements included in conceptual model 𝐶𝑚𝑐𝑜𝑛 considered on the level 1. Thus, the model
components, included in the working set models with higher weight also obtain the weight higher,
than the objects only included in the models with lower weights. The elements not included in models
belonging to the working set, or cues monitored can be ignored.
In turn, the weight of components in conceptual model 𝐶𝑚𝑐𝑜𝑛 influences the policies of data
collection from the environment. There is no point to collect data about the ignored entities, especially
in condition of severe lack of resources and high load.
5. Prioritizing tasks in the process of decision making and modeling their
impact
The next, third level of JDL/DFIG uses sets of models associated with each detected situation 𝑆𝑡 𝑖 .
There’s a mapping between detected situation and corresponding decision-making and assessment
𝑖 𝑖
models: 𝐹𝑑𝑒𝑐 : 𝑆𝑡 𝑖 → (𝐶𝑚𝑑𝑒𝑐 , 𝐶𝑚𝑖𝑚𝑝 )
Using those models, situation aware agent makes decisions and proposes actions to execute taking
in consideration the data from the actual state as described by context model 𝐶𝑚𝑐𝑜𝑛 . The impact and
possible consequences assessment model projects the results of proposed actions on agent’s goals and
current context model. If modeled impact is acceptable, then corresponding actions are initiated. After
the completion of action 𝐶𝑚𝑖𝑚𝑝 checks the success or fail of actions and updates the information for
situation model 𝐶𝑚𝑠𝑖𝑡 .
𝑖 𝑖
The 𝐶𝑚𝑑𝑒𝑐 , 𝐶𝑚𝑖𝑚𝑝 models are prioritized according to priorities of related situation models
𝑖
𝐶𝑚𝑠𝑖𝑡 . So, the most important situations get processed first. If situation is assessed as not important,
then corresponding decision making is also deprioritized.
Decision making and impact assessment models execution results influence the weight of
corresponding situation or task models. Thus, if decision impact was assessed as negligible, then
corresponding situation model is deprioritized. Conversely, if the consequences even for minor
situation in current context are assessed as grave, then this situation model gets a massive boost in
weight.
6. Monitoring overall performance and controlling attention parameters
The fourth level of JDL/DFIG model is often described as a level, where the overall
performance of the situation awareness system is monitored and analyzed. Depending on the results
of analysis the decisions are made, system parameters are changed, providing feedback for lower
levels of situational awareness system.
The situational awareness system on fourth level could be considered as situation aware system on
its own right, implementing the functions of all other levels of JDL model, but aiming to reach not the
awareness about environment, but self-awareness, that is the awareness of the state of awareness
detection system itself.
This system can use the sophisticated models describing the dependencies between system
parameters. However, the analysis of functions for such a system is outside the scope of this article.
The operation of self-awareness subsystem will require additional computing resources.
The attention management mechanism is implemented on level 4. It takes as an input the
information from the specific group of sensors, measuring the use of resources and interpreted as
processors load, communication channels load, queues lengths. As output, it makes decision to lower
or upper the attention thresholds 𝑇ℎ𝑐𝑜𝑛 , 𝑇ℎ𝑚𝑑 , 𝑇ℎ𝑐𝑢 .
This mechanism uses its own conceptual model 𝐶𝑚𝑎𝑡𝑡 , linking the values of load parameters, their
dependencies to form the decision of how much the attention threshold should be changed.
This change is communicated to all levels of situational awareness system and influences the
processing of conceptual models on those levels.
The summary of attention management feedback loops is shown on figure 1.
Figure1. Attention management feedback loops in situational awareness system
The level 3 subsystem does not have an attentional threshold, because its model set is dependent
on models, processed on level 2 of JDL/DFIG model. However, the feedback from level 4 can take a
form of indication, a hint of system being low on resources. For level 3 subsystem this will influence
its preferences to select simpler models, probably with larger margin of error, when such model
choice is available.
For the models on Level 2 attentional feedback just lowers or uppers the threshold of attention,
influencing models and cues processed. As a result, some models may fall below the threshold of
attention. The importance of components, included in the ignored model is recalculated – their weight
is changed. These objects can be ignored in the Level 1 and data selection polices of the Level 0 also
changed.
7. Conclusion and discussion
The implementation of attentional mechanisms in situational awareness system allows to quickly
adapt to the everchanging contextual environment in which an intelligent agent operates. It helps to
manage the limited computing resources, trying to dynamically specify the relative importance of
ontology components and processed knowledge models. The use of normalized weights across this
attentional mechanism presents a simple, and thus computationally not-taxing solution to the problem
of dynamic allocation of resources.
As alternatives, the contextual ontology could be extracted from main agent’s ontology with every
change in context. However, ontology extraction is itself a complex procedure because this extracted
ontology should be semantically complete logical theory [38] and continuously ascertaining this
imposes additional computational load to the system.
The promising approach to provide flexibility to agent’s ontology is the use of approach followed
in Palantir platform, where only high-level ontology objects are mandatory as they are used to
construct knowledge structures dynamically, depending on context [30].
The efficiency of assigning and processing relative weights to ontology components could be
enhanced, if we would assign weights not to specific concepts and relations in the ontology, but entire
patterns, clusters of related concepts used together [26].
Another unresolved problem is how the actual weights of ontology components or conceptual
models could be defined taking in consideration the real-world tasks and considerations. This is
complex problem, because there are so many unknown dependencies between objects which influence
the relative importance of conceptual models and ontology components. To resolve this problem, we
suggest the use of Machine learning approach, where on different levels of JDL model an artificial
neuron network is used as an intermediary for the definition of weights of model and ontology
components. Such a network will take as input the information coming from the result of conceptual
modeling from different models and map it to the weights, reflecting the relative importance of those
models. Such networks should be trained using real-world data.
Overall, the implementation of attentional mechanism in situational awareness systems is aimed
to the efficient usage of limiting computing resources, allowing the intelligent agent to focus on
important aspects of current context.
8. References
[1] APA dictionary of psychology, Situational awareness, 2022. URL:
https://dictionary.apa.org/situation-awareness.
[2] J. Bleiholder, F. Naumann. Data fusion. ACM computing surveys (CSUR) 41, no. 1 (2009): 1-41
[3] C. Richards, C. Boyd’s OODA loop. Richards, 2020.
[4] E. Shahbazian, D. Blodgett., P. Labbé, The extended OODA model for data fusion systems.
Fusion01. (2001)
[5] A. Jousselme, P. Maupin. Interpreted systems for situation analysis. In: 2007 10th International
Conference on Information Fusion, pp. 1-11. IEEE, 2007.
[6] M C Wright, , J M Taekman, M R Endsley. “Objective Measures of Situation Awareness in a
Simulated Medical Environment.” Quality & Safety in Health Care 13 Suppl 1 (2004) 65–71.
[7] A.N Steinberg. C.L. Bowman, Revisions to the JDL data fusion model. In: Handbook of
multisensor data fusion, CRC Press, 2008, pp. 65-88.
[8] J. Llinas,., C. Bowman, G. Rogova, A. Steinberg, E. Waltz, F. White,. Revisiting the JDL data
fusion model II. Space and naval warfare systems command san diego CA., 2004.
[9] E. Blasch, A. Steinberg, S. Das, J. Llinas, C. Chong, O.Kessler, E. Waltz, F.White,. Revisiting
the JDL model for information exploitation. In: Proceedings of the 16th International Conference
on Information Fusion, 2013, pp. 129-136.
[10] E. Blasch, “One decade of the Data Fusion Information Group (DFIG) model.” Commercial +
Scientific Sensing and Imaging (2015)
[11] E. Bossé, A. Jousselme, P. Maupin, P. Knowledge, uncertainty and belief in information fusion
and situation analysis. In: NATO SCIENCE SERIES SUB SERIES III COMPUTER AND
SYSTEMS SCIENCES, (2005) pp. 198_, 61
[12] Foo, Pek Hui, and Gee Wah Ng. "High-level information fusion: An overview." J. Adv. Inf.
Fusion 8, no. 1 (2013): 33-72.
[13] M.M. Kokar,;C.J.Matheus; K.Baclawski. “Ontology-Based Situation Awareness.” International
Journal of Information Fusion, no. 10(1) (2009): 83–98.
[14] G. Meditskos,Y. Kompatsiaris, Y. “iKnow: Ontology-driven situational awareness for the
recognition of activities of daily living”. Pervasive Mob. Comput., 40, (2017), pp. 17-41.
[15] H.A Noughabi, M. Kahani, A.Shakibamanesh, “Enhancing Situation Awareness Using Semantic
Web Technologies and Complex Event Processing” Annals of Data Science, 5, (2018), pp. 487-
496.
[16] Cyc’s knowledge base, 2022. URL: https://cyc.com/
[17] Wordnet . A lexical database for English, 2022. URL: https://wordnet.princeton.edu
[18] M. Strevens. Ontology, Complexity, and Compositionality. (2017), doi:10.1093/acprof:oso/
9780199363209.003.0003
[19] M. Klein,. Combining and relating ontologies: an analysis of problems and solutions. In: OIS@
IJCAI.(2001)
[20] G. Guizzardi, G. Wagner,. Using the unified foundational ontology (UFO) as a foundation for
general conceptual modeling languages. In: Theory and applications of ontology: computer
applications Springer, Dordrecht, 2010, pp. 175-196.
[21] H. Herre.. General Formal Ontology (GFO): A foundational ontology for conceptual modelling.
In: Theory and applications of ontology: computer applications. Springer, Dordrecht (2010),
pp. 297-345.
[22] C. Masolo, S. Borgo, A. Gangemi,N. Guarino, A. Oltramari, L. Schneider, The wonderweb
library of foundational ontologies. (2002).
[23] H. Stuckenschmidt, M. Klein, Structure-based partitioning of large concept hierarchies. In:
International semantic web conference. Springer, Berlin, Heidelberg (2004), pp. 289-303
[24] C.Vescovo, B. Parsia, U. Sattler, T. Schneider, The Modular Structure of an Ontology: Atomic
Decomposition. IJCAI. (2011).
[25] G. Guizzardi, Ontological patterns, anti-patterns and pattern languages for next-generation
conceptual modeling. In International Conference on Conceptual Modeling. Springer, Cham,
(2014), pp. 13-27.
[26] F. Ruy, R. Falbo, M. Barcellos, G. Guizzardi, G. Quirino, “An ISO-based software process
ontology pattern language and its application for harmonizing standards”. ACM SIGAPP
Applied Computing Review, 15(2), (2015). pp. 27-40.
[27] C. Bettini, O. Brdiczka, K. Henricksen, J. Indulska, D. Nicklas, A.Ranganathan, and D. Riboni.
"A survey of context modelling and reasoning techniques." Pervasive and mobile computing 6,
no. 2 (2010): 161-180.
[28] A. Turhan, A.Yasmin. "Description logic reasoning for semantic web ontologies." In:
Proceedings of the International Conference on Web Intelligence, Mining and Semantics, pp. 1-
5. 2011.
[29] V. Petrov, A.Scarfe, eds. Dynamic being: Essays in process-relational ontology. Vol. 2.
Cambridge Scholars Publishing, 2015.
[30] J. Iurygina, A. Anisimov, O. Lakhin, N.Stratilatova, A.Egorov, V. Skirmunt. "Ontology and
multi-agent technology application for intellectual property knowledge management." Indian
Journal of Science and Technology 9, no. 46 (2016).
[31] V. Lytvyn, V. Vysotska, D. Dosyn, O.Lozynska O.Oborska. Methods of Building Intelligent
Decision Support Systems Based on Adaptive Ontology. In: proceedings of 2018 IEEE Second
International Conference on Data Stream Mining & Processing (DSMP) (2018): 145-150.
[32] V. Lytvyn, D. Dosyn, V. Vysotska, Y. Burov. “Method for Ontology Content and Structure
Optimization, Provided by a Weighted Conceptual Graph”. Webology,Volume 15, Number 2,
December, 2018.
[33] P. Thier, T. Haarmeier, A.Ignashchenkova. "The functional architecture of attention." Current
Biology 12, no. 5 (2002): R158-R162.
[34] B. Hommel, B., Chapman, C. S., Cisek, P., Neyedli, H. F., Song, J.-H., and Welsh, T. N.”No one
knows what attention is”. Attent. Percept. Psychophys. 81, 2288–2303. doi: 10.3758/s13414-
019-01846-w
[35] G. Lindsay. "Attention in psychology, neuroscience, and machine learning." Frontiers in
computational neuroscience 14 (2020): 29.
[36] P. Gärdenfors. Conceptual spaces: The geometry of thought. MIT press, 2004. E0921211548
[37] J. Hampton, "Concepts as prototypes." Psychology of learning and motivation 46 (2006): 79-113.
[38] M. Bhatt, C. Wouters, A. Flahive, W. Rahayu, D. Taniar. "Semantic completeness in sub-
ontology extraction using distributed methods." In: Proceedings of the International Conference
on Computational Science and Its Applications, pp. 508-517. Springer, Berlin, Heidelberg, 2004.