Software agents have gained increasing attention in the field of creating digital twins of physical, biological, and human entities. The processing of sensory inputs, individual perception, and the selection of suitable actions are essential processes in here, and agent-based frameworks can be utilized for supporting the design, the implementation, and the test. This paper reflects a work-in-progress project at an early stage. However, a conceptual model is presented for an analytical situation awareness component combining agent-based approach and data science algorithms.
Decision-making is a high-level cognitive process based on cognitive processes like perception, attention, and memory [Pr17]. Gaining a better mechanistic understanding of the decision-making process of humans is an important research topic in psychology and cognitive sciences, whereas the design and implementation of models that represent these processes are where computer scientists come into play.
Cognitive architectures refer to both a theory about the structure of the human mind and to a computational representation of such a theory used in the fields of artificial intelligence (AI) and computational cognitive science [Li]. Here, multi-agent systems provide a capable platform for running experiments in this field [ TW12]. [KT] mentioned the connection between cognitive architectures and deep learning and predicted that deep learning methods would likely play an essential role in designing of cognitive architectures in the future. The conceptual approach described in this study reflects the work-in-progress state of ideas about how to combine agent-based modeling with data science methods. 1.1
Cognitive architectures are a part of research in general AI with the ultimate goal of modeling the human mind, eventually enabling us to build human-level artificial intelligence [KT]. A comprehensive review of implemented cognitive architectures has been undertaken
Towards Enhancing of Situational Awareness for Cognitive Software Agents 179 by [Sa10]. More recently, [KT] provides a summary of the history of architectures, noting 55 existing architectures.
One of the most referenced models in this context is ACT-R ((Adaptive Control of ThoughtRational) [RTO19]. ACT-R was initially designed to model the visual perception and attention of humans [AMC97]. The ACT-R theory has a computational implementation as an interpreter of a specific coding language. The interpreter itself is written in Common Lisp and might be loaded into any of the Common Lisp language distributions. ACT-R reflects a theory of mechanisms that make up cognition. That theory posits a fixed set of mechanisms that use task knowledge to perform a task, thereby predicting and explaining the steps of cognition that form human behavior [RTO19].
As an example, [Sa06] has created a set of declarative and procedural rules about how to drive a car. He then added this knowledge to ACT-R that had its vision and motor systems connected to a car simulator (example taken from [RTO19]).
Unfortunately, the structure of ACT-R and its implementation was dificult to incorporate into an agent-based simulation system. Therefore, the C4 brain architecture model [Is01] was evaluated and adapted for this purpose [TCK10]. C4 was originally intended to simulate autonomous and semi-autonomous creatures in games or other virtual environments. We enriched this architecture by an emotional component, a filter component following the OCC model [OCC88], and applied it to a pedestrian crowd dynamics scenario similar to what was described by [TGB09]. The results (see [TCK10] for some details) encouraged us to choose this architecture for this study as well. 1.1.1
MARS2 is an agent-based modeling and simulation framework, which was developed at the Hamburg University of Applied Sciences, Germany [Hü]. It incorporates a domain-specific modeling language [Gl17], which helps scientists from a large variety of disciplines and without significant skills in programming developing their models. Modeling of cognitive processes and human decision-making is one of the focus research topics of the MARS Group. [TCK10] described the impact of emotions on pedestrians at a market place, whereas [LWC] utilized the goal-oriented action planning paradigm in MARS to evaluate adaptive behavior. 1.2
Data science is a rapidly emerging field incorporating a large amount of data-driven approaches. A combination of data science methods and agent-based modeling is a challenging research area. Many approaches integrate, especially Reinforcement Learning, for their purpose. In [Ch17], the proposed approach is used to find time-eficient collision-free paths 2 www.mars-group.org for multi-agent systems. The authors develop a value network that encodes the estimated time to the goal given an agent’s joint configuration with its neighbors. The use of the value network not only increases eficiency for finding a collision-free velocity vector but also considers the uncertainty in the other agents’ motion.
Another approach using reinforcement learning focuses on a multi-agent setting for autonomous driving [SSS16]. The authors analyze especially the long term driving strategies and introduce a so-called option gr aph for reducing the variance of the gradient estimation. Several other works applied recently similar methods to the multi-agent domain, e.g., the Go [Ma14] and Altari [Ta17] games. The authors in [Su16] concentrate on the collaboration of multiple agents. The remarkable part of this work is the application of a rather simple neural model for a complex task of learning to communicate between the agents. 2
As we know from our own experiences, humans perceive their world more holistically, switching their attention to details if necessary. Thus, many of our everyday decisions are made unconsciously [Ka12].
Software agents, on the other side, explore their spatial vicinity sequentially, iterating through specific objects, e.g., other agents, points-of-interests. In some cases, it would be preferable that agents can perceive an entire situation in one step. The reasons behind might be a better alignment to human perception or a simple increase in processing performance. Mainly, if software agents are representing digital twins of biological or technical entities, both aspects could be substantial. 2.1
The cognitive perception of a scene relies on the input data provided by available sensors. These data can be used to provide a structured representation of the scene for further processing by the next components, such as the working memory component of the C4 architecture. The structured representation of the scene is used to define the situation awareness components of the agents.
For this purpose, we use the profile and portfolio models of the agents similar to [ JT14] and [TT10]. Analyzing the situation awareness of a particular agent, we use the profile of this agent for processing the available information based on the abilities of the agent. A profile defines the parameters of an agent, starting with a stereotype based on the social system and environment of the agent. It contains information about the personality that the agent represents with general properties, such as psychological context, education, training, behavioral pattern, preferences, expertise, knowledge, or experience. Using the learning component, the profile of the agent develops gaining more knowledge and experience as well as adapting to the communities and the changing environment and new situation parameters. The portfolio describes the involvement, tasks, and restrictions of an agent as well as their
Towards Enhancing of Situational Awareness for Cognitive Software Agents 181 operating domains. A portfolio is specified by responsibilities and combined with the possible outcomes of the tasks. A task is an activity with an initial and a target state that includes interfaces or channels for communication and collaboration. Tasks also contain operating functions and are characterized by a set of parameters representing conditions and restrictions. The refinement of those parameters at run-time determines the execution accomplishment of the task. The result of task execution defines the transition to the target state and should satisfy the target conditions. 2.2
Each agent constructs a predictive behavior model for potential behavior in the next step. This model is based on calculations with possible combinations of actions of an agent. These combinations can be constructed as graphs with edges representing transitions from one state (starting with the current state) to a possible next state. Possible transitions are defined by the actions from the portfolio of the agent.
The situation awareness component of each agent also contains the perception of other agents. In a simulation environment, an agent has permanent access to the profile and portfolio information of other involved agents. In a specific scene, an agent can observe the environment and get information about the visible agents.
This information can be used to predict possible situation development for all possible actions. Each possible action for the next step can be prioritized based on the evaluation of the resulting situation with a cost function.
The following picture (Fig. 1) shows a schematic representation of the next step from the viewpoint of the left (blue) agent. The agent’s portfolio contains (in this example) three possible actions. There are also two more agents involved, whose actions influence the possible outcome and are considered in the evaluation of situation development for each action taken by the left agent. There are three possible situation developments, depicted as colored ovals, one for each possible action of the agent. For the other agents, only the most probable action predicted by the personal behavior model is considered in the evaluation. The action marked with a star can be considered as the one leading to the preferable situation development.
However, it is not suficient to predict the actions only for the next step. A more reliable model should be built by calculations for the next three or more possible steps, leading to complex situation development.
A directed graph with possible action paths represents the predictive behavior model. Each path is evaluated based on probabilities and includes a priority function for the outcomes. The result of the evaluation is a ranking order of the paths. The next step for the agent will be chosen from the path with the highest ranking. Due to the exponentially increasing number of outcomes with each additional step, there are limitations for the calculated path length. To reduce the complexity caused by the variety of possible combinations and to define a robust priority function, a classifier will be trained based on historical data. This classifier calculates the probabilities of all possible actions in the portfolio of the agent. The decision on what action to take is then made based on the highest probability provided by the classifier.
The data science methods are responsible for the accurate situation evaluation, and adaptation to dynamic situation changes. Data generated in previous simulations will be used to build individual knowledge bases for the agents. This knowledge then can be used to improve the learning of the agent behavior model.
The new goal as a result of this is the learning of interactions between the agents. The agents communicate and collaborate in their shared environment — this collaboration influence the behavior and negotiation abilities of the participating agents.
One of the essential tasks is to keep the balance between adaptation to the unexpected behavior of other agents, or dynamic situation changes, and ’normal’ behavior (e.g., to be not too defensive). 3
Instantaneous risk assessment of a situation in our daily lives, decision making, and learning are essential factors for human beings. Our conceptual approach describes the first brief idea of how to equip software agents with these capabilities by incorporating modern data science algorithms into their cognitive architecture.
At the time of writing, concrete simulation results are still missing. However, capturing an entire scene instead of exploring it object by object would increase the operational
Towards Enhancing of Situational Awareness for Cognitive Software Agents 183 performance of software agents. That is an essential issue if agents are considered as digital twins of humans or autonomous cars, for example.
During the next months, we will implement the conceptual approach described above within the MARS Framework utilizing the C4 cognitive architecture. Together with colleagues from robotics, we will develop a digital twin for an autonomous vehicle using this concept. Here, real-time sensory input from an IoT infrastructure replaces the software sensors of the MARS agents. [Ch17] [Gl17] [Hü] [Is01] [JT14] [Ka12] [KT] [Li] [LWC] [Ma14]
Chen, Y. F.; Liu, M.; Everett, M.; How, J. P.: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 285–292, 2017.
Kotseruba, Iuliia; Tsotsos, John K.: 40 Years of Cognitive Architectures: Core Cognitive Abilities and Practical Applications. Artificial Intelligence Review.
Lieto, Antonio; Bhatt, Mehul; Oltramari, Alessandro; Vernon, David: The role of cognitive architectures in general artificial intelligence. Cognitive Systems Research, pp. 1–3. Lenfers, Ulfia A.; Weyl, Julius; Clemen, Thomas: Firewood Collection in South Africa: Adaptive Behavior in Social-Ecological Models. Land, (3):97, aug.
Maddison, Chris J; Huang, Aja; Sutskever, Ilya; Silver, David: Move evaluation in Go using deep convolutional neural networks. arXiv preprint arXiv:1412.6564, 2014. [Pr17]
Prezenski, Sabine; Brechmann, André; Wolf, Susann; Russwinkel, Nele: A cognitive modeling approach to strategy formation in dynamic decision making. Frontiers in Psychology, 8(AUG), 2017. [Sa06] [SSS16] [Ta17]