Joint Proceedings of Modellierung 2020 Short, Workshop and Tools & Demo Papers 178 Workshop on Models in AI Towards Enhancing of Situational Awareness for Cognitive Software Agents Marina Tropmann-Frick ,1 Thomas Clemen1 Abstract: Software agents have gained increasing attention in the field of creating digital twins of physical, biological, and human entities. The processing of sensory inputs, individual perception, and the selection of suitable actions are essential processes in here, and agent-based frameworks can be utilized for supporting the design, the implementation, and the test. This paper reflects a work-in-progress project at an early stage. However, a conceptual model is presented for an analytical situation awareness component combining agent-based approach and data science algorithms. Keywords: Agent-based modeling; data science; cognitive processes 1 Introduction Decision-making is a high-level cognitive process based on cognitive processes like perception, attention, and memory [Pr17]. Gaining a better mechanistic understanding of the decision-making process of humans is an important research topic in psychology and cognitive sciences, whereas the design and implementation of models that represent these processes are where computer scientists come into play. Cognitive architectures refer to both a theory about the structure of the human mind and to a computational representation of such a theory used in the fields of artificial intelligence (AI) and computational cognitive science [Li]. Here, multi-agent systems provide a capable platform for running experiments in this field [TW12]. [KT] mentioned the connection between cognitive architectures and deep learning and predicted that deep learning methods would likely play an essential role in designing of cognitive architectures in the future. The conceptual approach described in this study reflects the work-in-progress state of ideas about how to combine agent-based modeling with data science methods. 1.1 Cognition modeling Cognitive architectures are a part of research in general AI with the ultimate goal of modeling the human mind, eventually enabling us to build human-level artificial intelligence [KT]. A comprehensive review of implemented cognitive architectures has been undertaken 1 Hamburg University of Applied Sciences, Department of Computer Science, Berliner Tor 7, 20099 Hamburg, Germany, {Marina.Tropmann-Frick,Thomas.Clemen}@haw-hamburg.de Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Towards Enhancing of Situational Awareness for Cognitive Software Agents 179 by [Sa10]. More recently, [KT] provides a summary of the history of architectures, noting 55 existing architectures. One of the most referenced models in this context is ACT-R ((Adaptive Control of Thought- Rational) [RTO19]. ACT-R was initially designed to model the visual perception and attention of humans [AMC97]. The ACT-R theory has a computational implementation as an interpreter of a specific coding language. The interpreter itself is written in Common Lisp and might be loaded into any of the Common Lisp language distributions. ACT-R reflects a theory of mechanisms that make up cognition. That theory posits a fixed set of mechanisms that use task knowledge to perform a task, thereby predicting and explaining the steps of cognition that form human behavior [RTO19]. As an example, [Sa06] has created a set of declarative and procedural rules about how to drive a car. He then added this knowledge to ACT-R that had its vision and motor systems connected to a car simulator (example taken from [RTO19]). Unfortunately, the structure of ACT-R and its implementation was difficult to incorporate into an agent-based simulation system. Therefore, the C4 brain architecture model [Is01] was evaluated and adapted for this purpose [TCK10]. C4 was originally intended to simulate autonomous and semi-autonomous creatures in games or other virtual environments. We enriched this architecture by an emotional component, a filter component following the OCC model [OCC88], and applied it to a pedestrian crowd dynamics scenario similar to what was described by [TGB09]. The results (see [TCK10] for some details) encouraged us to choose this architecture for this study as well. 1.1.1 The MARS Framework MARS2 is an agent-based modeling and simulation framework, which was developed at the Hamburg University of Applied Sciences, Germany [Hü]. It incorporates a domain-specific modeling language [Gl17], which helps scientists from a large variety of disciplines and without significant skills in programming developing their models. Modeling of cognitive processes and human decision-making is one of the focus research topics of the MARS Group. [TCK10] described the impact of emotions on pedestrians at a market place, whereas [LWC] utilized the goal-oriented action planning paradigm in MARS to evaluate adaptive behavior. 1.2 Data science approaches Data science is a rapidly emerging field incorporating a large amount of data-driven ap- proaches. A combination of data science methods and agent-based modeling is a challenging research area. Many approaches integrate, especially Reinforcement Learning, for their purpose. In [Ch17], the proposed approach is used to find time-efficient collision-free paths 2 www.mars-group.org 180 Tropmann-Frick, Clemen for multi-agent systems. The authors develop a value network that encodes the estimated time to the goal given an agent’s joint configuration with its neighbors. The use of the value network not only increases efficiency for finding a collision-free velocity vector but also considers the uncertainty in the other agents’ motion. Another approach using reinforcement learning focuses on a multi-agent setting for au- tonomous driving [SSS16]. The authors analyze especially the long term driving strategies and introduce a so-called option graph for reducing the variance of the gradient estimation. Several other works applied recently similar methods to the multi-agent domain, e.g., the Go [Ma14] and Altari [Ta17] games. The authors in [Su16] concentrate on the collaboration of multiple agents. The remarkable part of this work is the application of a rather simple neural model for a complex task of learning to communicate between the agents. 2 Modeling the cognitive process As we know from our own experiences, humans perceive their world more holistically, switching their attention to details if necessary. Thus, many of our everyday decisions are made unconsciously [Ka12]. Software agents, on the other side, explore their spatial vicinity sequentially, iterating through specific objects, e.g., other agents, points-of-interests. In some cases, it would be preferable that agents can perceive an entire situation in one step. The reasons behind might be a better alignment to human perception or a simple increase in processing performance. Mainly, if software agents are representing digital twins of biological or technical entities, both aspects could be substantial. 2.1 Sensory capture of a scene The cognitive perception of a scene relies on the input data provided by available sensors. These data can be used to provide a structured representation of the scene for further processing by the next components, such as the working memory component of the C4 architecture. The structured representation of the scene is used to define the situation awareness components of the agents. For this purpose, we use the profile and portfolio models of the agents similar to [JT14] and [TT10]. Analyzing the situation awareness of a particular agent, we use the profile of this agent for processing the available information based on the abilities of the agent. A profile defines the parameters of an agent, starting with a stereotype based on the social system and environment of the agent. It contains information about the personality that the agent represents with general properties, such as psychological context, education, training, behavioral pattern, preferences, expertise, knowledge, or experience. Using the learning component, the profile of the agent develops gaining more knowledge and experience as well as adapting to the communities and the changing environment and new situation parameters. The portfolio describes the involvement, tasks, and restrictions of an agent as well as their Towards Enhancing of Situational Awareness for Cognitive Software Agents 181 operating domains. A portfolio is specified by responsibilities and combined with the possible outcomes of the tasks. A task is an activity with an initial and a target state that includes interfaces or channels for communication and collaboration. Tasks also contain operating functions and are characterized by a set of parameters representing conditions and restrictions. The refinement of those parameters at run-time determines the execution accomplishment of the task. The result of task execution defines the transition to the target state and should satisfy the target conditions. 2.2 Individual perception and behavior selection Each agent constructs a predictive behavior model for potential behavior in the next step. This model is based on calculations with possible combinations of actions of an agent. These combinations can be constructed as graphs with edges representing transitions from one state (starting with the current state) to a possible next state. Possible transitions are defined by the actions from the portfolio of the agent. The situation awareness component of each agent also contains the perception of other agents. In a simulation environment, an agent has permanent access to the profile and portfolio information of other involved agents. In a specific scene, an agent can observe the environment and get information about the visible agents. This information can be used to predict possible situation development for all possible actions. Each possible action for the next step can be prioritized based on the evaluation of the resulting situation with a cost function. The following picture (Fig. 1) shows a schematic representation of the next step from the viewpoint of the left (blue) agent. The agent’s portfolio contains (in this example) three possible actions. There are also two more agents involved, whose actions influence the possible outcome and are considered in the evaluation of situation development for each action taken by the left agent. There are three possible situation developments, depicted as colored ovals, one for each possible action of the agent. For the other agents, only the most probable action predicted by the personal behavior model is considered in the evaluation. The action marked with a star can be considered as the one leading to the preferable situation development. However, it is not sufficient to predict the actions only for the next step. A more reliable model should be built by calculations for the next three or more possible steps, leading to complex situation development. A directed graph with possible action paths represents the predictive behavior model. Each path is evaluated based on probabilities and includes a priority function for the outcomes. The result of the evaluation is a ranking order of the paths. The next step for the agent will be chosen from the path with the highest ranking. Due to the exponentially increasing number of outcomes with each additional step, there are limitations for the calculated path length. To reduce the complexity caused by the variety of possible combinations and to define a robust priority function, a classifier will be trained based on historical data. This classifier 182 Tropmann-Frick, Clemen Fig. 1: Analysis of the situation development from the point of view of the left agent. calculates the probabilities of all possible actions in the portfolio of the agent. The decision on what action to take is then made based on the highest probability provided by the classifier. 2.3 Learning The learning of an agent is based on data science methods. The data science methods are responsible for the accurate situation evaluation, and adaptation to dynamic situation changes. Data generated in previous simulations will be used to build individual knowledge bases for the agents. This knowledge then can be used to improve the learning of the agent behavior model. The new goal as a result of this is the learning of interactions between the agents. The agents communicate and collaborate in their shared environment — this collaboration influence the behavior and negotiation abilities of the participating agents. One of the essential tasks is to keep the balance between adaptation to the unexpected behavior of other agents, or dynamic situation changes, and ’normal’ behavior (e.g., to be not too defensive). 3 Conclusions Instantaneous risk assessment of a situation in our daily lives, decision making, and learning are essential factors for human beings. Our conceptual approach describes the first brief idea of how to equip software agents with these capabilities by incorporating modern data science algorithms into their cognitive architecture. At the time of writing, concrete simulation results are still missing. However, capturing an entire scene instead of exploring it object by object would increase the operational Towards Enhancing of Situational Awareness for Cognitive Software Agents 183 performance of software agents. That is an essential issue if agents are considered as digital twins of humans or autonomous cars, for example. During the next months, we will implement the conceptual approach described above within the MARS Framework utilizing the C4 cognitive architecture. Together with colleagues from robotics, we will develop a digital twin for an autonomous vehicle using this concept. Here, real-time sensory input from an IoT infrastructure replaces the software sensors of the MARS agents. Bibliography [AMC97] Anderson, John R.; Matessa, Michael; Christian Lebiere: ACT-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention. Human-Computer Interaction, 12:439–462, 1997. [Ch17] Chen, Y. F.; Liu, M.; Everett, M.; How, J. P.: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 285–292, 2017. [Gl17] Glake, D; Weyl, J; Hüning, C; Dohmen, C; Clemen, T: Modeling through Model Transfor- mation with MARS 2.0. In: Proceedings of the 2017 Spring Simulation Multiconference. ADS ’17, Society for Computer Simulation International, Virginia Beach, Virginia, USA, p. 12, 2017. [Hü] Hüning, Christian; Adebahr, Mitja; Thiel-Clemen, Thomas; Dalski, Jan; Lenfers, Ulfia; Grundmann, Lukas; Dybulla, Janus; Kiker, Gregory A.: Modeling & Simulation as a Service with the Massive Multi-Agent System MARS. In: Spring Simulation Multiconference. ADS ’16 section 2, Society for Computer Simulation International, San Diego, CA, USA, pp. 1–8. [Is01] Isla, Damian; Burke, Robert; Downie, Marc; Blumberg, Bruce: A layered brain architecture for synthetic creatures. In: IJCAI’01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2. pp. 1051–1058, 2001. [JT14] Jaakkola, Hannu; Thalheim, Bernhard: Multicultural Adaptive Systems. In: Information Modelling and Knowledge Bases XXVI, 24th International Conference on Information Modelling and Knowledge Bases (EJC 2014), Kiel, Germany, June 3-6, 2014. pp. 172–191, 2014. [Ka12] Kahneman, Daniel: Thinking, Fast and Slow. Penguin, 2012. [KT] Kotseruba, Iuliia; Tsotsos, John K.: 40 Years of Cognitive Architectures: Core Cognitive Abilities and Practical Applications. Artificial Intelligence Review. [Li] Lieto, Antonio; Bhatt, Mehul; Oltramari, Alessandro; Vernon, David: The role of cognitive architectures in general artificial intelligence. Cognitive Systems Research, pp. 1–3. [LWC] Lenfers, Ulfia A.; Weyl, Julius; Clemen, Thomas: Firewood Collection in South Africa: Adaptive Behavior in Social-Ecological Models. Land, (3):97, aug. [Ma14] Maddison, Chris J; Huang, Aja; Sutskever, Ilya; Silver, David: Move evaluation in Go using deep convolutional neural networks. arXiv preprint arXiv:1412.6564, 2014. 184 Tropmann-Frick, Clemen [OCC88] Ortony, Andrew; Clore, Gerald L.; Collins, Allan: The Cognitive Structure of Emotions. Cambridge University Press, New York, NY, 1988. [Pr17] Prezenski, Sabine; Brechmann, André; Wolff, Susann; Russwinkel, Nele: A cognitive modeling approach to strategy formation in dynamic decision making. Frontiers in Psychology, 8(AUG), 2017. [RTO19] Ritter, Frank E.; Tehranchi, Farnaz; Oury, Jacob D.: ACT-R: A cognitive architecture for modeling cognition. Wiley Interdisciplinary Reviews: Cognitive Science, 10(3):1–19, 2019. [Sa06] Salvucci, Dario D.: Modeling driver behavior in a cognitive architecture. Human Factors, 48(2):362–380, 2006. [Sa10] Samsonovich, Alexei V.: Toward a Unified Catalog of Implemented Cognitive Architectures. In: BICA 2010. 2010. [SSS16] Shalev-Shwartz, Shai; Shammah, Shaked; Shashua, Amnon: Safe, Multi-Agent, Reinforce- ment Learning for Autonomous Driving. CoRR, abs/1610.03295, 2016. [Su16] Sukhbaatar, Sainbayar; Fergus, Rob et al.: Learning multiagent communication with backpropagation. In: Advances in Neural Information Processing Systems. pp. 2244–2252, 2016. [Ta17] Tampuu, Ardi; Matiisen, Tambet; Kodelja, Dorian; Kuzovkin, Ilya; Korjus, Kristjan; Aru, Juhan; Aru, Jaan; Vicente, Raul: Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017. [TCK10] Thiel-Clemen, Thomas; Klingenberg, Arne: Kombination von zielorientiertem Verhalten und Emotionen in Individuen-orientierten Simulationen. In (Wittmann, Jochen; Maretis, D K, eds): Simulation in den Umwelt- und Geowissenschaften, Workshop Osnabrück. Gesellschaft für Informatik, Shaker, pp. 71–80, 2010. [TGB09] Tasse, Flora Ponjou; Glass, Kevin; Bangay, Shaun: Simulating crowd phenomena in African markets. Proceedings of AFRIGRAPH 2009: 6th International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa, 1(212):47–52, 2009. [TT10] Thalheim, Bernhard; Tropmann, Marina: Performance Forecasting for Performance Critical Huge Databases. In: Information Modelling and Knowledge Bases XXII, 20th European-Japanese Conference on Information Modelling and Knowledge Bases (EJC 2010), Jyväskylä, Finland, 31 May - 4 June 2010. pp. 206–225, 2010. [TW12] Tuyls, Karl; Weiss, Gerhard: Multiagent Learning: Basics, Challenges, and Prospects. AI Magazine, 33(3):41, Sep. 2012.