An Event-Schematic, Cooperative, Cognitive Architecture Plays Super Mario Fabian Schrodt Yves Röhm Martin V. Butz Department of Computer Science Department of Computer Science Department of Computer Science Eberhard Karls University of Tübingen Eberhard Karls University of Tübingen Eberhard Karls University of Tübingen tobias-fabian.schrodt@uni-tuebingen.de yves.roehm@student.uni-tuebingen.de martin.butz@uni-tuebingen.de Abstract—We apply the cognitive architecture SEMLINCS continuous sensorimotor stream into event codes, which are to model multi-agent cooperations in a Super Mario game also closely related to the common coding framework and environment. SEMLINCS is a predictive, self-motivated control the theory of event coding [12], [13]. Already in [10] it was architecture that learns conceptual, event-oriented schema rules. We show how the developing, general schema rules yield coop- proposed that such event codes are very well-suited to be erative behavior, taking into account individual beliefs and envi- integrated into event schema-based rules, which are closely ronmental context. The implemented agents are able to recognize related to production rules [14] and rules generated by antic- other agents as individual actors, learning about their respective ipatory behavior control mechanisms [15]. As acknowledged abilities from observation, and considering them in their plans. As from a cognitive robotics perspective, event-based knowledge a consequence, they are able to simulate changes in their context- dependent scope of action with respect to their own interactions structures are as well eligible to be embedded into a linguistic, with the environment, interactions of other agents with the grammatical system [16]–[18]. environment, as well as interactions between agents, yielding We apply the principles of predictive coding and active coordinated multi-agent plans. The plans are communicated inference and integrate them into a highly modularized, cogni- between the agents and establish a common ground to initiate tive system architecture. We call the architecture SEMLINCS, cooperation. In sum, our results show how cooperative behavior can be planned and coordinated, developing from sensorimotor which is a loose acronym for SEMantic, SEnsory-Motor, SElf- experience and predictive, event-based structures. Motivated, Learning, INtelligent Cognitive System [19]. The architecture is motivated by a recent proposition towards a I. I NTRODUCTION unifed subsymbolic computational theory of cognition [20], Most of the approaches on intelligent, autonomous game which puts forward how production rule-like systems (such agents are robust, but behavior is typically scripted, pre- as SOAR or ACT-R) may be grounded in sensorimotor expe- dictable, and hardly flexible. Current game agents are still riences by means of predictive encodings and free energy- rather limited in their speech and learning capabilities as well based inference. The theory also emphasizes how active- as in the way they act believably in a self-motivated manner. inference-based, goal-directed behavior may yield a fully While novel artificial intelligent agents have been developed autonomous, self-motivated, goal-oriented behavioral system over the past decades, the level of intelligence, the interaction and how conceptual predictive structures may be learned by capabilities, and the behavioral versatility of these agents are focusing generalization and segmentation mechanisms on the still far from optimal [1], [2]. detection of events and event transitions. Besides the lack of truly intelligent game agents, however, SEMLINCS is essentially a predictive control architecture the main motivation for this work comes from cognitive sci- that learns event schema rules and interacts with its world ence and artificial intelligence. Over the past two decades, two in a self-motivated, goal- and information-driven manner. It major trends have established themselves in cognitive science. specifies a continuously unfolding cognitive control process First, cognition is embodied, or grounded, in the sensory- that incorporates (i) a self-motivated behavioral system, (ii) , motor-, and body-mediated experiences that humans and event-oriented learning of probabilistic event schema rules, other adaptive animals gather in their environment [3]. Second, (iii) hierarchical, goal-oriented, probabilistic reasoning, plan- brains are predictive encoding systems, which have evolved ning, and decision making, (iv) speech comprehension and to be able to anticipate incoming sensory information, thus generation mechanisms, and (v) interactions thereof. learning predominantly from the differences between predicted Here, our focus lies on studying artificial, cognitive game and actual sensory information [4]–[7]. Combined with the agents. Consequently, we offer an implementation of SEM- principle of free-energy-based inference, neural learning, as LINCS to control game agents in a Super Mario game envi- well as active epistemic and motivation-driven inference, a uni- ronment123 . Seeing that the game is in fact rather complex, fied brain principle has been proposed [8], [9]. Concurrently, it has been emphasized that event signals may be processed 1 https://www.youtube.com/watch?v=AplG6KnOr2Q in a unique manner by our brains. The event segmentation 2 https://www.youtube.com/watch?v=ltPj3RlN4Nw theory [10], [11] suggests that humans learn to segment the 3 https://www.youtube.com/watch?v=GzDt1t iMU8 Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 10 the implementation of SEMLINCS faces a diverse collection (i) of tasks. The implemented cognitive game agents are capable Motivational of completing Super Mario levels autonomously or coopera- System tively, solving a variety of deductive problems and interaction Intrinsic drives t al se al e en go go le v e ct n ev ed tasks. Our implementation focuses on learning and applying ed t k (ii) vo schematic rules that enable artificial agents to cause behav- (iv) (v) in iorally relevant intrinsic and extrinsic effects, such as collect- Schematic Speech Schematic Knowledge System Planning ing, creating, or destroying objects in the simulated world, Condition+Action in / out Event anticipation → Event carrying other agents, or changing an agent’s internal state, n such as the health level. Signals of persistent surprise in these an io n (iii) rv nt pl act ev dict io pr se ve at en ion e er domains can be registered [21], which results in the issuance ob e t t in Sensorimotor of event schema learning [20], and which is closely related to Planning the reafference principle [22]. As a result, production-rule-like, A* sensorimotor-grounded event schemas develop from signals of surprise and form predictive models that can be applied Fig. 1. Overview of the main modules and the cognitive control loop in the for planning. SEMLINCS thus offers a next step towards implementation of SEMLINCS. complete cognitive systems, which include learning techniques and which build a hierarchical, conceptualized model of their environment in order to interact with it in a self-motivated, in Figure 1. The motivational system (i) specifies drives that self-maintenance-oriented manner. activate goal-effects that are believed to bring the system A significant aspect when considering multi-agent architec- towards homeostasis. The drives comprise an urge to col- tures inspired by human cognition is cooperation and commu- lect coins, make progress in the level, interact with novel nication: Unique aspects of human cognition are characterized objects, and maintain a specific health level. Goal-effects by social skills like empathy, understanding the perspective of selected by the motivational system are then processed by an others, building common ground by communication, and en- event-anticipatory schematic planning module (ii) that infers gaging in joint activities [23]. As a step towards these abilities, a sequence of abstract, environmental interactions that are we show that the developing event-oriented, schematic knowl- believed to cause the effects in the current context. The edge structures enable the implemented SEMLINCS agents to interaction sequence is then planned in terms of actual motor cooperatively achieve joint goals. Thus, our implementation commands by the sensorimotor planning module (iii), which shows how sensorimotor grounded event codes can enable infers a sequence of keystrokes that will result in the desired and thus bootstrap cooperative interactions between artificial interactions. Both, the schematic and sensorimotor forward agents. SEMLINCS is designed such that the developing models used for planning are also used to generate forward knowledge structures and the motivational system can be simulations of the currently expected behavioral consequences. coupled with a natural language processing component. In our These forwards simulations are continuously compared with implementation, agents are able to learn from voice inputs the actual observations by the event-schematic knowledge and of an instructor, follow instructed goals and motivations, and learning module (iv), where significant differences are regis- communicate their gathered plans and beliefs to the instructor. tered as event transitions that cause the formation of procedu- Moreover, they can propose to and discuss with other game ral, context-dependent, event-schematic rules. The principle is agents potential joint action plans. closely related to Jeffrey Zacks and Barbara Tversky’s event In the following, we provide a general overview of the segmentation theory [10], [11] and the reafference principle modular structure of SEMLINCS in application to the Su- [22]. After a desired goal effect was achieved, the respective per Mario game environment. Moreover, we outline key as- drive that caused the goal is lowered, and a new goal is pects for coordinated cooperation in our implementation. We selected, completing an action cycle. The speech system (v) evaluate the system in selected multi-agent deduction tasks, provides a natural user interface to all of these processes, and focusing on learning, semantic grounding, and conceptual additionally enables verbal communication between agents. In reasoning with respect to agent-individual abilities, beliefs, and the following, we focus on the steps most relevant for our environmental context. The final discussion puts forward the implementation of coordinated joint actions: Event-schematic insights gained from our modeling effort, highlights important knowledge and planning. design choices, as well as current limitations and possible system enhancements. A. Event-Schematic Knowledge and Planning An event can be defined as a certain type of interaction that II. SEMLINCS IN A PPLICATION TO S UPER M ARIO ends with the completion of that interaction. An event bound- Here we give a brief overview of the main characteristics ary marks the end of such an event by co-encoding the encoun- of SEMLINCS in application to the Super Mario game en- tered extrinsic and intrinsic changes or effects. Since the possi- vironment. A detailed description is available in [19]. The ble interactions with the environment are context-dependent in implementation consists of five interacting modules as seen nature, we describe an event-schematic rule as a conditional, Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 11 probabilistic mapping from interactions to encountered event boundaries. Production-rule like schemas can be learned by means of Bayesian statistics under assumptions that apply in the Mario environment: Object interactions immediately result in specific effects, such that temporal dependencies can be neglected. Furthermore, the effects always occur locally, such that spatial relations can be neglected. Thus, in the Mario world, interactions can be restricted to directional collisions, which may result in particular, immediate effects, given a specific, local context. In the SEMLINCS implementation, event boundary detec- tion is implemented by detecting significant sensory changes that the agent does not predict by means of its sensorimotor forward model. Amongst others, these include changes in an agents’ health level or the number of collected coins, the destruction or creation of an object, or the action of lifting or dropping an object or another agent. Fig. 2. Expansion of the scope of action by simulating environmental The context for the applicability of a schematic rule, interactions. Red fields mark the reachable positions, while blue arrows denote however, is determined by different factors: It includes a the registered object interaction options, while simulating the scope of action. procedural precondition for an interaction, which specifies in Top row: The scope of action is updated by simulating the destruction of an object. Bottom row: The scope of action is updated by simulating the our current implementation the identity of actor and target as interaction with another agent. well as the intrinsic state of the actor (i.e. its health level). On the other hand, an environmental context precondition limits the applicable rules to the current scope of an action. That In the first example shown in Figure 2, an agent aims at is, the target of a schema rule must be available and the collecting a specific item (the coin on the top right). However, interaction with the target must be expected to lead to the this item is blocked by destructible objects (the golden boxes desired effect given the current situation. While the compliance to the right of the agent). Assume that the agent has already with procedural constraints can be determined easily, the learned that it can destroy and collect the respective objects. In reachability of objects has to be ascertained by an intelligent the initial situation (top left picture), however, the learned rule heuristic, which we describe in the following. about how to collect the coin is not applicable. The schematic planning module thus first simulates the destruction of one B. Simulating the Scope of Action of the blocking objects, and then updates the simulated scope of action. When there is more than one destructible object in The scope of action in a simulated scene is determined by the current scene, it furthermore has to identify the correct a recursive search based on sensorimotor forward simulations. object for destruction, that is, degeneralize the schematic rule The search starts at the observed scene or environmental with respect to the context (in the example, both objects are context and then simulates a number of simplified movement suitable). Next, the agent realizes that the desired item can be primitives in parallel. Each of the simulations results in a num- collected, given that one of the blocks was destroyed, resulting ber of collisions (or interactions), as well as a new, simulated in a schematic action plan. scene. Sufficiently different scenes are then expanded in the same manner, until the scope of action is sufficiently explored. C. From Schematic Planning to Coordinated Cooperation As a result, it encompasses the reachable positions as well Schema structures gathered from sensorimotor experiences as attainable interactions in a local context as provided by can be embedded into hierarchical, context-based planning. the sensorimotor forward simulation, neglecting, however, the Human cognition, however, is highly interactive and social. To effects that may result from the interactions. enable our architecture to act in multi-agent scenarios, it has to The simulation of changes in the scope of action is ac- (i) recognize other agents as individual actors (ii) observe and complished using the abstract, schematic forward simulation learn about their actions and abilities, (iii) consider them as of the local environment. In the current implementation, the actors in own plans (iv) consider them as possible interaction schematic forward model is applied by a stochastic, effect targets, and (v) communicate emerging plans. Since agents probability based Dijkstra search. In contrast to the sensori- may have different knowledge and scopes of action, this can motor forward model, it neglects the actual motor commands already result in simple cooperative behavior, for example, if but integrates the estimated, attainable interactions in the local the destruction of a specific block is needed but in the scope context as provided by the recursive, sensorimotor search. of action of another agent only. When specific interactions relevant to the scope of action are To yield a greater variety of cooperative scenarios, we addi- simulated (for example the destruction of a block) the scope tionally equip the agents with individual abilities. Specifically, of action is updated. agents are equipped with different jumping heights or the Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 12 unique ability to destroy specific blocks. As shown in Figure behavior. Videos showcasing these scenarios are available on- 2, the agents may then expand their scope of action when line45 . An additional scenario showing the negotiation process considering interactions with other agents during schematic is also available, but it is not included in this paper because planning. As a consequence, depending on the situation, agents it is not the main focus here 6 . may be committed to include other agents into their plans, as A. Toad Transports Mario will be shown in the experiments. While these principles are sufficient to model cooperative The first scenario is shown in Figure 5. In the initial scene planning, additional mechanisms are needed to account for (top left picture), the agent ‘Mario’ stands on the left, below the coordination and communication of plans. In our imple- an object named ‘simple block’ while the agent ‘Toad’ stands mentation, all schematic plans are strictly sequential, meaning close to Mario to the right side. Neither Mario nor Toad have that only one interaction by one agent is targeted at a time, gathered schematic knowledge about their environment so far. eliminating the need for a time-dependent execution of plans. Mario is instructed to jump and learns that if he is in his ‘large’ The communication of plans is done via the speech system health state and collides with a simple block from the bottom, by communicating (grammatical tags corresponding to) the the block will be destroyed. Next, he is ordered to jump to planned, abstract, schematic interaction sequences from the the right– essentially onto the top of Toad – resulting in Toad planning agent to possibly involved agents. Neither the con- carrying Mario and the learning of the option to ‘mount’ Toad crete, contextualized interaction sequence, nor corresponding and thus be carried around. As Mario is instructed to jump to sensorimotor plans are communicated. As a consequence, the right again, he also learns how to dismount Toad. Figure 4 the addressed agent has to infer the concrete instances of shows a graph of Mario’s schematic knowledge at this point. targeted objects that the planning agent is talking about. To do Actor / Target Interaction P = 1.0 Effect so, the agent performs contextual replanning to comprehend Preconditions Health: Large Actor: Mario Target: Simple Block Collision from below with simple block DESTRUCTION of simple block the proposed plan using his own knowledge – essentially Actor / Target Interaction P = 0.6 Effect mentally reenacting it. Given that the involved agent has Actor: Mario Target: Toad Collision from above with Toad MOUNT the agent Toad learned a different set of knowledge than the planning agent, Interaction P = 0.6 Effect it is likely to end up with a different plan and a different Collision from left with Toad DISMOUNT the agent Toad overall probability of success. In our current implementation, an involved agent accepts a proposed plan when it does not Fig. 4. Mario’s schematic knowledge in scenario 1. The respective entries have another solution for the targeted goal that is more likely are put into context by the schematic planning module. successful than the proposed plan given its knowledge. Given the involved agent gets to a different plan, it makes a counter Equipped with this knowledge, Mario is ordered by voice proposal that is always accepted by the initial planning agent. input to ‘destroy a simple block’. This sets as goal effect the The process of negotiation is shown in Figure 3. destruction of a simple block object which activates planning in the schematic knowledge space. As can be seen in Figure 5, Makes plan to reach plan includes another agent? no Start sensorimotor the only simple block is located at the top right in the current a goal event planning context. In this implemented scenario, Toad is able to jump yes Propose plan to higher than Mario, such that he can jump to the elevation, involved agent while Mario is not able to do so. Thus, a direct interaction with the simple block is not possible for Mario as it is not in Contextual replanning Start sensorimotor yes accept plan ● Application of own knowledge Mario’s current scope of action. planning ● Schema degeneralization ● Plan probability comparison The schematic planning is thus forced to consider other pre- no viously experienced interactions in the context of the current Counterproposal of plan situation. We assume that all agents have full knowledge about accept plan the sensorimotor abilities of the others. Thus, inferring that it Start sensorimotor will expand his scope of action, Mario simulates to jump on planning the back of Toad, followed by Toad transporting Mario to the Fig. 3. Negotiation diagram for two agents. Blue boxes: Tasks of the planning elevated location on the right. Because the combined height of agent. Red boxes: Tasks of an agent involved in the initial plan. Grey boxes: Mario and Toad is too tall to pass through the narrow passage Both agents are planning. where the simple block is located, a dismount interaction is simulated subsequently. Finally, Mario is able to destroy the III. E VALUATION simple block since it is now in his scope of action. We evaluated the resulting cooperative capabilities of SEM- This interaction plan is then negotiated between the two LINCS by creating exemplar scenarios in the Super Mario agents before they start sensorimotor planning. As Toad ob- world, which illustrate the cooperative abilities of the agents. served Mario and thus learned the same knowledge entries, he We show two particular, illustrative evaluations. However, 4 Scenario 1: https://youtu.be/0zle8L6H- 4 we have evaluated SEMLINCS in various, similar scenarios 5 Scenario 2: https://youtu.be/WzOg WcNDik and have observed the unfolding of similarly well-coordinated 6 Additional Scenario: https://youtu.be/7RV4QCwDK8U Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 13 Fig. 6. Scenario 2: Mario helps Toad to collect a coin. ing agents establishes a common ground, consisting of the final goal an agent wants to achieve as well as the interactions it plans to execute while pursuing the final goal. IV. C ONCLUSION Fig. 5. Senario 1: Toad helps Mario to destroy a block. Humans are able to understand other agents as individual, intentional agents, who have their own knowledge, beliefs, perspectives, abilities, motivations, intentions, and so their infers the same schematic plan and thus considers the proposal own mind. [24]–[26]. Furthermore, we are able to cooperate useful and accepts. After the agreement, both agents plan with others highly flexibly and context-dependently, which their part of the interaction sequence in terms of keystrokes requires coordination. This coordination can be supported by (top right picture) and wait for the other agent to execute its communication, helping to establish a common ground about part when necessary. The resulting execution of the plan is a joint interaction goal. shown in the following pictures: Mario mounting Toad; Toad In the presented work, we showed how social cooperative transporting Mario to the elevated ground; Mario dismounting skills can be realized in artificial agents. To do so, we equipped Toad and finally Mario moving to the simple block and the agents with different behavioral skills, such that particular destroying it. goals could only be reached with the help of another agent. To coordinate a required joint action, SEMLINCS had to B. Mario Clears a Path for Toad enable agents to learn about the capabilities of other agents by In the second scenario, shown in Figure 6, Toad is at first observing other agent-environment interactions and to assign instructed to collect the coin object, while Mario is ordered the learned event schema rules to particular agents. Moreover, to destroy the simple block (see top left picture). We assume our implementation shows how procedural rules can be applied that Toad is not able to destroy a simple block by himself, to a local, environmental context, and how sensorimotor and and does not generalize that he can do so as well. Toad is more abstract schematic forward simulations can be distin- instructed to increase his number of coins (top right picture). guished in this process, and applied to build an effective, hier- Although he knows that a collision with a coin will yield the archical planning structure. Besides the computational insights desired effect, there is no coin inside his scope of action, since into the necessary system enhancements, our implementation the only coin in the scene is blocked by a simple block. Thus, opens new opportunities for future developments towards even the schematic planning module anticipates a destruction of more social, cooperative, artificial cognitive systems. the simple block by Mario (bottom left picture), expanding First of all, currently the agents always cooperate. A con- Toad’s scope of action. After that, Toad is able to collect the ditional cooperation could be based on the creation of an coin (bottom right picture). incentive for an agent to share its reward with the participating Both shown scenarios demonstrate how SEMLINCS agents partner agent. Indeed, it has been shown that a sense of fairness are able to learn about each other, include each other in their in terms of sharing rewards when team play was necessary is action plans by recognizing individual scopes of action in an a uniquely human ability [27]. While a sense of fairness is a environmental context, and coordinate the joint execution of motivation to share when help was provided – or also possibly the plans. Communicating cooperative goals to the participat- when future help is expected, that is, expecting that the partner Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 14 will return the favor – a more long term motivation can create O. Sigaud, and P. Gérard, Eds. Berlin Heidelberg: Springer-Verlag, social bonds by monitoring social interactions with partners 2003, pp. 86–109. [6] M. V. Butz, “How and why the brain lays the foundations for a conscious over time and preferring interactions and cooperations with self,” Constructivist Foundations, vol. 4, no. 1, pp. 1–42, 2008. those partners that have shared rewards in the past in a fair [7] K. Friston, “Learning and inference in the brain.” Neural Netw, vol. 16, manner. Clearly many factors determine if one is willing to no. 9, pp. 1325–1352, 2003. [8] ——, “The free-energy principle: a rough guide to the brain?” Trends cooperate, including social factors, game theory factors, and in Cognitive Sciences, vol. 13, no. 7, pp. 293 – 301, 2009. related aspects – all of which take the expected own effort [9] A. Clark, “Whatever next? predictive brains, situated agents, and the into account, the expected effort of the cooperating other(s), future of cognitive science,” Behavioral and Brain Science, vol. 36, pp. 181–253, 2013. as well as the expected personal gain and the gain for the [10] J. M. Zacks and B. Tversky, “Event structure in perception and concep- others. tion,” Psychological Bulletin, vol. 127, no. 1, pp. 3–21, 2001. It also needs to be noted that currently action plans are [11] J. M. Zacks, N. K. Speer, K. M. Swallow, T. S. Braver, and J. R. Reynolds, “Event perception: A mind-brain perspective,” Psychological executed in a strict, sequential manner. In the real world, Bulletin, vol. 133, no. 2, pp. 273–293, 2007. however, joint actions are typically executed concurrently, [12] B. Hommel, J. Müsseler, G. Aschersleben, and W. Prinz, “The theory of such as when preparing dinner together [25]. Thus, in the event coding (TEC): A framework for perception and action planning,” Behavioral and Brain Sciences, vol. 24, pp. 849–878, 2001. near future we will face the challenge to allow the parallel [13] W. Prinz, “A common coding approach to perception and action,” in execution of cooperative interactions, which will make the Relationships between perception and action, O. Neumann and W. Prinz, timing partially much more critical. Eds. Berlin Heidelberg: Springer-Verlag, 1990, pp. 167–201. [14] A. Newell and H. A. Simon, Human problem solving. Englewood Although our agents already communicate plans on an Cliffs, NJ: Prentice-Hall, 1972. abstract, schematic level, all sequential steps of the plans need [15] M. V. Butz and J. Hoffmann, “Anticipations control behavior: Animal to be fully verbalized in order to coordinate a joint action behavior in an anticipatory learning classifier system,” Adaptive Behav- ior, vol. 10, pp. 75–96, 2002. at the moment. An alternative would be to simply utter the [16] P. F. Dominey, “Recurrent temporal networks and language acquisition: goal and ask for help, thus expecting the other agent to help from corticostriatal neurophysiology to reservoir computing,” Frontiers under consideration of the known behavioral abilities of the in Psychology, vol. 4, pp. 500–, 2013. [17] K. Pastra and Y. Aloimonos, “The minimalist grammar of action,” individual agent. Therefore, more elaborate theories of mind Philosophical Transactions of the Royal Society B: Biological Sciences, would need to be taken into consideration [28]. For example, vol. 367, pp. 103–117, 2012. in the first scenario mentioned above, Toad may realize that [18] F. Wörgötter, A. Agostini, N. Krüger, N. Shylo, and B. Porr, “Cognitive agents–a procedural perspective relying on the predictability of object- he needs to transport Mario to the higher ground on the action-complexes (OACs),” Robotics and Autonomous Systems, vol. 57, right to enable Mario to destroy the box up there, because no. 4, pp. 420–432, 2009. Mario cannot reach this area. Humans are clearly able to [19] F. Schrodt, J. Kneissler, S. Ehrenfeld, and M. V. Butz, “Mario becomes cognitive,” TOPICS in Cognitive Science, in press. utter or even only manually signal a current goal and still [20] M. V. Butz, “Towards a unified sub-symbolic computational theory of come up with a joint plan, without verbally communicating cognition,” Frontiers in Psychology, vol. 7, no. 925, 2016. the plan in detail. While verbal communication certainly helps [21] M. V. Butz, S. Swarup, and D. E. Goldberg, “Effective online detection of task-independent landmarks,” in Online Proceedings for the ICML’04 in the coordination process, obvious interactions can also un- Workshop on Predictive Representations of World Knowledge, R. S. fold successfully without communication (e.g. letting another Sutton and S. Singh, Eds. online, 2004, p. 10. [Online]. Available: pedestrian pass; passing an object out of reach of another http://homepage.mac.com/rssutton/ICMLWorkshop.html [22] E. von Holst and H. Mittelstaedt, “Das Reafferenzprinzip (Wechsel- person, who apparently needs it). Although the Mario world wirkungen zwischen Zentralnervensystem und Peripherie.),” Naturwis- is rather simple, cooperative interactions of this kind could senschaften, vol. 37, pp. 464–476, 1950. actually be enabled when enhancing the current SEMLINCS [23] M. Tomasello, A Natural History of Human Thinking. Harvard University Press, 2014. architecture with the option to simulate potential goals of the [24] R. L. Buckner and D. C. Carroll, “Self-projection and the brain,” Trends other agent and plans on how to reach them, thus offering a in Cognitive Sciences, vol. 11, pp. 49–57, 2007. helping hand wherever it seems necessary. [25] N. Sebanz, H. Bekkering, and G. Knoblich, “Joint action: Bodies and minds moving together,” Trends in cognitive sciences, vol. 10, pp. 70–76, 2006. R EFERENCES [26] M. Tomasello, M. Carpenter, J. Call, T. Behne, and H. Moll, “Un- [1] S. M. Lucas, M. Mateas, M. Preuss, P. Spronck, and J. Togelius, derstanding and sharing intentions: The origins of cultural cognition,” “Artificial and Computational Intelligence in Games (Dagstuhl Seminar Behavioral and Brain Sciences, vol. 28, pp. 675–691, 2005. 12191),” Dagstuhl Reports, vol. 2, no. 5, pp. 43–70, 2012. [Online]. [27] K. Hamann, F. Warneken, J. R. Greenberg, and M. Tomasello, “Collab- Available: http://drops.dagstuhl.de/opus/volltexte/2012/3651 oration encourages equal sharing in children but not in chimpanzees,” [2] G. N. Yannakakis and J. Togelius, “A panorama of artificial and Nature, vol. 476, no. 7360, pp. 328–331, 2011. computational intelligence in games,” Computational Intelligence and [28] C. Frith and U. Frith, “Theory of mind,” Current Biology, vol. 15, no. 17, AI in Games, IEEE Transactions on, vol. PP, no. 99, pp. 1–1, 2014. pp. R644–R645, 2005. [3] L. W. Barsalou, “Grounded cognition,” Annual Review of Psychology, vol. 59, pp. 617–645, 2008. [4] J. Hoffmann, Vorhersage und Erkenntnis: Die Funktion von Antizipa- tionen in der menschlichen Verhaltenssteuerung und Wahrnehmung. [Anticipation and cognition: The function of anticipations in human behavioral control and perception.]. Göttingen, Germany: Hogrefe, 1993. [5] M. V. Butz, O. Sigaud, and P. Gérard, “Internal models and anticipations in adaptive learning systems,” in Anticipatory Behavior in Adaptive Learning Systems: Foundations, Theories, and Systems, M. V. Butz, Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 15