What do games teach us about designing effective human-AI cooperation? - A systematic literature review and thematic synthesis on design patterns of non-player characters Maximilian Wittmann1 and Benedikt Morschheuser1 1 Friedrich-Alexander-Universität Erlangen-Nürnberg, Gamification & Digital Customer Engagement Research Group, Lange Gasse 20, Nürnberg, Germany Abstract Effective cooperation between humans and technologies powered by Artificial Intelligence (AI) is decisive to fully exploit AI’s economic and social potentials. However, the adoption of AI is often opposed by a lack of humans’ trust in AI systems and a dearth of interest in working with them. Turning to games for getting inspiration on how to optimize human-AI cooperation seems promising, since games engage humans almost effortlessly in interacting and cooperating with artificial non-player characters (NPCs). However, a structured overview on how game design can optimize human-AI cooperation is missing in existing gamification research. Therefore, this paper presents a systematic review of NPC design patterns and elaborates on what developers of AI systems can learn from game design. Guided by a thematic analysis, we present a structured overview of relevant design patterns clustered along six focus fields - namely I) NPC responsiveness, (II) appearance of NPCs, (III) NPC communication patterns, (IV) emotional aspects, (V) behavioral characteristics, and (VI) player-NPC and NPC-NPC team structures – which advance our understanding of designing and investigating cooperation between humans and NPCs. The insights of this paper can guide practitioners and future research regarding the design of more effective AI systems, the gamification of human-AI cooperation, and the development of innovative NPC approaches. Keywords 1 Non-Player Characters, Human-AI Cooperation, Systematic Literature Review, Thematic Synthesis, Artificial Intelligence, Design Patterns 1. Introduction papers elaborating on this topic. Despite this interest, we lack a clear understanding of how specific design aspects of AI systems can With the rise of Artificial Intelligence (AI) and optimize the human-AI cooperation and establish increasingly autonomous machines, human-AI trust between humans and AI systems [1]. cooperation has received a surge in attention in One context where cooperation between industry and academia. In areas as diverse as humans and AI appears to emerge effortlessly is human-robot interaction, autonomous driving, or video games. Existing research demonstrated that the assistance of humans in complex decision- specific game design features could engage making with expert systems, seamless players in developing strong emotional cooperation between humans and AI technologies relationships [2] with non-player characters is decisive to enable society and businesses to (NPCs), support the perceived closeness, and fully exploit AI’s benefits and potentials. The even build trust. Design knowledge and patterns growing interest is reflected by a rise of research from game design and, in particular, NPC design Proceedings of the 6th International GamiFIN Conference 2022 (GamiFIN 2022), April 26–29, 2022, Tampere, Finland EMAIL: maximilian-wittmann@gmx.de (A. 1); benedikt.morschheuser@fau.de (A. 2) ORCID: 0000-0003-4042-7300 (A. 1); 0000-0002-7665-8971 (A. 2) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 95 can thus provide a hitherto scarcely explored and players can compete or cooperate with NPCs; treasure of knowledge for designing more however, humans can quickly become frustrated effective human-AI collaboration outside of with NPCs that show deviating, non-human-like games. Lending elements from video games and or predictable behaviors [12–16]. Recently utilizing them in other contexts, such as AI emerging developments in the field of advanced systems, has become popular in recent years. This AI pave the way towards more realistic NPCs and trend is called gamification and refers to the use thus more immersive gameplay [17, 18]. of design principles and features of games outside Even though NPCs are prevalent in games and traditional video game environments with the interest in developing more robust NPCs [23] is intention to afford similar experiences as in games high, game research missed studying NPC design and to influence behaviors [3]. patterns in greater detail [24]. One recent work While various studies indicate, that game investigates central design components of design knowledge can help improve the design of companions in video games [25], expanding a AI systems, the bulk of the gamification research design space proposed in [26]. While these that has emerged over the past ten years missed to contributions are relevant for this paper, provide a structured overview of gamifying companions only resemble one category within human-AI cooperation [4]. Therefore, this study the broader class of NPCs. Evidently, there is a aims at answering the research question: gap of systematic review papers that deal with design patterns of NPCs. Current literature in Which design patterns facilitate effective NPC design remains fragmented and little is cooperation between NPCs and humans? known on how to transfer the insights gained from NPCs to other non-game contexts. Different This paper’s major contribution is conducting studies indicate, however, that game design a systematic literature review and thematic knowledge could optimize future human-AI synthesis as well as investigating which patterns cooperation and improve AI systems [32, 35]. game developers and designers exploit for Gamification research has overlooked to provide building rich social interactions between NPCs structured knowledge on the gamification of and humans (i.e., player characters (PCs)). Our human-AI cooperation thus far [4]. results are based on performing axial and selective coding to derive subcategories and linkages 3. Research methodology between the codes and summarize the current body of knowledge in a systematic way. Finally, In this paper, we present a systematic literature we offer practical recommendations as to what AI review on the topic guided by Webster and software developers and experts in Human- Watson [27]. The literature review has been computer interaction can learn from the gaming conducted on the Scopus database. The choice of industry. scientific database is justified by two reasons: First, Scopus aggregates several relevant 2. Non-player characters in video databases such as ACM, IEEE, or Springer. games Second, the focus on one single scientific research database allows a replicable process and thus The term NPC refers to any character found in supports the rigor and objectivity of the procedure a game not controlled by the players [5]. In many [28]. games, players play with or against NPCs. NPCs We performed the literature search on August are used to increase the believability of games and 26th in 2021, querying the Scopus database in the a player’s immersion in the virtual game world following manner: TITLE-ABS-KEY(NON- [6–8]. Human players are keen to interact with PLAYER CHARACTERS AND DESIGN*). The realistic NPCs and research indicates that players search yielded results focusing on non-player can even establish strong relationships with NPCs characters and any permutation of the term design. [19]. By carefully limiting the search to the metadata, In the last decades, game developers and this approach enabled us to scan literature only for designers have placed a primary focus on publications concentrating on our intended search increasing NPC believability [11] and creating the terms. The search resulted in 295 hits. Next, we illusion of playing with human-like fellows. NPCs performed several screening steps based on the traditionally follow a deterministic AI behavior following criteria to include only relevant papers: 96 1) Removal of duplicates and false hits (-22 4. Results papers); 2) Abstract and title screening and subsequent removal of papers with a focus not in 4.1. Descriptive information line with the research question at hand (-77 papers); 3) Removal of papers not written in Out of the 174 reviewed full papers, 116 are English (-4 papers); 4) Removal of papers that are empirical studies. 72 papers contain empirical not full papers (-17 papers) and 5) Papers that results related to NPC design patterns and human- cannot be acquired (-1 paper). This screening AI cooperation. 26 papers are conceptual or process resulted in 174 full papers. Then, we present frameworks, methodologies, or models. coded the works by accumulating information on 22 papers are reviews, while 14 studies introduce bibliometric and descriptive information. preliminary results, describe systems, case studies Subsequently, we applied thematic synthesis or prototypes. 113 papers fall into the domain according to [29]. This approach was chosen as it entertainment. The second largest category is allows to investigate phenomena in qualitative education with 46 papers. Seven papers belong to data, such as prototype descriptions, and aims at the domain of culture/history/ethics, four papers generating implications for practice. This is in line deal with medicine and health, and two papers with our goal to encourage designers to draw belong to the domain engineering. The domains inspiration from NPC design for improved sports and tourism each comprise one paper. human-AI cooperation. The synthesis comprises three stages: 4.2. Responsiveness of NPCs Free coding: A sample of ten articles was read and reviewed. Inductive line-by-line coding led to The structured review of previous research on the identification of multiple design features. NPC design reveals that the majority of the Based on the number of papers in the dataset, we empirical studies employ design patterns related applied an additional second round of open coding to the responsiveness of NPCs. In this field of with five articles. Next we added codes to the research, the most popular design features can be fragments. The plausibility of this preliminary clustered in features related to how NPCs provide coding scheme was checked by carefully reading feedback and are able to learn and respond. all papers in the dataset. As a result, six additional The design patterns in these categories aim codes were added to prevent neglecting relevant directly at facilitating more effective cooperation design pattern subcategories. between NPCs and human players. Construction of descriptive themes: The A total of 38 studies investigates or employs obtained codes were iteratively compared. The NPC feedback mechanisms. Feedback has been findings were synthesized and similarities as well shown to be powerful in influencing people’s as differences between the obtained codes, were decision making [31] and bringing about behavior identified. Descriptive themes were generated change. The review indicates that NPC feedback through axial coding. can further be divided into four thematic groups Development of analytical themes: We (as visible in Table 1): Direct feedback (e.g., reviewed the entire body of knowledge and [32]), delayed feedback (e.g., [33]), NPC-PC co- mapped the content on the defined themes. All creation (e.g., [34, 35]), and persuasion of the data was classified along with the following player (e.g., [31, 36]). overarching themes: design patterns on (I) NPC In appropriate contexts, specific NPC responsiveness, (II) appearance of NPCs, (III) feedback seems to be able to serve as a stimulator NPC communication patterns, (IV) emotional of curiosity or even an augmenter of human aspects, (V) initiative of NPCs, and (VI) PC-NPC creativity. For instance, Ali et al. [31] demonstrate and NPC-NPC team structures. These analytical that NPCs designed as artistic playmates themes comprised several subcategories and thus providing creative feedback can increase kids’ resulted in a tree structure. creativity compared to similar playmates, which In terms of the presentation of the results, we provide less creative feedback. This type of follow Paré’s assessment [30] and present the feedback is shown to significantly increase the synthesized evidence mainly in tabular form. participant’s creativity and consequently improves the quality of the human-AI cooperation. 97 29 studies indicate that especially three human gameplay (for example learning by patterns related to feedback are highly relevant for demonstration, utilizing external hardware), and achieving effective PC-NPC cooperation: NPCs learning from fellow NPCs/AI (within or assessment of player’s performance/progress (15 outside the current domain). studies), immediate feedback (11 studies) and The generated overview reveals that while a unpredictability (13 studies). Embedding socio- large diversity of approaches exists, several focus emotional elements and unexpected moral fields can be identified (cf. Table 2). For example, questioning prompts can help augment NPC only three papers [68, 72, 73] deal with inter- and believability and the level of player immersion, as cross domain learning where NPCs can learn from called for in [12, 13]. The insights of [42] fellow NPCs. This is probably because this demonstrate that design features allowing players approach is quite new and complex to realize. The to observe an NPC’s vulnerability and experience results of [72] and [73] indicate, however, that its decision-making process first-hand can trigger minimizing the NPC’s learning and training times reflection on the player’s side and increase the can lead to faster acceptance and the player can emotional investment in the game. This can be exploit the NPC’s skill sooner and interact more achieved through perspective switching exercises naturally. The empirical findings demonstrate the that serve to confront a player with several daily benefits of this approach compared to scenarios social dilemmas (such as stealing in a shop, being where a new NPC in a game cannot lend skills bullied by peers) that NPCs face and make him from a fellow NPC and needs to be trained based assess the NPC’s decision-making. on human performance from scratch. For instance, inter-domain learning allows sharing of Table 1 knowledge as well as prior experiences among Coverage of patterns related to NPC feedback fellow NPCs within one domain. The second sub- Theme Reference category is cross-domain learning. Studies [68, Direct feedback and instant 73] demonstrate that this design pattern can replies facilitate the human-AI cooperation in the long ● Assessment of player’s [32–46] performance/progress term because it allows NPCs to transfer their skills ● Open-ended or free-flowing [33, 49] from one domain onto new fields and enable dialogue (PC-NPC | NPC-NPC) NPCs with more general capabilities. This can ● Socio-moral decision making [42, 50, 58–63] ● Immediate feedback [2, 26, 37–41, 44, 54–56] positively influence trust of humans in NPC by Delayed feedback simulating human learning processes and creating ● Gradual revealing of [33] a sense of likeness in terms of cognitive information capabilities. Inter-domain learning is particularly Persuade player and bring about change important in games that feature importing NPCs ● Evoking of strong emotional [48, 50, 57–59] from one game to another. For instance, agents reactions that were trained on how to ride a bike could ● Embedding of elements of [37, 59, 61, 62] surprise (e.g., humor, off-topic explicitly utilize that knowledge for riding a remarks) motorcycle in a new context. However, the bulk ● Increasing of unpredictability [34, 43, 44, 46, 48, 50, 58, (e.g., unexpected actions, 62–67] of the empirical studies apply features that allow shocking of player) humans to learn from an NPC or vice versa. In the NPC-PC co-creation empirical papers, NPCs training has been ● Real-time corrections [35] ● Augmentation of human [31, 34, 59, 68] achieved through e.g., the usage of learning by creativity demonstration [35], optimization algorithms ● Triggering of curiosity [31, 43, 64, 68–70] (such as Reinforcement Learning [81–84]), or Supervised Learning (e.g., Artificial Neural Moreover, this review indicates that the NPC’s Networks [78, 79]) approaches. ability to learn and respond is crucial for Several studies apply design patterns that enhancing both the level of game immersion [51] enable NPCs to learn from human gameplay. and the interestingness [71] of the PC-NPC These approaches serve to create profiles of interaction. 38 papers are dealing with this theme, human players and train for imitation (used in 11 as depicted in Table 2. This category can be papers), exploiting shared memories (used in 6 divided into three subcategories, differentiating papers), or utilizing external hardware (such as between humans learning from AI/NPC (through EEG-based BCI devices [48], webcams or Kinect social comparisons, switching perspectives, or systems [45], used in 4 papers) to capture triggering emotions), NPCs learning based on movements and emotions in real time. 98 The review emphasizes that certain design Moreover, several different communication approaches are particularly suitable for supporting patterns are found: a) the applied modalities (such humans to learn from their NPC counterparts. In as text-based, natural language or BCI), b) 7 studies, this is accomplished through either verbal/non-verbal communication enriched by social comparison, confronting human players gestures, body language, levels of assertiveness, with NPC decision-making, or deliberately and c) the direction of communication (e.g., PC- controlling the pace of the learning process (e.g., NPC, NPC-NPC, PC-PC, see [31] and [56]). The through inobtrusive buttons to deliberately call results indicate that lively conversations with NPCs for help [69]). Three features, however, are references to real-world experiences [17] and especially prominent in the reviewed studies: situations are more effective in terms of perspective switching (7 studies), deliberately engagement and player enjoyment than non- triggering emotions (6 studies), and monitoring interactive, pre-programmed NPC conversations. and adapting difficulty levels (7 studies). The category emotional aspects comprises patterns related to empathy, the power of narrative Table 2 and backstories, embedding motivational Coverage of patterns related to the ability to elements such as points, scores and leaderboards, learn and respond humor/satire, and love [6, 26, 70]. This is visible, Theme Reference for instance, in the study of Mallon and Lynch Humans learn from AI/NPC [62] that recommends integrating elements of ● Confronting human players with [39, 44] NPC decision-making romantic relationships with NPCs to add an ● Application of social [31–33, 50] additional dimension of human experience and comparison creating more intriguing PC-NPC partnerships. ● Control of the learning process [69] ● Perspective switching with AI [50, 65, 68, 69, 70, 82, 83] Further, our data show that the NPC’s degree ● Reinforce learning by triggering [36, 52, 59, 60, 71, 80] of autonomy and personality traits are relevant of emotions design patterns we summarize as behavioral ● Monitoring of PC and adapting [32, 37, 43, 45, 48, 69, 81] of difficulty level characteristics. These contain the degree of NPCs learn based on human involvement of an NPC in the PC’s game gameplay experience and an NPC’s own agenda (cf. e.g., ● Mimicking/Modeling of player [24, 32, 37, 65–67, 70, 74– and striving for imitation 77] [50, 52]). Creating unique NPCs and controlling ● Learning by demonstration [31, 35] when and how they intervene are demonstrated to ● Taking advantage of external [37, 38, 45, 48] be promising ways to increase the player’s hardware ● Exploitation of a (shared) [2, 31, 56, 62, 81, 85] curiosity and facilitate immersion [31, 44]. memory Lastly, the category PC-NPC and NPC-NPC NPCs learn from fellow team structures captures features related to the NPCs/AIs ● Inter-domain learning [72, 73] team dynamics and the role of each actor in ● Cross-domain learning [68, 72] sociotechnical systems. Our results indicate that NPCs that possess knowledge about previous 4.3. Other NPC design categories incidents and preferences of the player can more easily create a personalized game atmosphere. Through design patterns that allow NPCs to build The review highlights the importance of memories, a collection of relevant shared responsiveness for facilitating effective NPC-PC experiences with the player are created. By taking cooperation, which was reflected in the amount of advantage of this wealth of shared experiences coverage across the studies. Nonetheless, this and proactively suggesting actions based on review identifies five further categories with previous player preferences, the NPC comes design features that can improve the human-AI across as a non-static and adaptable counterpart interaction. [72]. This, in turn, serves to strengthen and mature Appearance comprises features related to the relationship with the player [56]. Additionally, anthropomorphism, such as human likeness, taking turns with the human can create a more customization, tone of voice, facial expression, captivating experience since the NPC’s reactions and embodiment (cf. [38, 44]). The patterns of appear more natural and may remind the player of this category can play a vital role in the human-human conversations. This pattern is cooperation because the player’s perception of the especially useful in dialogues [38] or when NPC highly affects the team dynamics [2]. elaborating choices at decision points [44]. 99 5. Discussion presented design patterns can guide the design of future AI systems outside games. For instance, designers of AI systems could implement aspects This study investigates design patterns of of perspective switching with an AI system, as NPCs that facilitate cooperation between NPCs shown to be promising in NPC design by [51]. and human players in existing research. This adds Further, design patterns such as the active design to previous research in the field of companion of perceivable vulnerable AI, reinforcing players’ design [25, 26] through a broader consideration of learning processes through deliberately triggering this relevant phenomenon. The study’s main emotions, or actively confronting users with the contribution is an explorative elaborated novel reasoning behind an AI’s decision-making could overview of categories and design patterns that guide future AI design for supporting human-AI advance our understanding of how specific design cooperation. features facilitate human-AI cooperation. Thirdly, we found that NPC design This research illustrates that reaching a high increasingly employs various patterns related to level of NPC believability is a difficult mission. It an AI’s learning from the player behavior. This involves elements such as goals, proper reaction trend is illustrated through 23 empirical papers in abilities, non-verbal communication [9], emotion which NPC learning is triggered by human and social-emotional cognition [10], dynamic gameplay. The corresponding approaches can dialogues [20], adapting to the player [21], and the also be very valuable in gamification design. They quest for more meaningful interaction [22]. The could guide future research on further study discovers that several clusters exist, such as personalization of gamification which is required feedback mechanisms that aim to influence player to prevent a one-size-fits-all approach [86]. behavior or approaches of mutual learning. Applying NPC learning approaches in To the best of our knowledge, this research is gamification may support personalized need the first work to holistically investigate NPC satisfaction and increase the effectiveness of learning processes in video games. The systematic gamification for various target groups. screening of the existing body of knowledge Further, our results reveal several reveals that learning can occur on several levels: shortcomings in the current body of knowledge a) NPCs being either directly responsible for it by that could guide further research in this field: triggering emotions or allowing for perspective- 1. Future studies should empirically taking, stimulating, or teaching humans, b) NPCs investigate the effects of single design learning based on human behaviors and patterns. The isolated consideration is gameplay, c) inter-domain and cross-domain important to assess the applicability as well as learning with NPCs learning from fellow bots. the actual effectiveness of the identified Our study adds to previous research in several patterns. ways: Firstly, this study can offer new pathways 2. Also, gamification research has largely for developing more compelling NPC characters overlooked applying NPC designs in non- in games and serious games. We recommend that game contexts [87]. Future research should designers actively embed NPC feedback develop empirically evaluated frameworks elements, including direct/delayed feedback or that can guide scientists and practitioners in NPC-PC co-creation. These features are shown to further leveraging the potentials of NPC be powerful in influencing people’s decision- design outside games. making and behaviors [31]. Consequently, game designers should diversify and enrich their NPC- PC interactions through timely feedback, 6. References emotional-triggering elements, and by increasing the unpredictability through unforeseen actions [1] E. Glikson and A. W. Woolley, “Human and plot twists. Trust in Artificial Intelligence: Review of Secondly, the results reveal novel approaches Empirical Research,” Acad. Manag. Ann., to human-AI cooperation and can offer practical vol. 14, no. 2, pp. 627–660, 2020 . guidance for software developers of AI-based [2] R. R. Wehbe, E. Lank, and L. E. Nacke, solutions. For instance, the review identified that “Left them 4 dead: Perception of humans certain aspects of NPC design have already been versus non-player character teammates in implemented in human-robot interaction with cooperative gameplay,” DIS 2017 - Proc. positive outcomes (cf. [31]). Furthermore, the 2017 ACM Conf. Des. Interact. Syst., pp. 100 403–415, 2017. Intell. AI Games, vol. 1, no. 2, pp. 93–104, [3] B. Morschheuser, “The Gamification of 2009. Crowdsourcing Systems: Empirical [15] D. Gamez, Z. Fountas, and A. K. Investigations and Design,” Karlsruher Fidjeland, “A neurally controlled Institut für Technologie (KIT), 2017. computer game avatar with humanlike [4] M. Funk, B. Dieber, H. Pichler, and M. behavior,” IEEE Trans. Comput. Intell. AI Coeckelbergh, “Gamification of Trust in Games, vol. 5, no. 1, pp. 1–14, 2013. HRI?,” Front. Artif. Intell. Appl., vol. 335, [16] L. Hoyet, R. McDonnelly, and C. pp. 632–642, 2020. O’Sullivanz, “Push it real: Perceiving [5] H. Warpefelt and H. Verhagen, “A model causality in virtual interactions,” ACM of non-player character believability,” J. Trans. Graph., vol. 31, no. 4, 2012. Gaming Virtual Worlds, vol. 9, no. 1, pp. [17] Y. H. Chang, R. Maheswaran, T. 39–53, 2017. Levinboim, and V. Rajan, “Learning and [6] K. Rogers, M. Aufheimer, M. Weber, and evaluating human-like NPC behaviors in L. E. Nacke, “Towards the Visual Design dynamic games,” Proc. 7th AAAI Conf. of Non-Player Characters for Narrative Artif. Intell. Interact. Digit. Entertain. Roles,” 2018. AIIDE 2011, pp. 8–13, 2011. [7] I. Mahmoud and D. Wloka, “Planning for [18] C. Bailey and M. Katchabaw, “An non-player characters using HTN and emergent framework for realistic visual perception,” Proc. - EMS 2015 psychosocial behaviour in non player UKSim-AMSS 9th IEEE Eur. Model. characters,” ACM Futur. Play 2008 Int. Symp. Comput. Model. Simul., pp. 321– Acad. Conf. Futur. Game Des. Technol. 327, 2016. Futur. Play Res. Play. Share, pp. 17–24, [8] I. M. Mahmoud, L. Li, D. Wloka, and M. 2008. Z. Ali, “Believable NPCs in serious [19] J. You and M. Katchabaw, “A flexible games: HTN planning approach based on multi-model approach to psychosocial visual perception,” IEEE Conf. Comput. integration in non player characters in Intell. Games, CIG, 2014. modern video games,” Futur. Play 2010 [9] K. Krejtz, A. Duchowski, H. Zhou, S. Jörg, Res. Play. Share - Int. Acad. Conf. Futur. and A. Niedzielska, “Perceptual Game Des. Technol., pp. 17–24, 2010. evaluation of synthetic gaze jitter,” [20] J. Siegel and D. Szafron, “Dialogue Comput. Animat. Virtual Worlds, vol. 29, patterns-A visual language for dynamic no. 6, 2018. dialogue,” J. Vis. Lang. Comput., vol. 20, [10] A. Chubarov and D. Azarnov, “Modeling no. 3, pp. 196–220, 2009. behavior of virtual actors: A limited turing [21] A. T. Abraham and K. McGee, “AI for test for social-emotional intelligence,” dynamic team-mate adaptation in games,” Adv. Intell. Syst. Comput., vol. 636, pp. Proc. 2010 IEEE Conf. Comput. Intell. 34–40, 2018. Games, CIG2010, pp. 419–426, 2010. [11] G. Fĺorez-Puga, M. Ǵomez-Mart́in, B. [22] M. P. Eladhari and M. Mateas, “Semi- D́iaz-Agudo, and P. A. Gonźalez-Calero, autonomous avatars in world of minds : A “Dynamic expansion of behaviour trees,” case study of AI-based game design,” Proc. 4th Artif. Intell. Interact. Digit. Proc. 2008 Int. Conf. Adv. Comput. Entertain. Conf. AIIDE 2008, pp. 36–41, Entertain. Technol. ACE 2008, pp. 201– 2008. 208, 2008. [12] C. Guckelsberger, C. Salge, and J. [23] G. Flórez-Puga, M. A. Gómez-Martín, P. Togelius, “New and Surprising Ways to P. Gómez-Martín, B. Díaz-Agudo, and P. Be Mean,” IEEE Conf. Comput. Intell. A. González-Calero, “Query-enabled Games, CIG, vol. 2018-August, 2018. behavior trees,” IEEE Trans. Comput. [13] Y. Li and D. W. Xu, “A game AI based on Intell. AI Games, vol. 1, no. 4, pp. 298– ID3 algorithm,” Proc. 2016 2nd Int. Conf. 308, 2009. Contemp. Comput. Informatics, IC3I [24] J. Frommel, C. Phillips, and R. L. 2016, pp. 681–687, 2016. Mandryk, “Gathering self-report data in [14] S. Bakkes, P. Spronck, and J. Van Den games through npc dialogues: Efects on Herik, “Rapid and reliable adaptation of data qality, data qantity, player experience, video game ai,” IEEE Trans. Comput. and information intimacy,” Conf. Hum. 101 Factors Comput. Syst. - Proc., 2021. [36] G. Lochmann, L. Reitz, J. Hunz, A. Sohny, [25] E. Bouquet, V. Mäkelä, and A. Schmidt, and G. Schmidt, “Haunted: Intercultural “Exploring the Design of Companions in communication training via information Video Games,” ACM Int. Conf. gaps in a cooperative virtual reality,” Proc. Proceeding Ser., pp. 145–153, 2021. Eur. Conf. Games-based Learn., vol. [26] K. Emmerich, P. Ring, and M. Masuch, 2015-January, pp. 303–312, 2015. “I’m glad you are on my side: How to [37] F. Negini, R. L. Mandryk, and K. G. design compelling game companions,” Stanley, “Using affective state to adapt CHI Play 2018 - Proc. 2018 Annu. Symp. characters, NPCs, and the environment in Comput. Interact. Play, pp. 153–162, a first-person shooter game,” Conf. Proc. - 2018. 2014 IEEE Games, Media, Entertain. [27] J. Webster and R. Watson, “Analyzing the Conf. IEEE GEM 2014, 2015. Past to Prepare for the Future: Writing a [38] M. Lankes and T. Mirlacher, “Affective Literature Review on JSTOR,” MIS Q., game dialogues: Using affect as an explicit vol. 26, no. 2, 2002. input method in game dialogue systems,” [28] S. K. Boell and D. Cecez-Kecmanovic, Lect. Notes Comput. Sci., vol. 7168 LNCS, “On being ‘Systematic’ in Literature pp. 333–341, 2012. Reviews in IS:,” J. Inf. Technol., vol. 30, [39] J. P. Rowe, L. R. Shores, B. W. Mott, and no. 2, pp. 161–173, 2015. J. C. Lester, “Individual differences in [29] J. Thomas and A. Harden, “Methods for gameplay and learning: A narrative- the thematic synthesis of qualitative centered learning perspective,” FDG 2010 research in systematic reviews,” BMC - Proc. 5th Int. Conf. Found. Digit. Games, Med. Res. Methodol., vol. 8, no. 1, pp. 1– pp. 171–178, 2010. 10, 2008. [40] N. E. Bassey and Q. Mehdi, “Learning [30] G. Paré, M. C. Trudel, M. Jaana, and S. agents in board games,” Proc. CGAMES Kitsiou, “Synthesizing information 2009 USA - 14th Int. Conf. Comput. systems knowledge: A typology of Games AI, Animat. Mobile, Interact. literature reviews,” Inf. Manag., vol. 52, Multimedia, Educ. Serious Games, pp. no. 2, pp. 183–199, 2015. 111–121, 2009. [31] S. Ali, H. W. Park, and C. Breazeal, “Can [41] P. Spangenberger, L. Kruse, S. Narciss, Children Emulate a Robotic Non-Player and F. Kapp, “Developing a serious game Character’s Figural Creativity?,” CHI for girls: Design of avatars and non-player Play 2020 - Proc. Annu. Symp. Comput. characters,” Proc. Eur. Conf. Games- Interact. Play, pp. 499–509, 2020. based Learn., vol. 2019-Octob, pp. 657– [32] B. L. Schroeder, N. W. Fraulini, W. L. Van 666, 2019. Buskirk, and C. I. Johnson, “Using a non- [42] M. S. Benlamine, A. Dufresne, M. H. player character to improve training Beauchamp, and C. Frasson, “BARGAIN: outcomes for submarine electronic warfare behavioral affective rule-based games operators,” Lect. Notes Comput. Sci., vol. adaptation interface–towards emotionally 12214 LNCS, pp. 531–542, 2020. intelligent games: application on a virtual [33] B. Cheng and T. C. N. Graham, “Playing reality environment for socio-moral with Persiflage: The Impact of Free-Form development,” User Model. User-adapt. Dialogue on the Play of Computer Role Interact., vol. 31, no. 2, pp. 287–321, Playing Games,” Lect. Notes Comput. Sci., 2021. vol. 11863 LNCS, pp. 187–200, 2019. [43] A. N. Muis, A. S. Prihatmanto, G. R. E. [34] W. Marley and N. Ward, “Tightly coupled Gitarana, and C. Fithratu, “Adaptive agents in live performance metacreations,” Companion-Mediated Behavior Changes C 2015 - Proc. 2015 ACM SIGCHI Conf. on Arithmatopia Games User: Case Study Creat. Cogn., pp. 299–302, 2015. of NPC Design,” 6th Int. Conf. Interact. [35] M. Miranda, A. A. Sánchez-Ruiz, and F. Digit. Media, ICIDM 2020, 2020. Peinado, “Building Non-player Character [44] R. Paradeda, M. J. Ferreira, R. Oliveira, C. Behaviors By Imitation Using Interactive Martinho, and A. Paiva, “The role of Case-Based Reasoning,” Lect. Notes assertiveness in a storytelling game with Comput. Sci., vol. 12311 LNAI, pp. 263– persuasive robotic non-player characters,” 278, 2020. CHI Play 2019 - Proc. Annu. Symp. 102 Comput. Interact. Play, pp. 453–465, and games,” 2016 IEEE Symp. Ser. 2019. Comput. Intell. SSCI 2016, 2017. [45] P. Paliyawan, T. Kusano, Y. Nakagawa, T. [56] R. Kortmann, E. Van Daalen, I. Mayer, Harada, and R. Thawonmas, “Adaptive and G. Bekebrede, “Veerkracht 2.0 motion gaming AI for health promotion,” embodied interactions in a servant- AAAI Spring Symp. - Tech. Rep., vol. SS- leadership game,” Lect. Notes Comput. 17-01-SS-17-08, pp. 720–725, 2017. Sci., vol. 8264 LNCS, pp. 44–51, 2014. [46] T. Plch, M. Marko, P. Ondráček, M. [57] J. Byun and C. S. Loh, “Audial Černý, J. Gemrot, and C. Brom, “An AI engagement: Effects of game sound on System for Large Open Virtual World,” learner engagement in digital game-based Proc. 10th AAAI Conf. Artif. Intell. learning environments,” Comput. Human Interact. Digit. Entertain. AIIDE 2014, pp. Behav., vol. 46, pp. 129–138, 2015. 44–51, 2014. [58] R. Zhu, J. Lin, B. Becerik-Gerber, and N. [47] Y. Zeng, H. Mao, F. Yang, and J. Luo, “An Li, “Influence of architectural visual optimization approach to believable access on emergency wayfinding: A cross- behavior in computer games,” vol. 7607 cultural study in China, United Kingdom LNAI, pp. 81–92, 2013. and United States,” Fire Saf. J., vol. 113, [48] J. Ilgner, R. Kuhlmann, H. Eirund, and M. 2020. Hering-Bertram, “Interacting in 3D virtual [59] Z. Menestrina and A. De Angeli, “End- worlds with Brain Computer Interfaces,” user development for serious games,” New 21st Int. Conf. Cent. Eur. Comput. Graph. Perspect. End-User Dev., pp. 359–383, Vis. Comput. Vision, WSCG 2013 - 2017. Commun. Pap. Proc., pp. 78–87, 2013. [60] D. A. Zachary, W. Zachary, J. Cannon- [49] C. R. Strong, M. Mateas, and D. Bowers, and T. Santarelli, “Backstory Grossman, “Generative conversation tool elaboration: A method for creating for game writers,” FDG 2009 - 4th Int. realistic and individually varied cultural Conf. Found. Digit. Games, Proc., pp. avatars,” Adv. Intell. Syst. Comput., vol. 183–190, 2009. 480, pp. 207–217, 2017. [50] V. H. H. Chen and W. J. D. Koek, [61] A. Christopoulos, M. Conrad, and M. “Understanding Flow, Identification with Shukla, “What Does the Pedagogical Game Characters and Players’ Attitudes,” Agent Say?,” 10th Int. Conf. Information, PervasiveHealth Pervasive Comput. Intell. Syst. Appl. IISA 2019, 2019. Technol. Healthc., 2020. [62] B. Mallon and R. Lynch, “Stimulating [51] J. C. F. Ho and R. Ng, “Perspective- Psychological Attachments in Narrative Taking of Non-Player Characters in Games: Engaging Players With Game Prosocial Virtual Reality Games: Effects Characters,” Simul. Gaming, vol. 45, pp. on Closeness, Empathy, and Game 508–527, 2014. Immersion,” Behav. Inf. Technol., 2020. [63] C. Pacheco, L. Tokarchuk, and D. Pérez- [52] M. Nayyar, Z. Zoloty, C. McFarland, and Liébana, “Studying believability A. R. Wagner, “Exploring the Effect of assessment in racing games,” ACM Int. Explanations During Robot-Guided Conf. Proceeding Ser., 2018. Emergency Evacuation,” Lect. Notes [64] C. T. Yang, B. C. Chen, H. T. Yeh, and G. Comput. Sci. vol. 12483 LNAI, 2020. X. Jian, “A study on smart deployment for [53] Y. Ferstl, E. Kokkinara, and R. real-time strategy games,” Adv. Intell. McDonnell, “Facial features of non-player Syst. Comput., vol. 535, pp. 185–190, creatures can influence moral decisions in 2017. video games,” ACM Trans. Appl. Percept., [65] C. T. Yang, H. T. Yeh, B. C. Chen, and G. vol. 15, no. 1, 2017. X. Jian, “Automatic tunable deployment [54] N. Zheng et al., “Hybrid-augmented for real-time strategy games,” Eng. intelligence: collaboration and cognition,” Comput. (Swansea, Wales), vol. 34, no. 2, Front. Inf. Technol. Electron. Eng. 2017 pp. 239–250, 2017. 182, vol. 18, no. 2, pp. 153–179, 2017. [66] T. Trescak and A. Bogdanovych, [55] X. Liu, K. Merrick, and H. Abbass, “Simulating complex social behaviours of “Designing artificial agents to detect the virtual agents through case-based motive profile of users in virtual worlds planning,” Comput. Graph., vol. 77, pp. 103 122–139, 2018. 2007. [67] R. Lovreglio et al., “Prototyping virtual [78] A. S. Ruela and F. G. Guimarães, reality serious games for building “Procedural generation of non-player earthquake preparedness: The Auckland characters in massively multiplayer online City Hospital case study,” Adv. Eng. strategy games,” Soft Comput., vol. 21, no. Informatics, vol. 38, pp. 670–682, 2018. 23, pp. 7005–7020, 2017. [68] J. Al-Gharaibeh and C. Jeffery, “PNQ: [79] W. Huang et al., “Verifying adaptation of Portable non-player characters with neuro-controlled game opponent by cross quests,” Proc. - 2010 Int. Conf. validation under supervised and Cyberworlds, CW 2010, pp. 294–301, unsupervised player modeling,” 2nd Int. 2010. Conf. Softw. Eng. Data Mining, SEDM [69] M. G. Christel et al., “Lessons learned 2010, pp. 172–177, 2010. from testing a children’s educational game [80] T. Selmbacherova, V. Sisler, and C. Brom, through web deployment,” SeriousGames “The impact of visual realism on the 2014 - Proc. 2014 ACM Int. Work. Serious authenticity of educational simulation: A Games, Work. MM 2014, pp. 45–50, 2014. comparative study,” Proc. Eur. Conf. [70] D. Vrajitoru, “NPCs and chatterbots with Games-based Learn., vol. 2, pp. 520–528, personality and emotional response,” 2014. Proc. 2006 IEEE Symp. Comput. Intell. [81] R. Edmundson, R. Danby, K. Brotherton, Games, CIG’06, pp. 142–147, 2006. E. Livingstone, and L. Allcock, [71] E. L. C. Law, D. E. Watkins, J. P. L. “Investigating combinations of machine Barwick, and E. S. Kirk, “An experiential learning and classification techniques in a approach to the design and evaluation of a game environment,” 2016 12th Int. Conf. gamified research tool for Law in Nat. Comput. Fuzzy Syst. Knowl. Discov. Children’s Lives,” Proc. IDC 2016 - 15th ICNC-FSKD 2016, pp. 1306–1311, 2016. Int. Conf. Interact. Des. Child., pp. 322– [82] N. Beume et al., “Measuring flow as 333, 2016. concept for detecting game fun in the pac- [72] Y. Hou, L. Feng, and Y. S. Ong, “Creating man game,” 2008 IEEE Congr. Evol. human-like non-player game characters Comput. CEC 2008, pp. 3448–3455, 2008. using a Memetic Multi-Agent System,” [83] S. P. Ting, S. Zhou, and N. Hu, “A Proc. Int. Jt. Conf. Neural Networks, vol. computational model of situation 2016-Octob, pp. 177–184, 2016. awareness for MOUT simulations,” Proc. [73] Y. She and P. Grogono, “An approach of - 2010 Int. Conf. Cyberworlds, CW 2010, real-time team behavior control in games,” pp. 142–149, 2010. Proc. - Int. Conf. Tools with Artif. Intell. [84] S. P. Ting, S. Zhou, and N. Hu, ICTAI, pp. 546–550, 2009. “Generating situation awareness for time [74] C. A. Cruz and J. A. R. Uresti, “HRLB^2: critical decision making,” Lect. Notes A reinforcement learning based Comput. Sci., vol. 6670 LNCS, pp. 183– framework for believable bots,” Appl. Sci., 205, 2011. vol. 8, no. 12, 2018. [85] W. C. Ho, K. Dautenhahn, and C. L. [75] H. Zhang, X. Luo, C. Miao, Z. Shen, and Nehaniv, “A study of episodic memory- J. You, “Adaptive goal selection for agents based learning and narrative structure for in dynamic environments,” Knowl. Inf. autobiographic agents,” Proc. AISB’06 Syst., vol. 37, no. 3, pp. 665–692, 2013. Adapt. Artif. Biol. Syst., vol. 3, pp. 26–29, [76] T. P. Hartley and Q. H. Mehdi, “In-game 2006. tactic adaptation for interactive computer [86] L. Rodrigues et al., “Personalization games,” Proc. CGAMES’2011 USA - 16th Improves Gamification,” Proc. ACM Int. Conf. Comput. Games AI, Animat. Human-Computer Interact., vol. 5, no. Mobile, Interact. Multimedia, Educ. CHIPLAY, 2021. Serious Games, pp. 41–49, 2011. [87] J. Koivisto and J. Hamari, “The rise of [77] K. E. Merrick and M. Lou Maher, motivational information systems: A “Motivated reinforcement learning for review of gamification research,” Int. J. adaptive characters in open-ended Inf. Manage., vol. 45, pp. 191–210, 2019. simulation games,” ACM Int. Conf. Proceeding Ser., vol. 203, pp. 127–134, 104