Affective VR serious game for firefighter training Jan K. Argasiński1 , Paweł Węgrzyn1 , and Paweł Strojny2,3 1 Department of Games Technology, Faculty of Physics, Astronomy and Applied Computer Science, Jagiellonian University in Krakow, Poland 2 Institute of Applied Psychology, Faculty of Management and Social Communication, Jagiellonian University in Krakow, Poland 3 Nano Games sp. z o.o. Abstract. The article discuss the use of design patterns for serious games with affective feedback in order to uphold immersion into Virtual Reality, and to influence motivation and effort put into training. The ex- ample of training simulation in Virtual Reality addressed to firefighters is described. We consider two different conceptual and theoretical frame- works, perhaps even two contradictory concepts, namely Motivational Intensity Theory and Embodied Cognitive Science. Virtual Reality game prototype created with the participation of firefighters by Nano Games sp. z o.o. is presented. Keywords: Affective Computing, Virtual Reality, Serious Games, Pat- terns in Game Design 1 Motivation 1.1 Affective feedback in firefighter VR training Firefighters represent specific occupational group of highly motivated specialists, whose daily duty consists in protecting human property and life. What follows, it would be difficult to imagine any kind of an artificial training that would per- manently increase their motivation, there may be simply no chance for such kind of influence. There is a hidden trap in the craft of screen-writing and designing for VR firefighter simulators. Firefighters must be ready for various situations, including hard to predict extremely rare events (black swans). This is the reason why their job consists of numerous hours of skill training. And here comes the trap, they may be highly motivated but their engagement in training may be significantly reduced because they usually struggle with more serious challenges on a daily basis. 1.2 The research problem Serious games and simulations intended for firefighters training appear to focus predominantly on either target-based practices with some professional equipment or on the logistics of rescue operations. Many training systems offer the possi- bility of including modules that could allow to simulate high stress situations. Our literature review [1,2,3,4,5,6,7,8,9] shows that there are no ready solutions for developing in a systematic way complex affective scenarios and designing simulation games based on controlling emotional feedback. The proposal here is to include the paradigm of affective computing in the area of simulations and training games. The idea is to adapt design patterns in game design [12,19]. The main research problem is to formulate appropriate theoretical framework allowing to capture the problem of engagement, motiva- tion and impact of affects in a manner applicable in VR simulations and serious games through patterns. 2 Materials & Methods The Design Patterns (here: patterns in game design) are reusable solutions that should evade low-level details of large software applications and allow us to express their architecture and complexity and discuss relations, processes and other issues at a higher level of abstraction. Our aim is to attain right design patterns to make the firefighter VR training highly immersive. It refers to the motivation of the trainee and his effort put into training. We are aware of two different theoretical frameworks of theories and concepts which we can apply to the problem. In this paper, we are submitting to Motivational Intensity Theory (MIT) and Embodied Cognitive Science (ECS). In both of these cases, our task is to define design patterns that make the conceptual framework and theoretical background apparent in our context. 2.1 Motivational Intensity Theory Effort may be defined as an investment of resources enabling execution of be- havior. Terms “motivation” and “effort” should not be confused. Psychologists define motivation as a complex cognitive structure. According to one of the most comprehensive theory of motivation, Motivation Intensity Theory [15], motiva- tion should be understood in terms of individual cognitions regarding need and possibility of achieving a goal. In other words, one’s motivation (i.e. potential motivation) depends on three antecedents: need, incentive value and probability that desired outcomes will be attained once instrumental behavior will be car- ried out [16]. Motivation may be extremely high but remain hidden in terms of effort manifesting itself in observable behavior. It is not a surprise that maxi- mum effort is limited by (potential) motivation. What should be stressed is the fact that no motivation but the task’s requirements directly determine the effort level actually invested in the task – there is simply no need of investing resources exceeding the level required by a task. The research problem concerns the an- swer to the question how to design a training to encourage trainees to high effort without involving means absent in natural environment (e.g. exam stress). There are three major categories of effort indicators used in psychological re- search: self-report based effort indices (e.g. NASA Task Load Index), behavioral indicators (e.g. perseveration) and physiological correlates (e.g. Pre-Ejection Pe- riod, PEP). All of them may be used separately but since each of those indicators has flaws, combining more than one may provide deeper and more reliable re- sults. It is the reason why the proposed affective system is going to use two relatively independent physiological indicators of effort [16]: Heart Rate Vari- ability (HRV) and Electrodermal Activity (EDA), and additionally self-report data that are going to be used as a source of supplementary data in the initial stages. These two are considered as indicators of auto- nomic nervous system (ANS) activation. ANS func- tions mostly without the consciousness of the per- son and regulates the most important activities of the body, such as heart rate, blood supply and di- gestion [17]. Its role is to regulate the functions im- portant for the survival of the organism, depend- ing on circumstances one of its two branches pre- vail (sympathetic and parasympathetic, some aca- demics indicate the third branch – enteric). Both branches innervate the same organs but (except spe- cific situations such as sexual arousal) act in the opposite way – sympathetic activation is related to quick mobilization of the system while parasympa- thetic activation is related to dampening which is also less dynamic. Electrodermal Activity (EDA) is considered as one of reliable indicators of sympa- Fig. 1. Biosensors & VR thetic activation while high frequency Heart-Rate simulation by Nano Games Variability (nHF-HRV) indicates parasympathetic branch activation. Combining these two indicators will make continuous assess- ment of subject’s affective state plausible which leads to possibility of modulating this state (e.g. effort) with affective computing techniques (test setup on Fig.1). Motivation intensity theory [15,16], described briefly above, may be utilized as a conceptual framework and theoretical background for the tool. Taking this point of view, we assume that a firefighter during real rescue action acts with high level of motivation which leads to potentially extreme effort moderated only by task requirements. In other words one would expect that the firefighter would do his or her best if task is extremely difficult because his or her motivation is high due to the importance of the task. On the other hand, proximal consequences of traditional training tasks are far less profound, it may result in significantly lower motivation which may lead to faster giving up in the face of difficulties which, in turn, may result in poor preparation for the most difficult tasks. We can believe that by combining highly immersive and realistic environments with adaptive difficulty level based on affective computing the training system that can encourage increased effort could be developed. 2.2 Embodied Cognitive Science Embodied Cognitive Science is an alternative theory to cognition in which it denies that the human mind or brain is an information input/output processing system and that thinking is a form of computing. The ECS theory considers the mind and body as a single entity, and an entire organism’s body determines how and what a human thinks. Following this point of view, emotions are also embod- ied. According to Prinz [11], emotions are perceptions (conscious or unconscious) of patterned changes in the body. The ECS framework is the most popular within the field of Affective Com- puting. In her paradigm establishing book Affective Computing, R. Picard states that emotions are both physical and cognitive [10]. This means that if we want to collect and process data on affects, we can rely on information obtained from the body of the surveyed subjects. We do not have to rely solely on the psychological aspects. According to old James and Lange theories, emotions are primarily (if not exclusively) reactions to the changes in physiology under the influence of in- ternal or external stimuli [11]. They can be either conscious or not. In the case of embodied emotions, the direction of the process proceeds from the physiological reaction to a possible psychological response. The only problem is the choice of biophysical information acquisition channels (sensors) and the question of the minimum necessary accuracy of the measurement. The term "affect" usually takes on a more complex meaning. Damasio [18] de- fines it as the collection of processes that includes drives, motivations, emotions, and feelings. Feelings (that are conscious and valenced here) are motivators, monitors and negotiators of the cultural process. Affects are powered by homeo- statis, that is a regulator of human organism processes such that human life can benefits. Therefore, feelings can be also interpreted as the mental expression of ongoing homeostatic states in a human organism. Emotive responses to sensory stimuli (emotional feelings) are some action programs superposed on ongoing physiological regulations provoked by external conditions. Feelings are natural reports on the momentary state of life within the human organism. They open a way to learn about all processes within the organism. Effective acquisition and processing of information on physiological condi- tions (implying emotional states) allows for their use in order to create a spe- cific kind of feedback. Detection of emotions (or feelings) can allow for dynamic strengthening or weakening of further affects. In the experience of affective loop, as Höök claims: "(i) emotions are seen as processes, constructed in the inter- action, starting from everyday bodily, cognitive or social experiences; (ii) the system responds in ways that pull the user into the interaction, touching upon end users’ physical experiences; and (iii) throughout the interaction the user is an active, meaning-making individual choosing how to express themselves — the interpretation responsibility does not lie with the system" [14]. Both computer games and simulations based on mechanics (in sense of [13]) seem to be on the one hand suitable systems allowing for increasing motivation (and thus affecting the effort), on the other hand - it is very easy to implement the mechanisms of affective feedback in them. One of the approaches to the problem of designing games that allow for stable and reproducible results in the field of gameplay is the use of a special type of design patterns for creating games (see: [12]). In our approach [19], we suggest using the modified proposal of Björk & Holopainen, enriched, among other things, with emotional correlates. 3 Current state of research and development 3.1 Game prototype The Nano Games sp. z o.o. company has created a prototype of the game intended for training fire- fighters. The game allows for simulation various variants of traffic accidents with different types of victims. Thanks to Virtual Reality technology fire- fighters wearing protective suits and equipped with real gear (see Fig. 2) can practice procedures associ- Fig. 2. VR simulator by ated with providing appropriate help at the accident Nano Games in testing site. Initial research is to check the real usefulness of VR simulations in the training and evaluation of firefighters. Different ways of collecting physiological data for the development of the affective feedback mechanisms and patterns are being tested and evaluated. The game is created in close cooperation with local fire training centers and is tested with the participation of real firefighting units. 3.2 Affective Patterns in Serious Games Based on the above premises and preliminary results, the new gameplay archi- tecture is under development for further testing and trial implementation. It is based on the concept of design patterns in game design. One of the basic elements of the game are mechanics - it is through them, beside the narrative and aesthetic design, that the creator has a direct impact on the player’s experience. They are also the most special element of designing game software. The rules of a game are translated into mechanics. They can be expressed using verbs that describe what an agent can do and what impacts it is subjected to. For instance, the player can "collect" the object, "open" the door, "solve" the problem. Mechanics are repetitive and their character is strongly associated with game genre special features. In their work, Björk and Holopainen [12] proposed a semi-formal description methods for patterns in game design. Our proposal is to identify some patterns as affective (e.g., "time pressure" as a stress inducing factor) and predict their impact. 3.3 Measurement of body signals For preliminary tests BIOPAC Bionomadix system of wearable wireless devices has been used with AcqKnowledge software. It includes ECG, EDA and EMG sensors. Due to the fact that users must be burdened with devices as little as possible - for further works on the prototype of the affective simulation game in VR it is planned to use the Bitalino platform (equipped with the ECG, EDA and EMG sensors). In this case, main advantages are: extremely compact size, modularity, numerous APIs and low price. 4 Future work In the near future, the following tasks are scheduled: – confront two conceptual and theoretical frameworks MIT and ECS in the light of suitability for the needs of designing VR firefighter simulators, – prepare a design of architecture for appropriate solutions (rule systems and context aware systems for MIT and affective design patterns for feedback loops in ECS), – create hardware and software to measure and visualize biosignals based on Bitalino, – prepare advanced affective game design patterns that allow to control en- gagement/effort and some selected emotional states, – build an affective plug-in that will automatically and in real time respond to changes in biophysiological states (affective feedback loop), – build a tool for level designers and firefighter trainers to create new environ- ments and scenarios, – conduct numerous tests with real firefighters units. References 1. L. Jaikyung, Ch. Moohyun, Ch. Byungil, K. Taesung. A Team-based Firefighter Training Platform Using the Virtual Environment. in: Proceedings - VRCAI, 2010 2. D.L. Tate, L. Sibert, T. King. Virtual Environments for Shipboard Firefighting Training. in: Virtual Reality Annual International Symposium, 1997 3. P. Backlund, H. Engstrom, C. Hammar, M. Johannesson, M. Lebram. Sidh – a Game Based Firefighter Training Simulation. in: Information Visualization, 2007 4. J. Bliss, P.D. Tidwell, M.A. Guest. The Effectiveness of Virtual Reality for Ad- ministering Spatial NAvigation Training to Firefighters. in: Human Factors and Ergonomics Society Annual Meeting Proceedings 6(1):73-86, 1997 5. Ch. Moohyun, H. Soonhung, L. Jaikyung, Ch. Byungil. A virtual reality based fire training simulator integrated with fire dynamics data. in: Fire Safety Journal, Vol. 50, 2012 6. J.M. Houtkamp, A. Toet, , F.A. Bos. Task-Relevant Sound and User Experience in Computer-Mediated Firefighter Training. in: Simulation & Gaming, Vol 43, Issue 6, 2012 7. N. Keren, Sh.T. Bayouth, W.D. Franke, K.M. Godby. Examining the effect of level of stress on firefighters’ time-to-decision in virtual reality. in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2013 8. D.L. Tate, L. Sibert, T. King. Using Virtual Environments to Train Firefighters. in: IEEE Computer Graphics and Applications, 1997 9. Z. Xu, Z. Lu, H. Guan, C. Chen, A.Z. Ren. A virtual reality based fire training simulator with smoke hazard assessment capacity. in: Advances in Engineering Software, Vol. 68, 2014 10. R. Picard. Affective Computing. MIT Press, 1997 11. J. Prinz. Gut Reactions. A Perceptual Theory of Emotion. Oxford University Press, 2006 12. S. Björk, J. Holopainen. Patterns in Game Design. Charles River Media, 2005 13. M. Sicart. Defining Game Mechanics in: Game Studies 8, 2008 14. K. Höök. Affective loop experiences: designing for interactional embodiment. in: Philos. Trans. R. Soc. Lond. B. Biol. Sci. 364(1535): 3585-3595, 2009 15. J.W. Brehm, E.A. Self. The Intensity of Motivation. in: Ann. Rev. Psychol, 40:109– 31, 1989. 16. R.A. Wright, L.D. Kirby. Effort determination of cardiovascular response: An integrative analysis with applications in social psychology. in: Adv. in Exp. Soc. Psych., 33:255-307, 2001. 17. R.F. Schmidt, G. Thews (eds.). Human Physiology. Springer, 1989. 18. A. Damasio. The Strange Order of Things: Life, Feeling, and the Making of Cul- tures. Pantheon, 2018. 19. J.K. Argasiński, P. Węgrzyn. Affective Patterns in Serious Games. to be published, 2018. Project "Widespread Disaster Simulator - research and preparation for im- plementation", realized by Nano Games is co-financed under the Smart Growth Operational Programme, sub-measure 1.1.1. Industrial research and develop- ment work implemented by enterprises.