Applying Affective Design Patterns in VR Firefighter Training Simulator

Abstract

We present a prototype of virtual reality training simulator for firefighters. Our approach is based on the concept of Affective Patterns in Serious Games. One of the most serious problems when it comes to training firefighters is to maintain the right level of their commitment. The idea to solve the problem of repetitive and monotonous exercises is to combine them with those implemented in VR. While creating the solution for optimizing a psychological background of knowledge acquisition in training, we used concepts from the Motivational Intensity Theory.

Keywords

• Design Patterns Affective Computing Virtual Reality VR Training VR Simulations

-

Firefighter training cannot be based solely on the traditional knowledge transfer model. There is an obvious need for training focused on gaining direct experience. Practice can be divided into three types: a) equipment skills training; b) responding to situation training (adherence to procedures); c) ability training (readiness to intervene, stress resistance, response to fast changing situations). In practice, it is much easier to train the first type of skill than the second and third. Virtual reality can combine various aspects of training, with particular emphasis on training procedures and abilities.

In this paper our aim is to present example on how the Affective Patterns in Serious Games framework [ 1 ], [ 2 ] allows, in coaction with Experiential Learning Theory, to create advanced VR training environments for developing not only operating skills but also abilities and motivational enhancement. 2

Virtual Reality and the Experiential Learning Theory

Experiential Learning Theory was proposed by psychologist David Kolb [ 3 ] influenced, among others, by Kurt Lewin and Jean Piaget. Kolb believed that knowledge is created by transformation of experience rather than peer to peer information transfer (e.g. lecture). The theory is not rooted in plain behaviorism nor simple cognitivism. It emphasizes role of environment and emotions - both factors crucial in virtual reality and proposed framework.

For Kolb two ways of grasping experience and then - later - transforming it are: concrete experience, abstract conceptualization and reflective observation along with active experimentation.

Another important observation by Kolb is that learning is best to be viewed as process, thus all learning is re-learning. This remark is also very important for VR serious games design because the technology allows for multiple repetition of the same scenarios with slight modifications - so that they are similar but not identical.

Virtual reality allows for creating various, immersive environments for active participation in training. This is also imperative for Experiential Learning Theory approach - Lewin’s famous formula for explanation of one’s behavior is:

B = f (P; E) where B is the behavior, P is person and E stands for environment. It means that: "A physically identical environment can be psychologically different even for the same man in different conditions" ([ 4 ], pp. 24-25). That statement translates very well to the reality of VR: "The psychological reality [...] does not depend upon whether or not the content [...] exists in a physical or social sense [...]. The existence or nonexistence [...] of a psychological fact are independent of the existence or nonexistence to which its content refers." ([ 4 ], p. 38).

There are many theoretical frameworks taking from before-mentioned concepts of situated learning. Urie Bronfrenbrenner’s work on the ecology of human development [ 5 ], Situated Learning Theory by Lave and Wenger [ 6 ], theory of knowledge creation by Nonaka and Konno [ 7 ] are worth mentioning here.

In presented case we strongly rely on the idea that environmental actions are among most important learning factors and that the virtual (mental) environments are essentially no different in this aspect than the real-life situations. 3

Motivational Intensity Theory and Affective

Computing

One of the most serious problems when it comes to training firefighters is to maintain the right level of their commitment. Generally, because of the nature of their work, firefighters are highly motivated. However, the fact that participation in real actions is associated with states of high emotional stimulation causes that training may appear monotonous despite awareness of its significance.

The idea to solve the problem of repetitive (in the negative sense) and monotonous exercises performed at the training field is to combine them with those implemented in a virtual environment – providing wealth of scenarios and surprising, and thus interesting situations.

While creating the solution for optimizing a psychological background of knowledge acquisition in training, we used concepts from the Motivational Intensity Theory (MIT). As Richter, Gendolla and Wright state "motivation science is concerned with the processes and mechanisms underlying the initiation, direction, persistence, and intensity of behavior" [ 8 ]. In this context MIT is focused around mobilization effort in the pursuit of the goal. It concerns i.a. level of energization related to presented task (and it’s valence). The most important thing is that one can use cardiovascular measures as indicators of effort mobilization [ 9 ], [ 10 ], [ 11 ], [ 12 ], [ 13 ] and activation of sympathetic and parasympathetic nervous system.

Software recognition of user’s mental states using physiological measurements in is one of the domains of Affective Computing. This paradigm takes as a starting point statement that: "emotions are both physical and cognitive" [19]. This means that to some extent the analysis of psychological states can be carried out by means of physiological data processing methods. The question arises - how to create such computing systems and how to use them efficiently in the training situations? Proposed answer involves applying affective patterns in designing serious virtual reality games. Multiple conducted by our research team studies present that it is possible to interpret some physiological indicators in relation to the course of gameplay as signs of arousal, stress, frustration – or more generally speaking – involvement [ 14 ], [ 15 ], [ 16 ], [17], [18]. 4

Affective patterns in serious games design

The idea of Affective Patterns in Serious Games design is based on the application of model developed by S. Björk and J. Holopainen in their book "Patterns in Game Design" [ 1 ] in connection with affective measurements to evidencecentered assessment design (ECD) [ 2 ], [20].

The model consists of a part focusing on the use of ECD (Evidence Model - Assembly Template for ECD) and game design methods (DPE framework Design, Play, Experience). The key part in designing the player’s experience is the use of gameplay-building mechanics. Mechanics [21] are constructed from general patterns, which additionally contain a description of the physiological correlates of emotions (and thus commitment) and evidences representing the possession of specific skills, knowledge and abilities. When particular pattern is applied in VR simulation it introduces specific mechanic - if the mechanic is applied by the player, the system is able to evaluate correctness of conducted action.

The problem faced by the designer of specific simulations concerns how to construct the Evidence Model (which activities require particular skills?) and what sets of stimuli should be prepared that will cause the appropriate reaction of the user.

In order to prepare an appropriate simulation based on the solution described above, we conducted an analysis of the content of the firefighters training program, and a survey regarding stimuli - stressful situations. 5

Creating Affective Patterns Based on Surveys

Conducted on Firefighters

Simpro sp. z o.o., the creators of VR simulator - the company we work with - conducted a survey aimed at predicting the stressful factors for firefighters. Each firefighter could indicate no more than ten stressful factors and assess their negative impact on a scale of one to five. 104 firefighters participated in the survey and 103 responses was collected. One man replied that he did not know any stressful situations. Next, only 93 responses had correctly assigned weights.

Stressful factor

Some responses have large dispersion of ratings. It means that either the firefighters did not agree on the assessments or the stressful factors can have different degrees of severity. The most often indicated factors are: children (present on the accident place), large number of injured persons, fire, noise, third parties and onlookers. This answers served as a basis for creating patterns of affective situations in the simulator.

Applying Affective Patterns in VR simulator

General design of VR simulator

Presented simulator is a project developed by Simpro4 sp. z o.o. (spinout company) and Nano Games5 sp. z o.o. (parent company). 4 See https://simprosoft.com/en 5 See https://nano-games.com

The project involves creating a VR simulation of a rescue operation with implemented affective feedback loop. Having a VR multiplayer solution and a wireless sensor set based on the Bitalino6 platform it is possible to apply the concept of affective patterns in practice. The affective factor is based on the usage of ECG (electrocardiography) and EDA (electrodermal activity) sensors. In the case of the former, the HRV (hear rate variability) value is calculated.

For the purposes of the simulator evaluation, two basic contexts were created:

1. a car accident at an intersection in a small city,

2. an incident when transporting passengers to the aircraft at the airport.

In the first case, users train the skills of helping victims at the scene of the accident, in the second, the triage procedure is practiced.

Based on surveys and analysis of the firefighters training program, prototype scenarios were prepared along with implementation in VR. Sets of patterns matching the given contexts have been developed.

Patterns consist of:

1. Number/Codename - arbitral; e.g. "C-1";

6 See https://bitalino.com/en/community/publications 2. Type - criterion was the existence of the procedure: patterns that have an affective impact but do not significantly affect the course of the rescue procedures are classified separately; separately, those that require a rescuer to adhere to another procedure; 3. Name of the affective agent - e.g. "aggressive injured person"; 4. Weight - according to analyzed surveys (1 to 5); 5. Design description - e.g. "injured", "aware", "aggressive towards the lifeguard"; "gradual - aggression can be verbal or physical"; "in the absence of intervention, a person can worsen his condition"; 6. Physiological description - e.g. "X increase in HR"; "Y decrease in HRV"; "Z increase in GSR"; 7. Pattern activation - a pattern programmed into a simulation that is permanently activated or activates as a result of certain conditions (e.g. specific reading from physiological sensors); 38 patterns that do not change the procedure; 12 affecting the procedure and 13 changing the procedure depending on the rescuer decisions were created for the prototype. 6.2

Example scene - short description

An example scenario can be described as follows:

– Location: Airport; – Context: Bus overturned with engine thrust; – Weather: Dense fog; On the scene, there are 6 victims described according to the formula: There are 8 onlookers on the stage described according to the formula: – ID: 1105; – Gender: Female; – Age: Adult; – Pattern: C-34; – Trigger: Physiological reading from sensor;

Patterns mentioned (short description):

– C-23: after the beginning of the action one of the victims dies; – C-34: one or more people behave intensely towards a rescuer: follow him, comment on his actions, claim that know how the rescuer should act, question him;

Summary and Future Research

The paper presents a prototype of a adaptable VR simulator for training firefighters. This application is a practical application of the concept of Affective Patterns in Serious Games [ 2 ]. Currently, the software is undergoing testing and adjustment of algorithms responsible for the interpretation of the results of physiological readings. Ultimately, the alpha version will implement 2 full contexts with 40 validated patterns.

For the future versions, we are working on the algorithmic description of patterns, as well as automation of the generation of the components of the simulation, e.g. victim description. Furthermore, control in VR is an important challenge. An important improvement would be the use of eye-tracking which could be available in the next versions of VR headsets. 17. Grabska-Gradzińska, I., Nowak, L., Argasiński, J.: Applying Oculography and

Biosignals Measurements to Gameplay Modes Analysis, (2020) 18. Grabska-Gradzińska, I., Argasiński, J.: Patterns in Video Games Analysis – Application of Eye-Tracker and Electrodermal Activity (EDA) Sensor, (2018) 19. Picard, R.: Affective Computing. MIT Press. Cambridge, MA, USA (1998) 20. Mislevy, R., Rusell, G. A., Lukas, J. F.: A Brief Introduction to Evidence-centered Design. Research Report, Research and Development Division Princeton Univ.

Princeton, NJ, USA (2003) 21. Sicart, M.: Defining Games Mechanics. Game Studies. 8(2), (2008)

References

1. Björk , S. , Holopainen , J.: Patterns in Game Design . Charles River Media, Rockland, MA, USA ( 2004 )

2. Argasiński , J. K. , Węgrzyn , P. ,: Affective Patterns in Serious Games. Future Generation Computer Systems 92 , 526 - 538 ( 2019 )

3. Kolb , D. : Experiential Learning: Experience As The Source Of Learning And Development. 2nd Edition . Pearson FT Press, Upper Saddle River, NJ, USA ( 2015 )

4. Lewin , K. : Principles of Topological Psychology . McGraw-Hill , New York, USA ( 1936 ), pp. 24 - 25

5. Bronfrenbrenner , U. : The Ecology of Human Development . Harvard Univ. Press, Cambridge, MA, USA ( 2079 )

6. Lave , J. , Wenger , E.: Situated Learning: Legitimate Peripheral Participation . Cambridge Univ. Press, Cambridge, UK ( 1991 )

7. Nonaka , I. , Konno , N. : The concept of “ba”: Building a foundation for knowledge creation . California Management Review 30 ( 3 ), 40 - 54 ( 1998 )

8. Richter , M. , Gendolla , G. , Wright , R.: Three decades of research on motivational intensity theory: What we have learned about effort and what we still don't know . Advances in Motivation Science 12 ( 3 ), ( 2016 )

9. Wright , R.A. : Brehm's theory of motivation as a model of effort and cardiovascular response . In: Gollwitzer, P. M. , Bargh , J. A . (Eds.): he psychology of action: Linking cogni- tion and motivation to behavior , Guilford, New York (2996)

10. Obrist , R. A. : Cardiovascular psychophysiology: A perspective . Plenum Press, New York ( 1981 )

11. Newlin , D. B. , Levenson , R. W. Pre-ejection period: measuring beta-adrenergic influences upon the heart . Psychophysiology , 16 ( 1979 )

12. Richter , M. , Friedrich , A. , Gendolla , G. H. E. : Task difficulty effects on cardiac activity . Psychophysiology , 45 ( 2008 )

13. Wright , R. A. , Killebrew , K. , Pimpalapure , D. : Cardiovascular incentive effects where a challenge is unfixed: demonstrations involving social evaluation, evaluator status, and monetary reward . Psychophysiology , 39 ( 2002 )

14. Nalepa , G. , Giżycka , B. , Kutt , K. , Argasiński , J.: Affective Design Patterns in Computer Games. Scrollrunner Case Study. 345-352 , ( 2017 )

15. Nalepa , G. , Argasiński , J. , Kutt , K. , Wegrzyn , P. , Bobek , Sz., Łępicki , M. Z. : Affective Computing Experiments in Virtual Reality with Wearable Sensors. Methodological considerations and preliminary results , ( 2016 )

16. Argasiński , J. , Grabska-Gradzińska , I. : Patterns in Serious Game Design and Evaluation Application of Eye-Tracker and Biosensors, ( 2017 )