=Paper=
{{Paper
|id=Vol-2609/AfCAI2019_paper_7
|storemode=property
|title=Applying Affective Design Patterns in VR Firefighter Training Simulator
|pdfUrl=https://ceur-ws.org/Vol-2609/AfCAI2019_paper_7.pdf
|volume=Vol-2609
|authors=Jan K. Argasinski, Grzegorz J. Nalepa, Pawel Strojny, Pawel Wegrzyn
|dblpUrl=https://dblp.org/rec/conf/afcai/ArgasinskiNSW19
}}
==Applying Affective Design Patterns in VR Firefighter Training Simulator==
https://ceur-ws.org/Vol-2609/AfCAI2019_paper_7.pdf
Applying Affective Design Patterns in VR
Firefighter Training Simulator
Jan K. Argasiński1[0000−0002−2992−718X] , Grzegorz J.
Nalepa1,2[0000−0002−8182−4225] , Paweł Strojny1,3[0000−0002−6016−044X] , and
Paweł Węgrzyn1[0000−0001−5616−0474]
1
Jagiellonian University in Krakow, Poland
{jan.argasinski, grzegorz.j.nalepa, p.strojny, pawel.wegrzyn}@uj.edu.pl
2
AGH University of Science and Technology in Krakow, Poland
gjn@agh.edu.pl
3
R&D Unit, Nano Games sp. z o.o., Kraków, Poland & Institute of Applied
Psychology, Faculty of Management and Social Communication, Jagiellonian
University, Kraków, Poland
Abstract. We present a prototype of virtual reality training simulator
for firefighters. Our approach is based on the concept of Affective Pat-
terns in Serious Games. One of the most serious problems when it comes
to training firefighters is to maintain the right level of their commitment.
The idea to solve the problem of repetitive and monotonous exercises is
to combine them with those implemented in VR. While creating the solu-
tion for optimizing a psychological background of knowledge acquisition
in training, we used concepts from the Motivational Intensity Theory.
Keywords: Design Patterns · Affective Computing · Serious Games ·
Virtual Reality · VR Training · VR Simulations.
1 Characteristics of Firefighting Training
Firefighters training is one of the areas particularly suitable for implementing
exercises in virtual reality. This is because the job combines a great deal of
versatility (firefighting, extracting victims out of vehicles, chemical protection,
providing assistance to the victims of the accidents, responding to natural disas-
ters and terrorists attacks, etc.) with common presence - in Poland (population
of ca. 38 million) there is almost 700.000 volunteers (OSP - Ochotnicza Straż
Pożarna) of which over 200.000 are able to directly participate in fire fighting
operations. In addition there are about 30.000 professional firefighters (PSP -
Państwowa Straż Pożarna). This creates an immense training demand.
Copyright c 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0).
Firefighter training cannot be based solely on the traditional knowledge trans-
fer model. There is an obvious need for training focused on gaining direct expe-
rience. Practice can be divided into three types: a) equipment skills training; b)
responding to situation training (adherence to procedures); c) ability training
(readiness to intervene, stress resistance, response to fast changing situations).
In practice, it is much easier to train the first type of skill than the second and
third. Virtual reality can combine various aspects of training, with particular
emphasis on training procedures and abilities.
In this paper our aim is to present example on how the Affective Patterns in
Serious Games framework [1], [2] allows, in coaction with Experiential Learning
Theory, to create advanced VR training environments for developing not only
operating skills but also abilities and motivational enhancement.
2 Virtual Reality and the Experiential Learning Theory
Experiential Learning Theory was proposed by psychologist David Kolb [3] in-
fluenced, among others, by Kurt Lewin and Jean Piaget. Kolb believed that
knowledge is created by transformation of experience rather than peer to peer
information transfer (e.g. lecture). The theory is not rooted in plain behaviorism
nor simple cognitivism. It emphasizes role of environment and emotions - both
factors crucial in virtual reality and proposed framework.
For Kolb two ways of grasping experience and then - later - transforming it
are: concrete experience, abstract conceptualization and reflective observation
along with active experimentation.
Fig. 1. Experiential learning cycle (after [3])
Another important observation by Kolb is that learning is best to be viewed
as process, thus all learning is re-learning. This remark is also very important for
VR serious games design because the technology allows for multiple repetition
of the same scenarios with slight modifications - so that they are similar but not
identical.
Virtual reality allows for creating various, immersive environments for ac-
tive participation in training. This is also imperative for Experiential Learning
Theory approach - Lewin’s famous formula for explanation of one’s behavior is:
B = f (P, E)
where B is the behavior, P is person and E stands for environment. It means that:
"A physically identical environment can be psychologically different even for the
same man in different conditions" ([4], pp. 24-25). That statement translates
very well to the reality of VR: "The psychological reality [...] does not depend
upon whether or not the content [...] exists in a physical or social sense [...].
The existence or nonexistence [...] of a psychological fact are independent of the
existence or nonexistence to which its content refers." ([4], p. 38).
There are many theoretical frameworks taking from before-mentioned con-
cepts of situated learning. Urie Bronfrenbrenner’s work on the ecology of human
development [5], Situated Learning Theory by Lave and Wenger [6], theory of
knowledge creation by Nonaka and Konno [7] are worth mentioning here.
In presented case we strongly rely on the idea that environmental actions are
among most important learning factors and that the virtual (mental) environ-
ments are essentially no different in this aspect than the real-life situations.
3 Motivational Intensity Theory and Affective
Computing
One of the most serious problems when it comes to training firefighters is to
maintain the right level of their commitment. Generally, because of the nature of
their work, firefighters are highly motivated. However, the fact that participation
in real actions is associated with states of high emotional stimulation causes that
training may appear monotonous despite awareness of its significance.
The idea to solve the problem of repetitive (in the negative sense) and
monotonous exercises performed at the training field is to combine them with
those implemented in a virtual environment – providing wealth of scenarios and
surprising, and thus interesting situations.
While creating the solution for optimizing a psychological background of
knowledge acquisition in training, we used concepts from the Motivational In-
tensity Theory (MIT). As Richter, Gendolla and Wright state "motivation sci-
ence is concerned with the processes and mechanisms underlying the initiation,
direction, persistence, and intensity of behavior" [8]. In this context MIT is fo-
cused around mobilization effort in the pursuit of the goal. It concerns i.a. level
of energization related to presented task (and it’s valence). The most important
thing is that one can use cardiovascular measures as indicators of effort mobiliza-
tion [9], [10], [11], [12], [13] and activation of sympathetic and parasympathetic
nervous system.
Software recognition of user’s mental states using physiological measurements
in is one of the domains of Affective Computing. This paradigm takes as a
starting point statement that: "emotions are both physical and cognitive" [19].
This means that to some extent the analysis of psychological states can be carried
out by means of physiological data processing methods. The question arises - how
to create such computing systems and how to use them efficiently in the training
situations? Proposed answer involves applying affective patterns in designing
serious virtual reality games. Multiple conducted by our research team studies
present that it is possible to interpret some physiological indicators in relation to
the course of gameplay as signs of arousal, stress, frustration – or more generally
speaking – involvement [14], [15], [16], [17], [18].
4 Affective patterns in serious games design
The idea of Affective Patterns in Serious Games design is based on the applica-
tion of model developed by S. Björk and J. Holopainen in their book "Patterns
in Game Design" [1] in connection with affective measurements to evidence-
centered assessment design (ECD) [2], [20].
Fig. 2. Affective patterns in serious games framework (after: [2])
The model consists of a part focusing on the use of ECD (Evidence Model
- Assembly Template for ECD) and game design methods (DPE framework -
Design, Play, Experience). The key part in designing the player’s experience is
the use of gameplay-building mechanics. Mechanics [21] are constructed from
general patterns, which additionally contain a description of the physiological
correlates of emotions (and thus commitment) and evidences representing the
possession of specific skills, knowledge and abilities. When particular pattern is
applied in VR simulation it introduces specific mechanic - if the mechanic is
applied by the player, the system is able to evaluate correctness of conducted
action.
The problem faced by the designer of specific simulations concerns how to
construct the Evidence Model (which activities require particular skills?) and
what sets of stimuli should be prepared that will cause the appropriate reaction
of the user.
In order to prepare an appropriate simulation based on the solution described
above, we conducted an analysis of the content of the firefighters training pro-
gram, and a survey regarding stimuli - stressful situations.
5 Creating Affective Patterns Based on Surveys
Conducted on Firefighters
Simpro sp. z o.o., the creators of VR simulator - the company we work with
- conducted a survey aimed at predicting the stressful factors for firefighters.
Each firefighter could indicate no more than ten stressful factors and assess
their negative impact on a scale of one to five. 104 firefighters participated in
the survey and 103 responses was collected. One man replied that he did not
know any stressful situations. Next, only 93 responses had correctly assigned
weights.
Stressful factor Indication Average Standard
percentage rating deviation
Injury to other lifeguards 4.3 % 4.75 0.50
Relationship with the injured person 16.1 % 4.73 0.59
Injury to the firefighter 5.4 % 4.60 0.55
Explosion 4.3 % 4.25 1.50
Children 47.3 % 4.23 0.91
Death 9.7 % 4.22 0.97
Change in the injured person state 15.1 % 4.14 0.53
Helplessness 10.8 % 4.00 0.82
Chemicals 7.5 % 4.00 1.00
Large number of injured persons 43.0 % 3.98 0.77
Poor condition of injured persons 24.7 % 3.91 0.90
Assisting injured persons 8.6 % 3.75 1.39
Adherence to procedures 4.3 % 3.75 1.89
Fire 32.3 % 3.63 1.13
Noise 35.5 % 3.58 0.90
Insufficient resources 20.4 % 3.53 0.96
The dynamics of the situation 21.5 % 3.50 0.95
Faulty equipment 15.1 % 3.50 0.94
Amputations 6.5 % 3.50 0.84
Families of injured persons 26.9 % 3.48 1.08
Lack of expertise 9.7 % 3.44 1.01
Lack of skills 7.5 % 3.43 1.40
Panic 7.5 % 3.43 0.53
Number of lifeguards is not sufficient 12.9 % 3.42 0.67
Life in danger 12.9 % 3.42 0.79
Blood 12.9 % 3.33 0.78
Presence of the higher commander 6.5 % 3.33 1.86
Person trapped in the vehicle 6.5 % 3.33 0.82
Time pressure 23.7 % 3.27 1.08
Aggressive injured persons 4.3 % 3.25 1.26
Access to the victim is hindered 11.8 % 3.18 1.08
Commander’s pressure 6.5 % 3.17 1.60
The rescue action is recorded 26.9 % 3.00 1.04
Dangerous car 12.9 % 3.00 1.28
Lack of equipment 8.6 % 3.00 0.93
Comments of the onlookers 19.4 % 2.78 0.88
Unknown situation and surprise 7.5 % 2.71 1.11
Leak 30.1 % 2.68 1.02
Inconvinient neighbourhood of the action 21.5 % 2.60 1.05
Third parties and onlookers 74.2 % 2.59 0.91
Media 12.9 % 2.58 1.24
Collaboration with teammates 8.6 % 2.50 1.20
Traffic 14.0 % 2.46 1.33
Weather 18.3 % 2.24 0.97
Table 1: Firefighter survey report
Some responses have large dispersion of ratings. It means that either the
firefighters did not agree on the assessments or the stressful factors can have dif-
ferent degrees of severity. The most often indicated factors are: children (present
on the accident place), large number of injured persons, fire, noise, third parties
and onlookers. This answers served as a basis for creating patterns of affective
situations in the simulator.
6 Applying Affective Patterns in VR simulator
6.1 General design of VR simulator
Presented simulator is a project developed by Simpro4 sp. z o.o. (spinout com-
pany) and Nano Games5 sp. z o.o. (parent company).
4
See https://simprosoft.com/en
5
See https://nano-games.com
The project involves creating a VR simulation of a rescue operation with
implemented affective feedback loop. Having a VR multiplayer solution and a
wireless sensor set based on the Bitalino6 platform it is possible to apply the
concept of affective patterns in practice. The affective factor is based on the
usage of ECG (electrocardiography) and EDA (electrodermal activity) sensors.
In the case of the former, the HRV (hear rate variability) value is calculated.
Fig. 3. Firefighter during test training
For the purposes of the simulator evaluation, two basic contexts were created:
1. a car accident at an intersection in a small city,
2. an incident when transporting passengers to the aircraft at the airport.
In the first case, users train the skills of helping victims at the scene of the
accident, in the second, the triage procedure is practiced.
Based on surveys and analysis of the firefighters training program, proto-
type scenarios were prepared along with implementation in VR. Sets of patterns
matching the given contexts have been developed.
Patterns consist of:
1. Number/Codename - arbitral; e.g. "C-1";
6
See https://bitalino.com/en/community/publications
Fig. 4. Example VR scene - car accident
Fig. 5. Example VR scene - airport accident
Fig. 6. Training the triage procedure
2. Type - criterion was the existence of the procedure: patterns that have
an affective impact but do not significantly affect the course of the rescue
procedures are classified separately; separately, those that require a rescuer
to adhere to another procedure;
3. Name of the affective agent - e.g. "aggressive injured person";
4. Weight - according to analyzed surveys (1 to 5);
5. Design description - e.g. "injured", "aware", "aggressive towards the life-
guard"; "gradual - aggression can be verbal or physical"; "in the absence of
intervention, a person can worsen his condition";
6. Physiological description - e.g. "X increase in HR"; "Y decrease in HRV";
"Z increase in GSR";
7. Pattern activation - a pattern programmed into a simulation that is per-
manently activated or activates as a result of certain conditions (e.g. specific
reading from physiological sensors);
38 patterns that do not change the procedure; 12 affecting the procedure and 13
changing the procedure depending on the rescuer decisions were created for the
prototype.
6.2 Example scene - short description
An example scenario can be described as follows:
– Location: Airport;
– Context: Bus overturned with engine thrust;
– Weather: Dense fog;
Fig. 7. Setting up context - the overturned bus in the fog
On the scene, there are 6 victims described according to the formula:
– ID: 1100;
– Gender: Randomly;
– Age: Randomly;
– Triage category: Red;
– Walking?: No;
– Breathing?: No;
– Respiratory tract blocked?: No;
– Breathing after cleaning respiratory tract: n/a;
– Pulse?: Yes;
– CPR is working?: Yes;
– Aware?: No;
– Injuries: Head injury;
– Pattern: C-23;
– Pattern trigger: After 2 minutes;
There are 8 onlookers on the stage described according to the formula:
– ID: 1105;
– Gender: Female;
– Age: Adult;
– Pattern: C-34;
– Trigger: Physiological reading from sensor;
Fig. 8. Asking onlookers questions
Patterns mentioned (short description):
– C-23: after the beginning of the action one of the victims dies;
– C-34: one or more people behave intensely towards a rescuer: follow him,
comment on his actions, claim that know how the rescuer should act, question
him;
Fig. 9. Pattern C-34: Annoying person at the scene
7 Summary and Future Research
The paper presents a prototype of a adaptable VR simulator for training fire-
fighters. This application is a practical application of the concept of Affective
Patterns in Serious Games [2]. Currently, the software is undergoing testing and
adjustment of algorithms responsible for the interpretation of the results of phys-
iological readings. Ultimately, the alpha version will implement 2 full contexts
with 40 validated patterns.
For the future versions, we are working on the algorithmic description of
patterns, as well as automation of the generation of the components of the sim-
ulation, e.g. victim description. Furthermore, control in VR is an important
challenge. An important improvement would be the use of eye-tracking which
could be available in the next versions of VR headsets.
References
1. Björk, S., Holopainen, J.: Patterns in Game Design. Charles River Media, Rockland,
MA, USA (2004)
2. Argasiński, J. K., Węgrzyn, P.,: Affective Patterns in Serious Games. Future Gen-
eration Computer Systems 92, 526–538 (2019)
3. Kolb, D.: Experiential Learning: Experience As The Source Of Learning And De-
velopment. 2nd Edition. Pearson FT Press, Upper Saddle River, NJ, USA (2015)
4. Lewin, K.: Principles of Topological Psychology. McGraw-Hill, New York, USA
(1936), pp. 24-25
5. Bronfrenbrenner, U.: The Ecology of Human Development. Harvard Univ. Press,
Cambridge, MA, USA (2079)
6. Lave, J., Wenger, E.: Situated Learning: Legitimate Peripheral Participation. Cam-
bridge Univ. Press, Cambridge, UK (1991)
7. Nonaka, I., Konno, N.: The concept of “ba”: Building a foundation for knowledge
creation. California Management Review 30(3), 40–54 (1998)
8. Richter, M., Gendolla, G., Wright, R.: Three decades of research on motivational
intensity theory: What we have learned about effort and what we still don’t know.
Advances in Motivation Science 12(3), (2016)
9. Wright, R.A.: Brehm’s theory of motivation as a model of effort and cardiovascular
response. In: Gollwitzer, P. M., Bargh, J. A. (Eds.): he psychology of action: Linking
cogni- tion and motivation to behavior, Guilford, New York (2996)
10. Obrist, R. A.: Cardiovascular psychophysiology: A perspective. Plenum Press, New
York (1981)
11. Newlin, D. B., Levenson, R. W. Pre-ejection period: measuring beta-adrenergic
influences upon the heart. Psychophysiology, 16 (1979)
12. Richter, M., Friedrich, A., Gendolla, G. H. E.: Task difficulty effects on cardiac
activity. Psychophysiology, 45 (2008)
13. Wright, R. A., Killebrew, K., Pimpalapure, D.: Cardiovascular incentive effects
where a challenge is unfixed: demonstrations involving social evaluation, evaluator
status, and monetary reward. Psychophysiology, 39 (2002)
14. Nalepa, G., Giżycka, B., Kutt, K., Argasiński, J.: Affective Design Patterns in
Computer Games. Scrollrunner Case Study. 345-352, (2017)
15. Nalepa, G., Argasiński, J., Kutt, K., Wegrzyn, P., Bobek, Sz., Łępicki, M. Z.: Af-
fective Computing Experiments in Virtual Reality with Wearable Sensors. Method-
ological considerations and preliminary results, (2016)
16. Argasiński, J., Grabska-Gradzińska, I.: Patterns in Serious Game Design and Eval-
uation Application of Eye-Tracker and Biosensors, (2017)
17. Grabska-Gradzińska, I., Nowak, L., Argasiński, J.: Applying Oculography and
Biosignals Measurements to Gameplay Modes Analysis, (2020)
18. Grabska-Gradzińska, I., Argasiński, J.: Patterns in Video Games Analysis – Ap-
plication of Eye-Tracker and Electrodermal Activity (EDA) Sensor, (2018)
19. Picard, R.: Affective Computing. MIT Press. Cambridge, MA, USA (1998)
20. Mislevy, R., Rusell, G. A., Lukas, J. F.: A Brief Introduction to Evidence-centered
Design. Research Report, Research and Development Division Princeton Univ.
Princeton, NJ, USA (2003)
21. Sicart, M.: Defining Games Mechanics. Game Studies. 8(2), (2008)