This paper describes a preliminary study on how Interaction Fidelity, shaped by a combination of visual, auditory and haptic modalities, impacts the user's workload. A VR escape room environment consisting of 5 puzzles to be solved in a pre-defined order is presented. Preliminary analysis shows that further investigation on the VR escape room environment could provide insights on how IF influences the user's workload depending on the type of task. CCS Concepts: • Software and its engineering → Interactive games; • Human-centered computing → Virtual reality; User studies. Additional Key Words and Phrases: VR, multimodality, interaction fidelity, user's workload
MAURIZIO
One of the goals of virtual reality (VR) is to ensure the user’s workload is not over or under loaded in problem-solving
tasks. While workload has been defined in many ways, our understanding falls in line of that of Hart [
Joint workshop on Games-Human Interaction (GHItaly21) and Multi-party Interaction in eXtended Reality (MIXR21), July 12, 2021, Bolzano, Italy Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Indeed, a user’s workload may be impacted by the amount of resources a person is required to complete a task,
which include the number of modalities, such as visual, auditory, and haptics, implemented [
According to Rogers and colleagues [
To our knowledge, research on how diferent levels of IF and sensory feedback impact performance remains nascent.
Indeed, Rogers et al. [
While recent improvements of VR technology (see, e.g., exergames [
In this paper, we describe a VR environment consisting of an escape room involving the solving of 5 puzzles in
a pre-defined order. We aim to study how diferent levels of IF, shaped by diferent combinations of visual, auditory
and haptic modalities, impact the user’s workload. The exploited escape room scenario was chosen as it is a mentally
and physically demanding activity consisting of everyday tasks, such as opening a locked drawer, using a screwdriver,
opening a safe, using a flashlight and finding some items (see Figure 1). For example, let us consider a person leaving
home to go for shopping. The person needs to find their shopping list, hidden somewhere in the kitchen; to enter their
car, they need to unlock the car door; while shopping in the supermarket, they have to scan products before checking
out. These activities, which are deemed fundamental for a healthy independent leaving [
This study was conducted during the COVID-19 pandemic, and therefore needed to be conducted remotely. As a result, another contribution of the paper is to propose a methodology for conducting VR evaluation studies in a remote setting in which participants are self-directing themselves while carrying out the study. 2
An escape room environment is a game in which one or more players are locked in a room setup according to a specific theme. We implemented a VR escape room environment by combining realistic visuospatial elements, with an ontologically supported narrative impetus. The opening shot of the game quickly establishes the visuospatial immersion for the player: Initially visible are the silhouette of a non-player character (NPC) watching the player from another room. The exit door and the pinpad to open the door are located to the player’s left, with the first puzzle to their right. The player has five seconds to orient themselves in the scene before the initial exposition dialogue begins. Narrative impetus is continued throughout the experience as prompts and taunts are delivered by the NPC at timed intervals, while clues and flavour decals carry an aesthetic consistent with the escape room. 2.1
With the VR escape room environment we aim to investigate how a diferent number of modalities influences user’s workload, based on the type of the performed task (exploration, search, manipulation, cognition). So, we designed and implemented 5 puzzles (see Figure 1) allowing the player to find the 4 digits of the pinpad and escape the room: (1) Padlock and Key: the desk in the room contains three drawers, with the middle one visibly padlocked; some books on the bookshelf have to be moved out to reveal a key; once the drawer is unlocked it reveals a grabbable screwdriver and the first digit of the pinpad; (2) Screwdriver: a picture frame hangs on the wall, with the words “look at me” textured on it, serving to initially draw the player’s attention and notice the screws, while also serving as a callback-clue once the flashlight has been acquired (see puzzle 4); the frame hides the safe puzzle behind it; (3) Safe: the safe puzzle is the focus of the pilot study presented in Section 3; the player is required to open the safe by rotating the knob through a series of actions intended to mimic a real safe; completing all the correct rotations enables rotation of the handle to open the door, which reveals the second digit of the pinpad; a blacklight lfashlight is placed inside the safe; (4) Flashlight: the flashlight functions as a UV-A light; the third pinpad code digit is hidden in the picture frame and can be revealed by directing the flashlight at it; (5) Bed / Hidden Note: with one digit left to be found, the only option for the player is to search the remaining locations of the room; the corner of a note is visible under the pillow on the bed, giving a hint to move it out to reveal the final digit of the pinpad.
The escape room (Figure 2 is written in C# and was designed in the Unity engine. Unity was selected as it is a powerful 3D and scene-oriented engine, with sophisticated support for mainstream VR platforms. The included package editor allowed for the OpenXR, Oculus SDK, and VRTK packages to be easily implemented in the application, with the Oculus SDK providing prefab assets that can be used to rapidly construct a player rig. Blender was used to import and adjust open-source models that were not displaying correctly in the Unity environment, or to modify certain meshes used to create the escape room geometry. Audacity was used to edit audio clips and sound efects, with these clips being acquired from open-source libraries such as freesound.org. Voicemod was used to record dialogue for the NPC, and edited via Audacity. GIMP was used to modify and create textures throughout the experience, and was particularly efective for ensuring visual consistency across the clue style. The escape room application is deployed as a reasonably small executable file (about 80 MB) that can be easily sent and played remotely with an Oculus Rift or Quest (v1 and v2) headset. We designed and implemented a pilot experiment on puzzle 3 of the escape room described in the previous section: the safe opening task. So, in the pilot experiment we focus on the workload of a manipulation task (i.e., the user has to pay attention on how to use their hands in manipulating the safe knob) in relation to IF. In particular, we aim to investigate how many and which interaction modalities (independent variable) influence user’s workload (dependent variable) of a user performing a manipulation task in a VR scenario (RQ1). 3.1
To evaluate the efects of the independent variable (the modality combinations) on the dependent variable (the user’s
workload), four conditions using diferent combinations of modalities were implemented: visual only (VNN),
visualauditory (VAN), visual-haptic (VNH) and visual-auditory-haptic (VAH). According to existing works on IF (see [
If present, the interaction modalities look/sound/feel the exactly same throughout the 4 conditions. That is, the IF of an individual modality does not change (i.e., it is constant within that modality), while the overall IF, that takes into account all the modalities of an experiment condition as a whole, changes (i.e., it varies among the modalities). 3.2
The safe opening is the most complex puzzle developed for the escape room game (see Section 2.1). The safe has a grabbable knob and handle, and the player, to open the safe, is required to rotate the knob through a series of pre-defined positions, mimicking the opening of a real safe. Correct positions progress the puzzle, while incorrect positions reset progress. Completing all logic positions enables the rotation of the handle which, once the target value is reached, triggers the opening of the safe door.
The experiment was approved by the Ethical Committee of the University College Cork (Ireland).
An example video demonstrating the puzzle is available at: https://www.youtube.com/watch?v=uMLZEraWXis
To create a realistic safe knob, light haptic feedback is applied to the VR controller as the knob is rotated, with an
array of lock-click sounds played at random in conjunction with the haptics. Additional feedback occurs when the knob
enters positive and negative trigger zones. Positive zones trigger positive-click sounds and the player receives a hard
vibration, while negative zones trigger negative-click sounds and an extra-hard vibration. The click-sounds were chosen
to align with earcon design philosophy [
A sophisticated logic controller script was created for the safe as it involved a large number of objects and assets to achieve the desired functionality. This script facilitates, e.g., enabling or disabling individual sensory modalities, which is efective when implementing the safe across the experiment conditions (see Section 3.1). 3.3
Participant recruitment was achieved through advertising the experiment on mailing lists and forums of researchers in VR, who could potentially have access to their personal VR equipment. In total, 8 participants (labeled P1-P8) agreed to be involved in the study. 3.4
We adopted the NASA Task Load Index (NASA-TLX) questionnaire [
The experiment survey was split into 3 main sections:
• preliminary questions: we asked the participant to provide informed consent to take part in the experiment and
to enter anonymous demographics information (age range, gender, VR knowledge, manipulation skills);
• NASA-TLX questions: after playing each of the 4 conditions (VNN, VAN, VNH, VAH), the participant had to fill
out the full NASA-TLX questionnaire [
After answering some preliminary questions about informed consent and demographics, participants were instructed to install and launch the VR application. Then, they were asked to put on the VR headset and play the first condition. Next, they filled out the NASA-TLX questions of the survey and reiterated the entire process (gameplay, followed by NASA-TLX questions) for the remaining 3 conditions. The conditions were presented in random order for each participant. Finally, they were asked to fill out the final questions that are described in Section 3.1. 3.6
We conducted and report here a preliminary exploration of the experimental data. The safe opening puzzle workload was computed, for each participant in each condition, as the sum of the weighted ratings provided by participants while iflling out the NASA-TLX. Moreover, we computed the median of the workload and the time to open the safe within participants playing the same condition (VNN, VAN, VNH, VAH). The values are reported in Table 1.
Since the workload and timing scores were not normally distributed, we ran two separate Friedman tests with the modalities combination (and, consequently, the IF) as within-subjects variable and the workload score and safe opening time as dependent variables. The results did not indicate significant diferences between the conditions: 2 (3) = 1.412, , = 0.06, = 0.7 (workload); 2 (3) = 1.95, = 0.08, = 0.583 (safe opening time). These results were not surprising, due to the very low sample size (8 participants). However, as we said above, we are still collecting participants, and we expect significant diferences between conditions to emerge as the number of participants will increase.
In the third part of the survey we asked participants to answer a set of qualitative questions about the task workload. Below, we reveal how the participants responded to the scenarios across two questions: • What elements were the most distracting? Ranked in order of frequency chosen, the most distracting conditions were VAH (4), VNH (2), VAN, (1), VNN (1). Visual-only ranked as one of the least distracting. When prompted to justify their choice, participants who chose VAH, VNH and VAN were distracted by the diversity of feedback. For P6, the simultaneous presence of audio and haptics resulted in them making “some mistakes in rotating the knob”. For P3, P4, P5 and P6, the ticking sound of the timer heightened anticipation, which impacted their focus on the task. As P3 highlighted: “the ticking of the timer made me nervous for like 3-5 seconds until I realized there is enough time to complete the task”. P3 also commented on the realism of the haptics as also being a potential distraction: “it just doesn’t feel like gripping in the real world” (P3). • Which condition required the most/least efort? For 5 participants, the last condition they chose was ranked as the least efort. As P1 emphasised, they were “already familiar with the safe logic”. However, for P4, who chose VNH, the reason was because “there was the haptic feedback only”. P4 indicated that the sound of the timer was distracting, which might appear that removing this stimulus helped the user to focus on the task at hand. 4
This paper presents a VR escape room environment consisting of 5 puzzles that must be completed in a given order.
We aim to study if and how diferent levels of IF, shaped as modalities combinations, influence the user’s workload depending on the type of task. In the pilot experiment described in the paper, we focus on one of these puzzles, the safe opening one.
Despite their number being relatively low, we collected some qualitative impressions from participants, carrying out a preliminary analysis of the workload and time to open the safe.
The main contributions of the paper are: (1) to conduct a first investigation on the workload of performing diferent types of task in a VR environment; (2) to define a methodology to carry out VR studies in a remote setting.
Further research on the combination of modalities, e.g., sequential vs parallel [