=Paper=
{{Paper
|id=Vol-2246/GHItaly18_paper_01
|storemode=property
|title=Evaluation of Virtual Reality Games: Simulator Sickness and Human Factors
|pdfUrl=https://ceur-ws.org/Vol-2246/GHItaly18_paper_01.pdf
|volume=Vol-2246
|authors=Kent Norman
|dblpUrl=https://dblp.org/rec/conf/avi/Norman18
}}
==Evaluation of Virtual Reality Games: Simulator Sickness and Human Factors==
https://ceur-ws.org/Vol-2246/GHItaly18_paper_01.pdf
Evaluation of Virtual Reality Games: Simulator Sickness
and Human Factors
Kent L. Norman
Department of Psychology, University of Maryland, USA
klnorman@umd.edu
ABSTRACT Simulator Sickness
This study seeks to evaluate a virtual reality system and Simulator sickness was first observed when persons were
games available on the PlayStation 4 VR system. being trained using flight simulators and other training
Participants completed two questionnaires during the environments particularly in the military. The common
experiment. The first questionnaire was developed to symptoms are general discomfort, headache, nausea,
evaluate simulator sickness in high-fidelity simulators [3]. vomiting, pallor, sweating, fatigue, drowsiness, and
The second questionnaire was developed by our lab to disorientation [4]. To quantify the extent of simulator
assess a number of characteristics of the VR environment sickness on participants, Kennedy, Lane, Berbaum, &
(e.g., image quality, auditory surround, orientation, Lilienthal [3] developed the simulator sickness
pointing, lag, and control). We have collected a sample of questionnaire (SSQ) as shown in Figure 2.
85 participants. Participants were asked to complete
surveys about demographics and current video game use
and then choose a game from a list of ten different games to
play on the VR system. The seven games were Demo Disk,
Batman, Dangerball, Eve Valkyrie, Headmaster, London
Heist, Rush of Blood, and VR Luge. Each participant played
the game for about 30 minutes. Finally, they completed the
simulator sickness and the VR system questionnaires.
Simulator sickness was low for 58% of the participants,
mild for 33%, and strong for 9%. Scores varied among the
games. Ratings for sound location and image lag were good
but calibration, head gear, some image blurriness were less
favorable. These results can be used by game producers to
develop more satisfactory VR game environments.
Author Keywords
Augmented reality; human factors; simulator sickness;
video games; virtual reality
ACM Classification Keywords
H.5.m. Information Interfaces and Presentation (e.g. HCI):
Miscellaneous.
INTRODUCTION
Virtual reality systems have been in research and
development for decades since the early days of Sutherland
and his students [7]. However, in the past couple of years
virtual reality games have been introduced to the consumer
market primarily for immersive games. These systems
integrate a number of components to create the user
experience as shown in Figure 1.
Figure 1. Components of a virtual reality system from [5].
Virtual reality environments create a number of challenges
for the user which will be discussed in the next two Users of virtual reality systems often experience the same
sections. discomforts of training simulators. VR sickness can be a
major deterrent to the adoption and use of VR [1].
Human Factors
nd
Virtual and augmented reality environments pose a number
GHItaly18: 2 Workshop on Games-Human Interaction, May 29th, 2018,
Castiglione della Pescaia, Grosseto (Italy)
of human-factors issues which have been listed by Jerald
Copyright © 2018 for the individual papers by the papers' authors. Copying [2]. A number of these are possible contributors to
permitted for private and academic purposes. This volume is published and simulator sickness. These include headgear discomfort,
copyrighted by its editors. system calibration, image lag, image blurriness, auditory
surround, control of movement, ease of pointing and Demo Disk: Selection of demo games. Head position, PS
selection, and awareness of body location. controller, or PS Move controllers can be used to select
demo games. Demo games use a variety of interaction
After testing out a number of VR games and considering styles.
the literature on the human factors in VR, a questionnaire
was designed to assess a number of these factors on a 9-
point scale. The virtual reality system questionnaire
(VRSQ) is shown in Figure 3. This study will be used to
evaluate this questionnaire and improve it for future
research.
Figure 2. The simulator sickness questionnaire (SSQ) taken
from Kennedy, Lane, Berbaum, & Lilenthal [3].
Usability and Play Testing
In order to assess levels of simulator sickness and evaluate
games and systems, participants were asked to engage in a
form of play testing [6]. They were asked play a game for a
certain length of time and then respond to questions about
how they felt (SSQ) and how the system performed
(VRSQ).
Study
In order to assess the prevalence of virtual reality sickness
and to assess human factors issues, a popular commercial
system was selected and a variety of different games were
selected for testing. Participants played the games and
completed the questionnaires.
METHODOLOGY
System and Games
The PlayStation 4 VR system was used with its headgear,
PlayStation Eye, and either a standard controller or two
PlayStation Move controllers. The system was set up in a
user testing room that was 10 x 18 feet. The participant was
usually seated in a swivel chair or for some games standing
in a fixed position. An assistant set up the game, helped the
participant with the headgear and controller(s), and in some
instances helped to instruct the participant as to what to do
during the game.
The seven games were selected to cover a range of genre
and interaction styles. The games were as follows:
Procedure
Participants were greeted and told what was expected of
them in the study and signed an informed consent form.
First they completed a demographic question and a survey
of their video game interests and experience. They were
then asked to select a game to play from a list giving the
title of the game, its ESRB rating, cover art for the game,
and a short explanation of what the game was about. A lab
assistant set up the game, helped with the headgear and
controllers, and started the game. Participants played the
game for 30 minutes unless they asked to stop due to
sickness. Eight of the 85 participants asked to stop due to
sickness.
After playing the game, participants were asked to fill out
the SSQ and VRSQ questionnaires.
RESULTS
VR Sickness
Figure 4 shows the order of occurrence of the 16
symptoms, with headache, nausea, sweating, fatigue,
general discomfort, “fullness of head” and eyestrain
Figure 3. Items on the virtual reality systems questionnaire topping the list.
(VRSQ).
Batman: Arkham VR: Action adventure mystery. Move
controllers are used to grab and throw objects and for menu
selection. Movement is from one discrete location to
another via teleportation. Virtual hands are displayed.
Danger Ball: VR Worlds: Sports action similar to
dodgeball. Head, hands, and feet are used to dodge and hit
balls at targets. No controller.
EVE: Valkyrie: Action shooter. PS controller is used to
steer a spaceship and to fire weapons at enemies. Player is
seated in the moving spaceship.
London Heist: Action adventure first person shooter. PS
controller or PS Move controllers are used to shoot and
grab objects. Player is in fixed positions. Virtual hands and
guns are displayed. Figure 4. Mean rating of each SSQ item.
Rush of Blood: First person shooter. PS Move controllers Participants varied greatly in their susceptibility to VR
appear as guns in the player’s hands. Virtual legs and feet sickness. Clearly some participants experienced VR
are also displayed. Player is seated in a roller coaster car sickness in some games. Eight participants requested to
that moves on its own. stop the game due to sickness. Figure 5 shows the
distribution of SSQ scores for the 85 participants. The
VR Luge: Sports racing. Steering of the sled is done by majority of participants fell in the low range (58%), a third
moving the head to left or right. No controller is used. fell in the mid range (33%), and 9% fell in the high
Virtual body is shown on the sled. range.When clustered by game as shown in Figure 6, it is
clear that some games led to higher levels of VR sickness
than others. Eve: Valkyrie was the worst, probably due to
Participants the disorientation of spinning around in the spaceship. The
Undergraduates were recruited from psychology courses at Demo Disk was second. In this case, participants went
the University of Maryland and received partial credit for from a central VR space into demo games and back. This
participation. The sample was composed of 36 females and resulted in a change in orientation and mode of controlling
49 males and ranged in age from 18 to 45. The mean age the interaction which could lead to disorientation. The VR
was 19.87, the median 19, and the mode 18. Participants Luge was third worst. In this game, the player is whipping
varied widely in video game experience, time and down a winding road at high speed trying to avoid
frequency of play per day, and types of games played. collisions with moving vehicles and stationary objects.
None of the participants had played a VR game before.
Mean SSQ Score
Figure 5. Distribution of SSQ scores. Figure 6. Mean SSQ score by game
Games in which the player was in a stationary or seated The lab assistant usually had to help them. Methods of
position (The London Heist, Rush of Blood, and Batman) calibration varied from game to game and again the lab had
led to the least VR sickness. to help. On the positive side, there were few auditory
problems and sounds were easy to locate. The image did
Human Factors not skip or lag, it appeared in 360 surround, and it was easy
Figure 7 shows the mean rating for each item on VRSQ. to look to the left or the right.
All of the ratings were above midpoint of 5 except for
“Some of the image was blurred.” The lab assistants and When sorting these ratings out by game as shown in Figure
participants found that perspiration often fogged up the 8, the largest differences were for “Trying to see what is
headgear and blurred part of the field of view. “Trying to behind.” Batman faired the best probably because the game
see what is behind” was difficult either turning the head or action was always directly in front of the player. Eve:
swiveling the chair. Headgear was also a problem. Valkyrie was the worst due to the configuration of the
Participants were not initially familiar with how to adjust spaceship.
the helmet and the viewing screen for maximum comfort.
Figure 7. Mean rating of each VRSQ item.
Figure 8. Mean rating of each VRSQ item by game.
Large differences also occurred for “Locating buttons on In order to control the effects of the VR environment,
the PS controller.” This appeared to be due to the extent of developers need to evaluate games along a number of
use of the PS controller in the game. The same was true for human factors. The VRSQ was designed in this study to
“Aiming at targets using the head” again probably due to assess these factors. The visual quality of the environment
the extent of use of this method in the game. Other depends on the clarity of the image of game focus as well
differences also occurred due to the game’s reliance one as the 360 surround for a number of games. The image lag
that feature. “Fast image lag” was a problem for VR Luge is particularly relevant for fast moving environments in the
due to the high rate of speed of the player moving through game play. Many different methods for menu selection,
the streets; whereas in Rush of Blood, the VR space and pointing, aiming, shooting, and grasping are being
targets within it did not move as fast. In games such as explored. Movement through the VR space whether
Eve: Valkyrie that requires the player to look all around for continuous or discrete point-to-point must be worked out.
objects, “All of the image blurred” was a problem whereas Each factor and interaction style needs to be evaluated by
in London Heist and Batman, the player really only needs players as the game is developed and in final play testing.
to focus on one spot. The VRSQ is meant to assist in game development to
minimize virtual reality sickness and maximize the player’s
CONCLUSION VR game experience.
The use of virtual reality in video games is in its infancy.
Considerable research and development is needed to ACKNOWLEDGMENTS
integrate game play into the VR environment. Game Thanks to Emily Maranto, Kaitlin Cossentino, Karsyn
producers need to play-test their games to evaluate them Wagner, Meredith Reid, Joey Barnet, and Reza Mahmud
across a number of factors for playability [7]. For VR for running participants and assisting in the data analysis.
games, developers need to evaluate how likely the VR
environment will lead to discomfort and to VR sickness REFERENCES
and for what proportions of the player market. Clearly from [1] Johnell O. Brooks, Richard R. Goodenough, Matthew
these data, different games evoke different levels of VR C. Crisler, Nathan D. Klein, Rebecca L. Alley,
sickness. Beatrice L. Koon, William C. Logan Jr., Jennifer H.
Ogle, Richard A. Tyrrell, Rebekkah F. Wills. 2010. [5] Kent L. Norman. 2017. Cyberpsychology: An
Simulator sickness during driving simulation studies. Introduction to Human-Computer Interaction, 2nd
Accident Analysis & Prevention, 42, 3: 788-796. Edition. Cambridge University Press, 167.
[2] Jason Jerald. 2015. The VR book: Human-centered [6] Randy J. Pagulayan, Daniel V. Gunn, Jerome R.
design for virtual reality, ACM Books (Book 8), Hagen, Deborah J. O. Hendersen, Todd A. Kelley,
Morgan & Claypool Publishers. Bruce C. Phillips, J. J. Guajardo, and Tim A. Nichols.
[3] Robert S. Kennedy, Norman E. Lane, Kevin S. 2018. Applied user research in games. In The Wiley
Berbaum, & Michael G. Lilienthal. 1993. A simulator Handbook of Human Computer Interaction Set,
sickness questionnaire (SSQ): A new method for Volume 1, Kent. L. Norman & Jurek Kirakowski
quantifying simulator sickness. International Journal (eds.), 299-346.
of Aviation Psychology, 3, 3: 203-220. [7] Ivan E. Sutherland. 1968. A head-mounted three
[4] Eugenia M. Kolasinski. 1995. Simulator Sickness in dimensional display. Proceedings of the AFIPS Fall
Virtual Environments. U.S. Army Research Institute Joint Computer Conference, Thompson Books,
for the Behavioral and Social Sciences, Technical Washington, D. C.,757-764.
Report 1027.