Effects of Environmental Clutter and Motion on User Performance in Virtual Reality Games Lal Bozgeyikli Srinivas Katkoori Abstract University of South Florida University of South Florida With the increasing prevalence of virtual reality, games Tampa, FL 33620, USA Tampa, FL 33620, USA for this medium have also been increasing in number gamze@mail.usf.edu katkoori@mail.usf.edu lately. Several factors affect the balancing of video games. However, effects of environmental clutter and Andrew Raij Redwan Alqasemi motion on game design for virtual reality have not been University of Central Florida University of South Florida well explored yet. Environmental clutter and motion are Orlando, FL 32826, USA Tampa, FL 33620, USA important in making virtual environments more raij@ucf.edu alqasemi@usf.edu interesting and closer to real life since real world environments usually include some form of clutter or motion. Total exclusion of clutter and motion may make the virtual environment drab and dull. On the other hand, these components may affect the user performance, making the tasks and the game in general more difficult. This study aims at exploring the effects of clutter and motion on task performance in a virtual reality searching game. A user study was performed with 15 participants. Results indicated that clutter effected user performance negatively, made the users feel more restricted inside the virtual environment and made it more difficult to concentrate. No effect of motion on user performance was observed. Author Keywords Virtual reality; game design; clutter; motion; virtual environments. ACM Classification Keywords effective and well-designed virtual environments may H.5.1. Information interfaces and presentation (e.g., leverage these applications and provide more user HCI): Multimedia Information Systems—Artificial, friendly everyday experiences. As an endeavor in this augmented, and virtual realities. area, in this study, we explored the effects of clutter and motion on user performance in a VR searching Introduction game. The goal of the game was to find as many boxes Video games intend to provide entertaining, fulfilling as the user could within a short timeframe. Effects of and meaningful experiences for the users. Several clutter and motion on user performance and user components effect user experience such as game experience was evaluated in a user study with 15 mechanics, narrative and interface [10]. Balancing of participants. challenge and pace is crucial to keeping the users in flow zone; giving them enough challenge so that they Effects of clutter and movement on task performance in do not feel bored yet over-challenged at the same time. VR games has not been well studied yet. It might be Besides the goals, obstacles and the rules of the game; thought that less clutter and movement may simplify several factors may affect the perceived difficulty of a the task and lead to better user performance. However, task such as the environmental properties. it may also make the games too simple and boring, and degrade the quality of training offered by serious In the last few years, with the release of the second games. The motivation behind this study is to give generation virtual reality headsets such as Oculus [9] insight into future VR games for better balancing and HTC Vive [11], number of games that are designed (entertainment games) and more effective training for this medium have also increased in number. Virtual (serious games). This study tries to emphasize the reality (VR) have been attracting attention for both importance of environmental elements in game design entertainment and serious games. Serious games can since these elements may have a direct impact on the be described as video games with an aim of teaching perceived difficulty of the task, hence the game new skills or training the users on the already learned balancing. ones [7]. Since VR offers several advantages such as safety, easy customization, automated data collection Related Work and no severe real life consequences of mistakes, it has There have been very few studies that explored the been a popular choice for training in various areas [2, effects of clutter and movement on user performance 3, 6]. Training in VR is usually achieved via serious or experience in VR games. Ragan et al. studied the games. However, since VR is relatively new, video effects of visual complexity and field of view on training game design for this medium have not been well effectiveness for visual scanning tasks in virtual reality studied yet. Virtual environments are anticipated to systems [8]. The authors included clutter, dynamism have prevalent use in our daily lives in the future in and textural fidelity in their visual complexity many areas such as gaming, education, training, component so that the increased visual complexity sports, entertainment events and communication. Thus, included more realistic graphics and more static objects such as vehicles, people, plants and street lights. The made it more difficult for the users to track them since users were requested to find the targets around an there were no consistent reference points. As the urban environment while automatically moved under participants were interviewed, twenty-five out of the different conditions. Results indicated that higher visual thirty stated that the motion made the task more complexity worsened the performance. However, the difficult rather than the clutter. Since the study definition of visual complexity in the study included examined the clutter and motion in a single condition, many elements and the effects of separate elements effects of separate components on user performance were not examined. Bacim et al. explored the effects of were excluded from the study’s scope. visual display fidelity, visual complexity and task scope on spatial understanding of graphs in virtual reality [1]. The Searching Game The study included abstract visuals such as lines, A virtual reality searching game was designed and numbers and geometric shapes. The cluttered case implemented to evaluate the effects of clutter and included more of these visuals than the non-cluttered motion on user performance. The goal of the user was case. The users were requested to perform four tasks to find and mark the boxes with the matching labels using 3D graphs: intersection search, path following, with the one that was shown on the virtual display. The connection identification and length comparison. boxes were positioned on six virtual shelves Results indicated that higher clutter led to slower surrounding the user (Figure 1). As the user touched a performance in terms of time. However, clutter did not box with their hands, the color of the box changed and affect the correctness of the results. Since the study did got highlighted in pink to provide real time feedback. not include realistic visual elements but only abstract The users could walk on the tracked area to select the ones, the results may not be applicable to games and virtual boxes (Figure 2). The virtual shelves were applications having realistic visual elements. Ferrer et placed to fit inside the tracked area so that the users al. studied the effects of background motion and visual didn’t need to step out of the tracked area. If the users clutter on perception of virtual object motion in selected a box unintentionally, they could deselect it by augmented reality [4]. Although augmented reality is touching again. When deselected, the boxes returned to different than virtual reality, the study is related in their original color. The labels of the boxes were always terms of implications of clutter and motion on user facing the user so that they did not need to move performance on a perception task. The users were around the shelves to see the other faces of the boxes. requested to track the velocity changes in moving There was a time limit of 25 seconds that was imposed virtual particles on a black background and a to be able to explore the effects of different conditions cluttered/high motion background. Results indicated on user performance. Each shelf had three levels. Each that the presence of clutter and motion degraded user level of each shelf had three boxes and each shelf had performance and made it more difficult for the users to twelve boxes in total. For clutter, there were two perceive the velocity of the tracked particles. The conditions: no-clutter and clutter (Figure 3). For authors interpreted that clutter and motion caused motion, there were also two conditions: no-motion and perceptual illusions in tracking the moving particles and motion. Moving objects were as follows: a forklift, five small fans on the shelves, a big fan on the ceiling, flowing text on six displays attached to shelves, three blinking reflectors, two smoking bins and two swinging machine arms. Clutter and motion were not only in the background, but also in the workspace. Figure 1: A layout sketch of the virtual reality searching game. Hardware 12 Opti Track V100R2 FLEX cameras were used for real time motion tracking. The size of the tracked area was 8ft by 8ft. A VR2200 head mounted display (HMD) was used for viewing [12]. HMD was tracked by the system in real time via markers attached on top. The game was implemented using the Unity game engine and worked around 60 frames per second. Users also wore hand bands that were equipped with reflective markers Figure 2: Four conditions of the VR searching game. From top to bottom: (1) No clutter - no motion. (2) Clutter - no motion. for real time hand tracking. (3) No clutter - motion. (4) Clutter - motion. Experiment Design Research Questions and Hypothesis Two by two within subjects experiment was performed Our study aims to answer the following research with the independent variables of clutter and motion. question: What are the effects of clutter and motion on Both independent variables had two levels: no clutter - user performance in virtual reality games? We clutter and no motion - motion, making four levels in developed the following two hypotheses: (H1) Clutter total. The level of these two variables were varied will effect user performance negatively. (H2) Motion will within subjects in four conditions: no clutter-no motion, effect user performance negatively. clutter-no motion, no clutter-motion, and clutter- motion. Each participant completed three instances Data Collection with each condition. The configurations were assigned Automated data was collected for the following: box to each user in a random order with counterbalancing. distribution, number of hits, misses, unintentional In each instance, a different box label was presented to touches and corrections with their time logs. After the the users so that no two instances of the total twelve users completed three repetitions of a condition, a had the same two labels. The users needed to find as survey was given to them that had questions on the many boxes with the requested label as they could perceived difficulty of the task, frustration, ease of within 25 seconds. This duration was found by in-house finding the boxes, distraction, feeling of being testing as the duration in which a user familiar with VR restricted, ease of concentration, presence and motion could select all of the boxes. We avoided giving the sickness as well as user comments. users too much time to be able to detect the differences between different conditions. Each trial Participants contained 10 boxes with the requested label that were 15 adult individuals participated in the study (N = 15) distributed roughly evenly among the six shelves. The who were recruited via e-mail announcements and users were not informed on the number of boxes in the word of mouth. All participants were undergraduate or scene with the requested labels to make them keep graduate university students from several different searching. Box distributions and the labels were majors. Participants were aged between 21 and 33 (µ = assigned randomly to the instances to eliminate any 25.80, SD = 3.05). Gender distribution was 5 female possible learning effect. The score was calculated as and 10 male. 13 participants had no prior virtual reality follows: Number of hits - number of misses + experience, 2 participants had minimal prior virtual (0.5)*number of corrections. Hits were the boxes that reality experience. The user study was conducted under were correctly selected (matching with the requested the IRB Pro00013008. label). Misses were the boxes that were incorrectly selected (not matching with the requested label). Procedure Corrections were the deselections of the misses. Participants arrived at the laboratory, read and signed the consent form and filled out the demographics questionnaire. The research staff briefly explained the VR equipment and the user’s objective in the game. The research staff helped the users to wear the head presented in Table 1 were obtained. Effect of clutter on Average Score mounted display and the hand bands, and a user performance was statistically significant when 6.82 familiarization session began. The aim of the there was no motion. Plot of the means of scores for 8 5.91 familiarization session was to make the users the clutter and the motion variables are presented in 6 6.96 comfortable with the VR system and the game Figure 5. As it can be observed in Figure 5, change in 5.56 4 mechanics. The familiarization session included one the clutter variable creates a significant change in the 2 display and one shelf with the no clutter - no motion mean score. Presence of clutter worsens the score. Motion No Motion condition. The familiarization session ended when the However, no significant effect of motion on the score 0 No Clutter user stated that they were comfortable with the VR was observed. Clutter system, which took 53 seconds on average. The experiment then began. The users were randomly Condition N df t-stat p Figure 4: Bar charts of the scores assigned with a condition and then presented with Clutter - Motion, for different conditions. three instances of that condition. After the users 15 14 -2.027 0.062 No Clutter - Motion completed one condition’s three instances, they were Clutter - No Motion, given a survey. Then, they were assigned with another 15 14 -3.883 0.002 No Clutter - No Motion condition and completed three instances with that Clutter - Motion, condition that was followed by a survey. After the users 15 14 0.898 0.384 Clutter - No Motion completed the four condition’s three instances (12 No Clutter - Motion, instances in total), the experiment ended and the 15 14 -0.370 0.717 No Clutter - No Motion research staff helped the users to take off the worn equipment. The labels, the boxes in the scene and the Table 1: Paired sample t-tests for the score data. distribution of the boxes on the shelves were assigned randomly for each instance to eliminate any bias and Survey Data learning effect. At the end of each condition’s trial of three instances, we asked each user to give a score for the following Results variables: the perceived difficulty of the task, level of Performance Data frustration, ease of finding the boxes, level of Average scores for the four conditions are presented in distraction, feeling of being limited (restricted), ease of Figure 4. As we analyzed the data for the effects of concentration, presence and motion sickness, within a conditions on score using two way ANOVA with Likert scale of 5 to 1 (5: very much, 1: not at all). The repeated measures with α = 0.05 and Bonferroni presence questions were from the Witmer and Singer’s correction; for clutter/no-clutter variable there was questionnaire [13] and the motion sickness questions Figure 5: Plot of the means of statistically significant difference (F(1, 14) = 14.259, p were from the questionnaire of Gianaros et al. [5]. scores for the clutter and the Results of these survey variables are presented in = 0.002), for motion/no-motion variable there was no motion variables. statistically significant difference (F(1, 14) = 0.177, p = Figure 6. Error bars represent the standard error of the 0.680). As paired t-tests were performed, the results mean. Two way ANOVA with repeated measures with α = 0.05 and Bonferroni correction resulted in statistical Qualitative Survey Results significance only for the following variables: clutter In the surveys, there were open ended questions about variable for feeling of being limited (F(1, 14) = 7.977, p what the users liked the most and the least about the = 0.014) and clutter variable for ease of concentration game, and if they had any additional comments or (F(1, 14) = 5.091, p = 0.041). As paired t-tests were suggestions. There were a lot of positive comments performed, results in Table 2 and Table 3 were about how fun and realistic the VR game was. User 1: obtained respectively. For the limitation, effect of “Very realistic. Boxes seemed like they were there.” clutter was statistically significant in both the presence User 13: “It is fun and interactive. Also going against and the absence of motion. For the ease of the time makes it challenging.” User 18: “Overall, I concentration, effect of clutter was statistically thought this was a lot of fun. I liked how real it felt and significant only for the absence of motion. the difficulty of it.” Condition N df t-stat p A lot of users stated positive comments about the clutter. For the clutter - no motion condition, the Clutter - Motion, 15 14 2.168 0.048 following comments were made: User 5: “I think this No Clutter - Motion one has more decorations on the shelf. I feel this Clutter - No Motion, 15 14 2.168 0.048 makes it look more real.” User 10: “I liked the mess.” No Clutter - No Motion User 17: “I liked the extra materials added.” User 18: Clutter - Motion, 15 14 0.619 0.546 “I liked how there were other things around the room Clutter - No Motion making it more difficult to find the boxes.” For the No Clutter - Motion, 15 14 0.695 0.499 clutter - motion condition, the following comments were No Clutter - No Motion made: User 2: “I liked all the detail.” User 5: “The Table 2: Paired sample t-tests for the limitation. scene looks very real.” User 20: “The boxes were usually easy to spot and the environment seemed like Condition N df t-stat p one I would find in the real world.” Clutter - Motion, 15 14 -1.323 0.207 A few users stated negative comments about the No Clutter - Motion Clutter - No Motion, distraction caused by the clutter: User 1: “Clutter (tools 15 14 -2.168 0.048 etc.) made the task difficult to focus on the boxes. The No Clutter - No Motion distractions made the task significantly more difficult.” Clutter - Motion, 15 14 -0.323 0.751 User 13: “The mess was the most distracting thing. Clutter - No Motion No Clutter - Motion, 15 14 -1.000 0.334 No Clutter - No Motion Table 3: Paired sample t-tests for the ease of concentration. 5 4 3 2 1 Difficulty of Frustration Ease of Distraction Limitation Ease of Presence Motion the Task Finding Concentration Sickness Clutter Motion Clutter NoMotion NoClutter Motion NoClutter NoMotion Figure 6: Bar charts for the average scores of the survey variables for different conditions. Discussion augmented reality, which may be another contributing The user study results and the statistical analysis factor for the difference in the results. indicated that clutter effected user performance negatively, supporting H1. This aligns with the findings A few users stated that they perceived that the shelves in [8] and [1]. Clutter also made it significantly difficult were farther away in the conditions with no-clutter. for the users to concentrate. Although clutter worsened Although the layout was the same between all four the user performance, a lot of users stated that the conditions, lack of clutter made the virtual environment presence of clutter made the game seem more real and seem larger. This might be incorporated into game interesting. Hence, the addition of clutter may increase design when the scene is desired to look spacious. the level of difficulty in VR games whereas contributing to the naturalness and richness of the virtual On a different note, some users stated that when the environment. Clutter made the users feel more limited label to search for was a brand they liked or they were inside the virtual world. In VR games where a feeling of familiar with, the task was more fun to perform. restriction/limitation is desired such as confined places, clutter may be utilized. No significant effect of motion In light of these results, when task performance isn’t on user performance were observed, rejecting H2. This important in the designed game, extra elements may does not align with the results found in [4]. However, be added in the scene to give the users a more realistic the task in [4] was tracking the particles by constantly and more interesting game environment. However, in watching and estimating their velocity changes. The serious games where the task performance is task in our study was searching, which may be the important, it might be a better practice to keep the reason behind this difference. Also, medium of [4] was virtual environment empty at first, eliminating any distractions and then gradually inducing clutter and dynamism as the user gets trained. This way, the users References would be better prepared for real world conditions 1. Felipe Bacim, Eric Ragan, Siroberto Scerbo, without getting overwhelmed. Nicholas F. Polys, Mehdi Setareh, and Brett D. Jones. 2013. The effects of display fidelity, visual complexity, and task scope on spatial Limitations understanding of 3D graphs. In Proceedings of This study focuses on virtual reality games. The results Graphics Interface 2013 (GI '13). Canadian may or may not be applicable for games for other Information Processing Society, Toronto, Ont., mediums such as computer or console. It should also Canada, 25-32. be noted that this study was based on a searching task, 2. Herbert H. Bell and Wayne L. Waag. 1998. which shaped the results and their interpretation. Age Evaluating the Effectiveness of Flight Simulators for of the participants may be another limiting factor in Training Combat Skills: A Review. The International applying the conclusions of the study for the children or Journal of Aviation Psychology, 8, 3, 223-242. http://dx.doi.org/10.1207/s15327108ijap0803_4. elderly population groups. 3. Frederick P. Brooks. 1999. What's Real About Virtual Reality? IEEE Comput. Graph. Appl. 19, 6, Conclusions and Future Work 16-27. http://dx.doi.org/10.1109/38.799723. In this study, effects of clutter and motion on user 4. Vicente Ferrer, Yifan Yang, Alex Perdomo, and John performance in a virtual reality searching game was Quarles. 2013. Consider your clutter: Perception of examined in the search of more effective virtual virtual object motion in AR. IEEE International environment design. A two by two user study was Symposium on Mixed and Augmented Reality performed with 15 college aged participants. Results (ISMAR), Adelaide, SA, pp. 1-6. indicated that clutter effected user performance 10.1109/ISMAR.2013.6671835 negatively, made the users feel more limited and made 5. Peter J. Gianaros, Eric R. Muth, J. Toby Mordkoff, it more difficult to concentrate. No effect of motion on Max E. Levine, and Robert M. Stern. 2001. A user performance was observed. We believe these Questionnaire for the Assessment of the Multiple results will provide insight into the design of future Dimensions of Motion Sickness. Aviation, space, and environmental medicine, 72, 2, 115-119. virtual environments that will lead to more user friendly everyday VR experiences. 6. T. P. Grantcharov, V. B. Kristiansen, J. Bendix, L. Bardram, J. Rosenberg, and P. Funch-Jensen. 2004. Randomized clinical trial of virtual reality Future work may include evaluating different levels of simulation for laparoscopic skills training. British clutter and dynamism, or other environmental Journal of Surgery, 91, 2, 146-150. properties on user experience in virtual environments. http://dx.doi.org/10.1002/bjs.4407. 7. Kelly S. Hale and Kay M. Stanney. 2014. Handbook Acknowledgements of virtual environments: Design, implementation, The authors thank the Florida Department of Education and applications. CRC Press. for funding the VR4VR project, under which this study was performed. 8. Eric D. Ragan, Doug A. Bowman, Regis Kopper, Cheryl Stinson, Siroberto Scerbo, and Ryan P. McMahan. 2015. Effects of Field of View and Visual Complexity on Virtual Reality Training Effectiveness for a Visual Scanning Task. In IEEE Transactions on Visualization and Computer Graphics, vol. 21, no. 7, pp. 794-807. 10.1109/TVCG.2015.2403312 9. Rift. Oculus. Retrieved April 13, 2016 from https://www.oculus.com/en-us/rift/ 10. Jari Takatalo, Jukka Häkkinen, Jyrki Kaistinen, and Gote Nyman. 2010. Presence, Involvement, and Flow in Digital Games. Evaluating User Experience in Games: Concepts and Methods. R. Bernhaupt. London, Springer London: 23-46. 11. Vive. HTC. Retrieved April 13, 2016 from https://www.htcvive.com/us/product/ 12. Vr2200. Virtual Realities Llc. Retrieved April 13, 2016 from http://www.vrealities.com/products/featured/vr220 0 13. Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3, 225-240. http://dx.doi.org/10.1162/105474698565686