Using a VR Field Study to Assess the Eects of Visual and Haptic Cues in "In-the-Wild" Locomotion Ana de Oliveira Mohamed Khamis Augusto Esteves ana.lipa.oliveira@tecnico.ulisboa.pt mohamed.khamis@glasgow.ac.uk augusto.esteves@tecnico.ulisboa.pt Instituto Superior Técnico, University School of Computing Science, ITI / LARSyS, Instituto Superior of Lisbon University of Glasgow Técnico, University of Lisbon Lisbon, Portugal Glasgow, UK Lisbon, Portugal Figure 1: Two screenshots from our VR scene. On the left we have highlighted the several distractors put in place to assess participants awareness of them across the cue conditions: (a) a passing car; (b) a pedestrian; and (c) a crossing light. On the right we present a closer look to stimulus in the visual cue condition. ABSTRACT KEYWORDS This work aims to assess the eect of visual and haptic cues in Parkinson’s Disease, Gait, Visual cues, Haptic Cues, Virtual-reality, users with gait impairments; not only in performance, but also Virtual Field Study, Usability, Attention, Eye-tracking in terms of usability, perceived cognitive load, and safety. These haptic cues were delivered via wrist-worn devices, with the goal Reference Format: of supporting these users while out in-the-wild – three types of Ana de Oliveira, Mohamed Khamis, and Augusto Esteves. 2020. Using a VR haptic cues were tested. To further assess the impact of haptic and Field Study to Assess the Ef ects of Visual and Haptic Cues in "In-the-Wild" visual cues outside of a laboratory environment, we used a Virtual Locomotion. In Cross-Reality (XR) Interaction, ACM ISS 2020 (International Reality Field Study to safely assess the impact of these cues in Workshop on XR Interaction 2020). users’ awareness of their surroundings (measured via gaze hits and dwell). Despite conducting a preliminary study with participants not suering from gait impairments (N=6), our results seem to 1 CROSS-REALITY INTERACTION indicate a positive eect of the haptic cues in regards to participant We agree with Speicher et al.’s expectation that the distinctions cadence, step length, and general awareness of their surroundings between AR and VR will fade away in time [24]. In that sense, when compared to the visual cue. One of the simpler haptic cues we see Cross-Reality Interaction not so much as a system-centred was also the preferred stimulus by all participants. series of in-app transitions across the Reality-Virtuality continuum [12], but as user- or experience-centred transitions. That is, how CCS CONCEPTS can we build mixed-reality systems that enable users to seamlessly • Applied computing  Health informatics; • Human-centered transition their attention between digital content and the physical computing  Empirical studies in accessibility; Ubiquitous and world? How can they transition from ready-at-hand and present- mobile computing systems and tools. at-hand operations when interacting with mixed-reality tools [4]? How can users ooad cognitive processes to a blend of digital and physical spaces? In that regard, the work we present in this paper focuses on a small subset of those transitions: how can we Copyright © 2020 for this paper by its authors. Use permied under Creative model and study real world behavior via a VR experience? How Commons License Aribution 4.0 International (CC BY 4.0). Cross-Reality (XR) Interaction, ACM ISS 2020, November 8 2020, Lisbon, Portugal can we transition abstractions, data, observations, and ultimately knowledge across these realities? We use this premise to study the eect of various cues in users’ gait, relying on VR to safely simulate a variety of competing stimulus that can aect users’ performance with these during in-the-wild locomotion. International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Ana de Oliveira, et al. 2 INTRODUCTION AND RELATED WORK but the usability, perceived cognitive load, and safety of these types Gait disorders, which greatly contribute to a decrease in quality of systems. of life and increased mortality, are common and often devastating companions of the ageing process [2]. These disorders increase 3 USER STUDY from around 10% between the ages of 60 and 69 years, to more than 60% in those over 80 years of age [10]. Age is not the only source of 3.1 Participants these impairments, as strokes, Parkinson’s disease, myelopathy, or Mostly due to COVID-19 constraints, our preliminary study relies sensory ataxia are some of the most known and studied neurological on six patients without any gait impairments. Except for one, these conditions with repercussions in patients’ gait [18]. were aged between 18 and 25 years of age (M = 27.0; SD = 11.52); Our work was primary motivated by Parkinson’s disease, the and the majority were students (66.6%). Using a 5-point Likert second most common neurodegenerative disorder that a�ects over scale, participants reported being somewhat comfortable with VR 10 million people all over the world [20]. As the disease progresses technologies (M = 2.00; SD = 1.10). All participants had experience many are the e�ects in patients’ ability to walk: their gait pattern with smartwatches prior to this study. becomes usually characterized by a shortened gait stride, their walking speed is reduced, their gait variance is increased, and they can be a�ected by what is known as festinating gait [7]. As there 3.2 Experimental Setup is no cure or treatment that completely addresses the e�ect of This study was performed in a hallway 1m wide and 6.5m long. We Parkinson’s disease on gait, these symptoms can be minimized with relied on VR to simulate a street environment where participant lifestyle changes and physiotherapy. Another approach is what is walked in a straight line along a 5m long sidewalk. Several events known as cueing. were included (described as distractors) such as a passing car, a Cueing consists of sensory spatial and temporal stimulus that pedestrian that would start walking, and crossing light that would have been shown to minimize the e�ect of Parkinson’s disease in change from red to green (see Figure 1 – left). These events took users’ gait [1, 8, 17, 25, 27]. Visual and auditory stimuli are the most place after participants walked 1.5, 2.5, and 3.5m, respectively. This used and studied types of cues to this e�ect. And although many was developed using the Unity Game Engine, and deployed on an studies have demonstrated that these two types of cues are quite HCT Vive Pro Eye head-mounted display (combined resolution of e�ective in normalizing patient’s gait parameters – respectively, 2880×1600 px, 615 PPI, 90Hz, 110° FoV) and eye-tracker (120Hz, spatial (step length and stride length) [1, 6, 14, 25, 30] and temporal 0.5° 1.1° accuracy). Finally, the haptic cues were played on two parameters (velocity and cadence) [5, 8, 9, 11, 27] – very few studies Huawei Watch 2 and controlled through an Android application exist that demonstrate the e�ect and usability of these systems where the researcher started and stopped the cues and the VR simu- outside of a controlled environment (i.e., a research laboratory). lation. The communication between these devices was done via the That is, very few studies explore these cues while the users are Open Sound Control (OSC) protocol, and the study complied with out in-the-wild, where they need to engage in simple tasks such the ethics guidelines and COVID-19 regulations in our institution. as walking through a crosswalk – a task that requires undivided attention and concentration [23]. In fact, recent studies show that 3.3 Experimental Design texting, talking on a smartphone, sur�ng the web, or playing games negatively a�ects the safety of pedestrians while crossing the road Our study followed a within participants design counterbalanced [15, 16, 26]. These distractions have been proven to be even more using a Latin square. It included four cue conditions: problematic and di�cult for Parkinson’s patients [13, 22]. Visual. This followed a classic approach [25] where bright trans- In this paper we propose to focus particularly on haptic cues. verse bars 45cm wide were displayed on the �oor covering the entire These types of cues have been demonstrated to be less cognitively scene (see Figure 1 – right). The distance between bars varied be- taxing than visual stimuli in navigation tasks, and can be provided tween participants to match 150% of their baseline step length [1]. to users in the less distracting and more private form factor of Haptic (one pattern, one watch [1P1W]). Another classic a wrist-worn device such as a smartwatch or �tness tracker; ul- cue that uses a simple vibration pattern at speci�c intervals [29]. timately leading to a system that is more feasible for continued This was played on the participant’s wrist, and provided them use out in-the-wild. Haptic cues have been explored brie�y in the with a rhythmic stimulus. The temporal property of this stimulus past, demonstrating improvements in users’ posture [31], balance varied between participants in order to match the cadence measured [19], and gait [17, 21, 28]. We propose to expand this work in the during the baseline trial with no stimulus (we follow this rationale following ways. First, we propose the study of three distinct haptic for the remaning two haptic cues). cues against a visual baseline. These were designed to explore both Haptic (one pattern, two watches [1P2W]). This designed temporal and spatial properties of these cues – the latter using this cue to explore the idea of playing the haptic pattern above two wrist-worn devices mapped to left and right steps. Second, alternatively over two smartwatches, placed on participants’ left we will conduct our study in a simulated street environment in and right wrists. This would provide participants with a rhythm virtual-reality (VR), enabling us to measure participants’ engage- with temporal and spatial properties (left and right). ment with various points-of-interest in the scene via gaze data (hits Haptic (two patterns, one watch [2P1W]). Two distinct vi- and dwell). In sum, the goal of our work is to assess the e�ect of bration patterns were played in sequence on a single smartwatch, visual and haptic cues not only in participants’ gait performance, attempting to explore the temporal and spatial properties of [1P2W] using a single device. Using a VR Field Study to Assess Visual and Haptic Cues in "In-the-Wild" Locomotion International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal 0 6 0.4 Visual Haptic [1P1W] Haptic [1P2W] Haptic [2P1W] -2 Mean △ cadence [steps/min] 4 Mean △ Step Length [cm] Mean △ Velocity [m/s] -4 3.74 0.2 -6.28 2 -6 2.10 0.14 -8 0 0 -10 -2 -0.04 -0.03 -0.06 -0.07 -12 -12.91 -13.07 -13.98 -4 -14 -0.2 -6 -16 -5.72 -18 -8 -0.4 Figure 2: Results for cadence (lower is better, left), step length (higher is better, center), and velocity (right). These represent the mean delta to each participant’s baseline results. 3.4 Metrics Table 1: SUS results across conditions (std. dev. in brackets). In order to understand the e�ects of the cues and distractors on participants’ gait and experience, we measured: Visual Haptic [1P1W] Haptic [1P2W] Haptic [2P1W] Performance. This included participants’ cadence (steps per 49.16 (3.76) 85.41 (4.59) 91.66 (6.07) 62.50 (3.02) min.), step length, and velocity (meters per second). This was cal- culated by visually counting the number of steps in a trial, and by automatically recording how long it took participants to reach the 4.1 Performance end of the trial (�ve meters). We emphasize that our goal is to improve users’ gait, i.e., have Usability. Participants completed the System Usability Scale them produce less but longer steps (as opposed to, e.g., the small (SUS) [3] for each cue, and a preference questionnaire at the end shu�ing steps seen with Parkinsonian gait). Despite not having of the study. In the latter they were asked to comment on they any gait impairments, our participants’ seem to have been able to favourite and least favourite cues. improve their cadence and step length in the majority of the haptic Gaze. In order to assess participants’ awareness of the three conditions (see Figure 2 – left and center), while completing the distractors included in the scene, we measured the number of gaze trial in the approximately same amount of time as with no stimuli hits and dwell time on these across cue conditions. (see Figure 2 – right). 3.5 Procedure 4.2 Usability The study was conducted in a empty and quiet hallway. Partici- The SUS results for each of the conditions is seen in Table 1. This pants were asked to properly disinfect their hands with an 70% highlights a preference for the haptic cues relying on a simple vibra- alcohol solution, and to clean their face and wrists with disinfecting tion pattern played over one or two smartwatches (well above the wipes. This was followed by collecting participants’ demographic average SUS score of 68). These results are further corroborated by information in addiction to to previous experience with VR and the preference rankings. All participants’ agreed their favourite cue smartwatches. was the Haptic [1P1W], mostly due to its simple nature requiring Afterwards, we asked participants to put on both smartwatches, very little attention; and all agreed the visual cue to be their least one on each wrist, and to adjust them so they were tight and com- favourite as it required them to continuously look at the �oor, often fortable. This was followed by the setup and calibration of the VR loosing track of their surroundings. headset and eye-tracker. The study started by a trial with no stim- uli, where baseline measures of participants’ gait parameters were 4.3 Gaze captured (i.e., cadence, step length, and velocity) and feed into the The gaze results can be seen in Figure 3. These seem to suggest system for personalized stimuli. Participants were asked to walk participants were quite aware of their surroundings in both the in a straight line towards the crossing light at the end of the scene baseline (no stimuli) and haptic conditions. As expected, the visual (5m), and that the trial would stop when they were close to reaching condition wielded a potentially lower number of gaze hits and it. Finally, at the end of each condition participants completed the dwell times across distractors – some of these are zero or close to SUS and took a small break. zero, indicating some participants were not aware of some of these At the end of the study participants completed the preference distractors at all. While further studies are required, this highlights questionnaire. The researcher completed the session by following how impractical and potentially dangerous is this well-studied cue thoroughly cleaning the headset and watches with disinfecting outside of a controlled laboratory environment. wipes with at least 70% alcohol. 5 LIMITATIONS AND FUTURE WORK Our immediate future work includes expanding the number of par- 4 RESULTS ticipants in our study, and following-up with participants with some Below we present our preliminary results from six participants. form of gait impairment (particularly participants with Parkinson’s International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal Ana de Oliveira, et al. 12 Parkinson’s Disease Patients. Physiotherapy 77, 6 (June 1991), 415–420. https: Baseline Visual Haptic [1P1W] Haptic [1P2W] Haptic [2P1W] //doi.org/10.1016/S0031-9406(10)62035-4 10 [2] Bastiaan R. Bloem, Joost Haan, Anne M. Lagaay, Wim van Beek, Axel R. Wintzen, and Raymund A. C. Roos. 2016. Investigation of Gait in Elderly Subjects Over 88 Years of Age:. Topics in geriatrics (Sept. 2016). https://doi.org/10.1177/ Mean Hits [count] 8 002383099200500204 Publisher: SAGE PublicationsSage CA: Los Angeles, CA. 7.7 [3] John Brooke. 1996. SUS: a “quick and dirty’usability. Usability evaluation in 6 industry (1996), 189. 6.5 6.5 [4] Paul Dourish. 2004. Where the action is: the foundations of embodied interaction. 5.5 MIT press. 4 4.5 4.5 4.7 [5] Robert L Freedland, Carmel Festa, Marita Sealy, Andrew McBean, Paul Elghazaly, 3.8 4.0 Ariel Capan, Lori Brozycki, Arthur J Nelson, and Je�rey Rothman. [n.d.]. The 3.2 e�ects of pulsed auditory stimulation on various gait measurements in persons 2 2.2 with Parkinson’s Disease. ([n. d.]), 7. 1.30.01.5 1.0 [6] Luis I. Gómez-Jordana, James Sta�ord, C. (Lieke) E. Peper, and Cathy M. Craig. 0 2018. Virtual Footprints Can Improve Walking Performance in People With Car Pedestrian Crossing Light Parkinson’s Disease. Frontiers in Neurology 9 (2018). https://doi.org/10.3389/ fneur.2018.00681 6 [7] Je�rey M. Hausdor�, Merit E. Cudkowicz, Renée Firtion, Jeanne Y. Wei, and Ary L. Goldberger. 1998. Gait variability and basal ganglia disorders: Stride-to-stride variations of gait cycle timing in parkinson’s disease and Huntington’s disease. 5 Movement Disorders 13, 3 (1998), 428–437. https://doi.org/10.1002/mds.870130310 Mean Dwell Time [s] [8] Je�rey M. Hausdor�, Justine Lowenthal, Talia Herman, Leor Gruendlinger, Chava 4 4.5 Peretz, and Nir Giladi. 2007. Rhythmic auditory stimulation modulates gait variability in Parkinson’s disease. European Journal of Neuroscience 26, 8 (2007), 3.8 3 2369–2375. https://doi.org/10.1111/j.1460-9568.2007.05810.x 3.1 [9] T E Howe, B Lövgreen, F Wj Cody, V J Ashton, and J A Oldham. 2003. Auditory 2.8 2.7 cues can modify the gait of persons with early-stage Parkinson’s disease: a method 2 2.3 for enhancing parkinsonian walking performance? Clinical Rehabilitation 17, 4 2.0 (June 2003), 363–367. https://doi.org/10.1191/0269215503cr621oa 1.5 1.7 1.6 [10] Philipp Mahlknecht, Stefan Kiechl, Bastiaan R. Bloem, Johann Willeit, Christoph 1 1.3 1.4 Scher�er, Arno Gasperi, Gregorio Rungger, Werner Poewe, and Klaus Seppi. 0.7 0.1 0.0 2013. Prevalence and burden of gait disorders in elderly men and women aged 0 60-97 years: a population-based study. PloS One 8, 7 (2013), e69627. https: Car Pedestrian Crossing Light //doi.org/10.1371/journal.pone.0069627 [11] G C McIntosh, S H Brown, R R Rice, and M H Thaut. 1997. Rhythmic auditory- motor facilitation of gait patterns in patients with Parkinson’s disease. Journal Figure 3: Top: mean gaze hits across conditions (and base- of Neurology, Neurosurgery & Psychiatry 62, 1 (Jan. 1997), 22–26. https://doi.org/ 10.1136/jnnp.62.1.22 line) for each of the three distractors. Bottom: mean gaze [12] Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual dwell results. displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321– 1329. [13] Meg Morris, Robert Iansek, Fiona Smithson, and Frances Huxham. 2000. Postural instability in Parkinson’s disease: a comparison with and without a concurrent disease) – the ultimate stakeholders of such a system. We will con- task. Gait & Posture 12, 3 (Dec. 2000), 205–216. https://doi.org/10.1016/S0966- sider running these studies in a wider space, as we suspect our 6362(00)00076-X [14] Meg E. Morris, Robert Iansek, Thomas A. Matyas, and Je�ery J. Summers. 1996. narrow hallway might have a�ected participants’ gait at points (for Stride length regulation in Parkinson’s disease: Normalization strategies and fear of hitting a wall). Ultimately, we aim to develop a smartwatch underlying mechanisms. Brain 119, 2 (1996), 551–568. https://doi.org/10.1093/ brain/119.2.551 application for in-the-wild haptic cues that not only aims to nor- [15] Jack Nasar, Peter Hecht, and Richard Wener. 2008. Mobile telephones, distracted malize users’ gait but is comfortable and safe to use outside of a attention, and pedestrian safety. Accident Analysis & Prevention 40, 1 (Jan. 2008), laboratory. We also suggest a replication of our study via a standard 69–75. https://doi.org/10.1016/j.aap.2007.04.005 [16] Jack L. Nasar and Derek Troyer. 2013. Pedestrian injuries due to mobile phone �eld study in order to compare �ndings. This would enable us to use in public places. Accident Analysis & Prevention 57 (Aug. 2013), 91–95. https: further validate virtual �eld studies as a novel research paradigm, //doi.org/10.1016/j.aap.2013.03.021 particularly in the context of locomotion and mobility tasks. [17] Peter Novak and Vera Novak. 2006. E�ect of step-synchronized vibration stimula- tion of soles on gait in Parkinson’s disease: a pilot study. Journal of NeuroEngineer- ing and Rehabilitation 3, 1 (May 2006), 9. https://doi.org/10.1186/1743-0003-3-9 6 CONCLUSION [18] Walter Pirker and Regina Katzenschlager. 2017. Gait disorders in adults and the elderly. Wiener klinische Wochenschrift 129, 3 (Feb. 2017), 81–95. https: This paper presented a work-in-progress where we explored pop- //doi.org/10.1007/s00508-016-1096-4 ular and new stimuli to normalize users’ gait in the context of a [19] Ely Rabin, Jason Chen, Lisa Muratori, Joanne DiFrancisco-Donoghue, and William G. Werner. 2013. Haptic feedback from manual contact improves balance VR Field Study. The latter was employed so that we could explore control in people with Parkinson’s disease. Gait & posture 38, 3 (July 2013), the impact of these cues while walking in a simulated sidewalk; 373–379. https://doi.org/10.1016/j.gaitpost.2012.12.008 allowing us to start to assess not only the impact of these cues in the [20] Sridevi V. Sarma, Ming L. Cheng, Uri T. Eden, Ziv Williams, Emery N. Brown, and Emad Eskandar. 2012. The e�ects of cues on neurons in the basal ganglia overall user experience, but their safety outside of a controlled lab. in Parkinson’s disease. Frontiers in Integrative Neuroscience 6 (2012). https: environment (measured via gaze and awareness of several events). //doi.org/10.3389/fnint.2012.00040 [21] Ervin Sejdić, Yingying Fu, Alison Pak, Jillian A. Fairley, and Tom Chau. 2012. The We aim to expand this work via a wearable application that can E�ects of Rhythmic Sensory Cues on the Temporal Dynamics of Human Gait. help address gait disorders in a comfortable, usable, and safe way. PLOS ONE 7, 8 (Aug. 2012), e43104. https://doi.org/10.1371/journal.pone.0043104 [22] M. H. Sharpe. 1996. Is There a Divided Attention De�cit in Patients with Early Parkinson’s Disease? Cortex 32, 4 (Dec. 1996), 747–753. https://doi.org/10.1016/ REFERENCES S0010-9452(96)80044-X [1] Susan Bagley, Barbara Kelly, Nancy Tunnicli�e, George I Turnbull, and Joan M [23] Anae Sobhani and Bilal Farooq. 2018. Impact of smartphone distraction on Walker. 1991. The E�ect of Visual Cues on the Gait of Independently Mobile pedestrians’ crossing behaviour: An application of head-mounted immersive Using a VR Field Study to Assess Visual and Haptic Cues in "In-the-Wild" Locomotion International Workshop on XR Interaction 2020, November 8 2020, Lisbon, Portugal virtual reality. Transportation Research Part F: Tra�c Psychology and Behaviour [28] E. van Wegen, C. de Goede, I. Lim, M. Rietberg, A. Nieuwboer, A. Willems, D. 58 (Oct. 2018), 228–241. https://doi.org/10.1016/j.trf.2018.06.020 Jones, L. Rochester, V. Hetherington, H. Berendse, J. Zijlmans, E. Wolters, and G. [24] Maximilian Speicher, Brian D. Hall, and Michael Nebeling. 2019. What is Mixed Kwakkel. 2006. The e�ect of rhythmic somatosensory cueing on gait in patients Reality?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing with Parkinson’s disease. Journal of the Neurological Sciences 248, 1-2 (Oct. 2006), Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, 210–214. https://doi.org/10.1016/j.jns.2006.05.034 New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300767 [29] E. van Wegen, I. Lim, C. de Goede, A. Nieuwboer, A. Willems, D. Jones, L. [25] M. Suteerawattananon, G.S. Morris, B.R. Etnyre, J. Jankovic, and E.J. Protas. Rochester, V. Hetherington, H. Berendse, J. Zijlmans, E. Wolters, and G. Kwakkel. 2004. E�ects of visual and auditory cues on gait in individuals with Parkinson’s 2006. The e�ects of visual rhythms and optic �ow on stride patterns of patients disease. Journal of the Neurological Sciences 219, 1-2 (April 2004), 63–69. https: with Parkinson’s disease. Parkinsonism & Related Disorders 12, 1 (Jan. 2006), //doi.org/10.1016/j.jns.2003.12.007 21–27. https://doi.org/10.1016/j.parkreldis.2005.06.009 [26] Hagai Tapiro, Tal Oron-Gilad, and Yisrael Parmet. 2016. Cell phone conversations [30] Sonya Weissenborn. 1993. The E�ect of Using a Two-step Verbal Cue to a Visual and child pedestrian’s crossing behavior; a simulator study. Safety Science 89 Target Above Eye Level on the Parkinsonian gait: A case study. Physiotherapy (Nov. 2016), 36–44. https://doi.org/10.1016/j.ssci.2016.05.013 79, 1 (Jan. 1993), 26–31. https://doi.org/10.1016/S0031-9406(10)60537-8 [27] M. H. Thaut, G. C. McIntosh, R. R. Rice, R. A. Miller, J. Rathbun, and J. M. Brault. [31] Ying Zheng and John B. Morrell. 2010. A vibrotactile feedback approach to 1996. Rhythmic auditory stimulation in gait training for Parkinson’s disease posture guidance. In 2010 IEEE Haptics Symposium. 351–358. https://doi.org/10. patients. Movement Disorders 11, 2 (1996), 193–200. https://doi.org/10.1002/mds. 1109/HAPTIC.2010.5444633 ISSN: 2324-7347, 2324-7355. 870110213