Tangible interactions in Virtual Reality environments Matteo Sirizzottia , Simone Guerciob , Flavio Lampusa , Patrizia Martia , Luca Lusuardia and Alessandro Innocentia a Department of Social, Political and Cognitive Sciences, University of Siena, via Roma 56, Siena, Italy b Santa Chiara Fab Lab, University of Siena, via Valdimontone 1, Siena, Italy Abstract Virtual Reality (VR) provides opportunity for immersive experiences in several fields of application, in particular in training and entertainment. Integrating tangible interaction with virtual environments that mimic real world situations can provide a tremendous enrichment to the user experience. Indeed, one of the main barriers to immersivity virtual objects is the use of fixed controllers with poor affordance. Touching, grasping, pushing, squeezing, if executed with common controllers are unnatural, impractical and sometimes frustrating. The paper presents the outcomes of a multidisciplinary panel conducted during ETIS 2020. The purpose of the panel was to engage a reflection on possible ways to integrate tangible interactions in virtual reality scenarios. To this extent, a case study was presented describing the development of a physical controller used in a virtual simulation of a DNA extraction procedure in a biotechnology laboratory. The case study shows the potential of using a real pipette to make the extraction task easy to perform in VR exploiting the affordance of the real object when gripping, pushing, moving the pipette. This opens new possibilities for enhanced multisensory interactions in VR environments. Keywords Virtual Reality, Tangible interactions, Immersivity, 1. Introduction Virtual Reality (VR) is nowadays wide-spreading since several commercial products are available for reasonable prices. VR is exploited in several domains: from design and manufacturing process [1], to treatment of phobia and other psychological disorders [2][3]. Education and training is one of the most relevant domains of application. Especially in critical situations and risky procedures, it is possible to train operators in safe virtual environments so that the experience gained in VR can be transferable to the real world. Sound training applications of VR in the medical field are described in [4][5][6], while an overview of applications of experiential learning for construction workers is provided by [7][8]. Another interesting field of application is cultural heritage, where VR can enhance the museum visitor experience, acting as a time machine to immerse the visitors into other places and times [9]. Proceedings of ETIS 2020, November 16–-20, 2020, Siena, Italy Envelope-Open matteo.sirizzotti@gmail.com (M. Sirizzotti); s.guercio@gmail.com (S. Guercio); flavio.lampus@unisi.it (F. Lampus); patrizia.marti@unisi.it (P. Marti); lucarma79@gmail.com (L. Lusuardi); alessandro.innocenti@unisi.it (A. Innocenti) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) To improve the effectiveness of VR applications, on one hand, the effort is towards the design of high fidelity virtual environments. The realism of the environment relies not only on the visual reconstruction of the environment itself but also on the multisensory stimulation which can be provided, including haptic feedback to perceive the forces and the material characteristics of objects [10]. For example, ElasticVR [11] is a wearable device that provides force-feedback through an elastic band, servo motors and mechanical locks. It is able to provide both passive force-feedback (e.g. simulating resistance and reaction force) and active force-feedback (e.g. simulating recoil and impact). TacTiles is a flexible haptic array to provide localized tactile feedback [12], that is able to render continuous contact with objects and the exploration of object surfaces and volumes. Haptic Revolver [13] is a handheld VR controller that renders fingertip haptics when interacting with virtual surfaces. On the other hand, interactivity and more in particular tangible interaction - that is the possibility to manipulate objects and the ability of the environment to react to user’s action - is regarded to be a fundamental factor compared to the realism of the environment [14][15][16]. The current design trend is focused on intuitive interfaces by introducing natural interaction through gesture recognition and hand-tracking to interact with virtual objects (see for example: Oculus Blog, 2019). While commercial VR devices are provided with their own controllers, some studies investi- gate new concepts exploiting hand-tracking sensors. Caggianese, Gallo and Neroni [17] compared HTC Vive controllers with Leap Motion Camera in object manipulation tasks (e.g. selection, positioning and rotation). The authors found a better performance and less perceived difficulty of the Vive Controllers thanks to its stability, accuracy, and lower learning curve compared to that required by the Leap Motion sensor. Nevertheless, other difficulties occurred with both the devices, so the authors conclude that “neither of the two devices can be considered as the definitive solution to the problems relating to effective interactions in virtual environment”. CLAW represents an augmentation of the traditional VR controller functionality [18]: it enables different interactions (grasping virtual objects, touching virtual surfaces, and triggering) and it generates vibrations and force-feedback in reaction to user’s action. A brand new concept is proposed by Aguerreche, Duval and Lécuyer [19] who designed the Reconfigurable Tangible Device: it is composed of three or four arms rigidly linked together which can be compressed or stretched, to manipulate the shape and the motion of the virtual objects. The challenge of the proposed panel was to reflect on different concepts of controllers to allow tangible interaction in virtual environments. Participants were engaged in a discussion on future applications of virtual controllers in virtual simulations. The challenge of this panel was to envision future physical controller for VR applications integrating natural, playful, intriguing tangible interactions. 2. Objectives and expected outcomes As said above, one of the main barriers in object manipulation in VR is the lack of realistic haptic feedback and controllers with proper affordance [20] to naturally execute tasks involving touching, squeezing, moving, pushing etc. as it happens in the real world [21]. Improving tangibility in VR offers tremendous opportunities to enrich VR experiences. To face this challenge, the panel engaged participants in the design of new hand-held haptic controllers, acting as input devices in VR applications. The expected outcomes were new concepts of VR controllers to use in virtual environments to manipulate virtual objects. Such new concepts were intended to augment or totally replace the current VR controllers. The panel was based on a case study) developed by the recently inaugurated VR Lab managed by faculty and staff of the Department of Social, Political and Cognitive Science of the University of Siena, and funded under the Project of Excellence granted by the Italian Ministry of the University in 2018 (https://interdispoc.unisi.it/en). The VR Lab is provided with a platform for immersive virtual reality testing equipped with Oculus Quest head mounted displays (HMD from now on), HTC Vive HMDs, and an immersive CAVE room (Cave Automatic Virtual Environment - CAVE) where rear projections of computer generated 3D images are rendered for experimental simulations. 3. Schedule of the panel Table 1 Plan of the panel Duration Activities 1 20 min Introduction to VR technology including research projects and application 2 5 min Break 3 15 min Brainstorming with the participants using Miro virtual whiteboard platform 4 10 min Discussion and reflection about the generated concepts 4. Activity description After a brief introduction focused on the two actually most used technologies in VR applications (HMDs and CAVE rooms) and a reflection about controllers and inputs in VR simulations, a case study was presented to participants. 4.1. Case study: DNA extraction procedure It is actually really hard to mimic natural interaction within a VR experience. The gestures are always mediated by controllers; this results in unnatural and “crippled” interactions (with respect to the real world). Despite the fact that a lot of different types of controllers are developed Figure 1: The pipette used for the DNA extraction task Figure 2: The VR controller by different HMDs producers (HTC, Oculus, Samsung etc.), the interaction possibilities offered are more or less the same: pushbuttons and analog input peripherals (sticks, pads etc.). This kind of input is not suitable when the user has to perform tasks based on interactions with specific devices. Some examples could be: • A surgeon using a lancet • A mechanician using an electrical screwdriver In each of these cases, developing a VR simulation introduces unnatural interactions since the controller does not have the required physical characteristics and affordance of the physical device. Qualities like shape, weight, material affordance, handlings, are completely lost and the entire control is reduced to the use of virtual buttons and input devices which do not mimic the interaction with the physical arttefact. To better explore the issue a case study was presented: a VR simulation of a DNA extraction procedure in a biotechnology laboratory. The simulation, developed by the VR Lab of the University of Siena is currentl under devel- opment and will be used for learning and training activities. Wearing the HMD, the user will be projected into a biotechnology laboratory and will be asked to perform the procedure of extraction of some DNA fragments from a liquid solution. The simulation will las approximately 30 minutes. Figure 1 shows the physical pipette used for the DNA extraction task (left), Figure 2 shows the VR controller (right). The objective of the simulation is to build a custom controller that mimics the physical properties of the real DNA extractor (shape, weight, handling, buttons etc.) and the related interactions to perform the task. 4.2. Brainstorming on Miro Since the panel was conducted in virtual modality, participants used Miro to support the brainstorming. The ultimate goal of the panel was to respond the following question:‘Can a physical controller for VR applications be designed to enhance multisensory experiences in simulation environments in order to increase immersivity?’. In order to answer the question the participants were invited to: • Retrieve an image (from the web) of an object/tool (eg. a bicycle) • List the material properties and possible real multisensory experiences/feelings that can be perceived using that object (considering the bicycle: weight, shape, friction with terrain, wind, speed etc. ) • Imagine and describe a possible scenario and a VR multisensory experience (es: biking by the sea). • Reflect on ways to reproduce the actual experience in VR, integrating multisensory interactions possibilities, the material properties and real feelings in the simulated scenario 4.3. Final discussion and reflections After 20 minutes of work, we gathered the results and started an open discussion about the possibilities presented by the participants trying to cluster similar results in order to produce a conceptual map of the different scenarios. The map was discussed analyzing devices that are actually on the market (eg. Icaros, Tesla Suit, Virtuix Omni etc.) that can be used to recreate immersive multisensory experiences in simulated virtual reality environments. Some interesting examples highlighted during the final analysis, were related to the possibility of integrating the perception of basic sensations useful to understand if the activity is well performed or not; for instance temperature or airflow perception (a glass-blower in front of an anvil relies on his temperature perception to understand if something is wrong; a paraglider or a skydiver relies on his wind perception to perform a correct maneuver). Participants proposed different design concepts to integrate such perceptions in a VR simulation using actual technologies in order to build new controllers/systems (eg. convectors and fans mounted on the HMDs to simulate temperature changes and airflow movements). To wrap up, what emerged from the final reflections is that the actual possibility of interactions offered by the controllers, are still not realistic enough to increase immersivity. The more we step away from the real objects trying to map their material properties into the interactions offered by controllers, the more we decrease immersivity. Having said that, exploring the possibilities offered by a mixed approach that combines physical and virtual objects could result in a definition of criteria and design guidelines that can strongly increase the immersivity level and the engagement of a VR simulation. References [1] A. Jimeno-Morenilla, A. Puerta, State of the art of the virtual reality applied to design and manufacturing processes, International Journal of Advanced Manufacturing Technology 33 (2007). doi:1 0 . 1 0 0 7 / s 0 0 1 7 0 - 0 0 6 - 0 5 3 4 - 2 . [2] C. Botella, S. Quero, R. Baños, C. Perpiñá, A. Garcia-Palacios, G. Riva, Virtual reality and psychotherapy, Studies in health technology and informatics 99 (2004) 37–54. doi:1 0 . 3 2 3 3 / 978- 1- 60750- 943- 1- 37. [3] C. Botella, A. García-Palacios, H. Villa, R. M. Baños, S. Quero, M. Alcañiz, G. Riva, Virtual reality exposure in the treatment of panic disorder and agoraphobia: A con- trolled study, Clinical Psychology & Psychotherapy 14 (2007) 164–175. URL: https: //onlinelibrary.wiley.com/doi/abs/10.1002/cpp.524. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 0 2 / c p p . 5 2 4 . arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpp.524. [4] F. Mantovani, G. Castelnuovo, A. Gaggioli, G. Riva, Virtual reality training for health-care professionals, Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society 6 (2003) 389–95. doi:1 0 . 1 0 8 9 / 1 0 9 4 9 3 1 0 3 3 2 2 2 7 8 7 7 2 . [5] P. B. Andreatta, E. Maslowski, S. Petty, W. Shim, M. Marsh, T. Hall, S. Stern, J. Frankel, Virtual reality triage training provides a viable solution for disaster-preparedness, Acad Emerg Med 17 (2010) 870–876. [6] M. Reznek, P. Harter, T. Krummel, Virtual reality and simulation: Training the future emergency physician, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine 9 (2002) 78–87. doi:1 0 . 1 1 1 1 / j . 1 5 5 3 - 2 7 1 2 . 2 0 0 2 . t b 0 1 1 7 2 . x . [7] R. Sacks, A. Perlman, R. Barak, Construction safety training using immersive virtual reality, Construction Management and Economics 31 (2013) 1005–1017. doi:1 0 . 1 0 8 0 / 01446193.2013.828844. [8] Q. T. Le, A. Pedro, C. Park, A social virtual reality based construction safety educa- tion system for experiential learning, J Intell Robot Syst 79 (2014) 1–20. doi:1 0 . 1 0 0 7 / s10846- 014- 0112- z. [9] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, J. Gain, A survey of augmented, virtual, and mixed reality for cultural heritage, J. Comput. Cult. Herit. 11 (2018). URL: https://doi.org/10.1145/3145534. doi:1 0 . 1 1 4 5 / 3 1 4 5 5 3 4 . [10] E. Gobbetti, R. Scateni, Virtual reality: Past, present and future, Studies in health technology and informatics 58 (1998) 3–20. doi:1 0 . 3 2 3 3 / 9 7 8 - 1 - 6 0 7 5 0 - 9 0 2 - 8 - 3 . [11] H.-R. Tsai, J. Rekimoto, Elasticvr: Providing multi-level active and passive force feedback in virtual reality using elasticity, in: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–4. URL: https://doi.org/10.1145/3170427.3186540. doi:1 0 . 1145/3170427.3186540. [12] V. Vechev, J. Zarate, D. Lindlbauer, R. Hinchet, H. Shea, O. Hilliges, Tactiles: Dual-mode low-power electromagnetic actuators for rendering continuous contact and spatial haptic patterns in vr, in: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 312–320. doi:1 0 . 1 1 0 9 / V R . 2 0 1 9 . 8 7 9 7 9 2 1 . [13] E. Whitmire, H. Benko, C. Holz, E. Ofek, M. Sinclair, Haptic revolver: Touch, shear, texture, and shape rendering on a reconfigurable virtual reality controller, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–12. URL: https://doi.org/10.1145/ 3173574.3173660. doi:1 0 . 1 1 4 5 / 3 1 7 3 5 7 4 . 3 1 7 3 6 6 0 . [14] M. Sanchez-Vives, M. Slater, From presence to consciousness through virtual reality, Nature reviews. Neuroscience 6 (2005) 332–9. doi:1 0 . 1 0 3 8 / n r n 1 6 5 1 . [15] P. Zahorik, R. L. Jenison, Presence as being-in-the-world, Presence: Teleoperators and Virtual Environments 7 (1998) 78–89. URL: https://doi.org/10.1162/105474698565541. doi:1 0 . 1 1 6 2 / 1 0 5 4 7 4 6 9 8 5 6 5 5 4 1 . a r X i v : h t t p s : / / d o i . o r g / 1 0 . 1 1 6 2 / 1 0 5 4 7 4 6 9 8 5 6 5 5 4 1 . [16] M. Slater, Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments, Philos Trans R Soc Lond B Biol Sci 364 (2009) 3549–3557. [17] G. Caggianese, L. Gallo, P. Neroni, The Vive Controllers vs. Leap Motion for Interac- tions in Virtual Environments: A Comparative Evaluation, 2019, pp. 24–33. doi:1 0 . 1 0 0 7 / 978- 3- 319- 92231- 7_3. [18] I. Choi, E. Ofek, H. Benko, M. Sinclair, C. Holz, Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–13. URL: https://doi.org/10.1145/ 3173574.3174228. doi:1 0 . 1 1 4 5 / 3 1 7 3 5 7 4 . 3 1 7 4 2 2 8 . [19] L. Aguerreche, T. Duval, A. Lécuyer, Reconfigurable tangible devices for 3d virtual object manipulation by single or multiple users, in: Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, VRST ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 227–230. URL: https://doi.org/10.1145/1889863.1889913. doi:1 0 . 1145/1889863.1889913. [20] S. B. Schorr, A. M. Okamura, Fingertip tactile devices for virtual object manipulation and exploration, in: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 3115–3119. URL: https://doi.org/10.1145/3025453.3025744. doi:1 0 . 1 1 4 5 / 3 0 2 5 4 5 3 . 3 0 2 5 7 4 4 . [21] C. Bermejo, P. Hui, A survey on haptic technologies for mobile augmented reality (2017).