Virtual Reality (VR) provides opportunity for immersive experiences in several fields of application, in particular in training and entertainment. Integrating tangible interaction with virtual environments that mimic real world situations can provide a tremendous enrichment to the user experience. Indeed, one of the main barriers to immersivity virtual objects is the use of fixed controllers with poor afordance. Touching, grasping, pushing, squeezing, if executed with common controllers are unnatural, impractical and sometimes frustrating. The paper presents the outcomes of a multidisciplinary panel conducted during ETIS 2020. The purpose of the panel was to engage a reflection on possible ways to integrate tangible interactions in virtual reality scenarios. To this extent, a case study was presented describing the development of a physical controller used in a virtual simulation of a DNA extraction procedure in a biotechnology laboratory. The case study shows the potential of using a real pipette to make the extraction task easy to perform in VR exploiting the afordance of the real object when gripping, pushing, moving the pipette. This opens new possibilities for enhanced multisensory interactions in VR environments.
Virtual Reality (VR) is nowadays wide-spreading since several commercial products are available
for reasonable prices. VR is exploited in several domains: from design and manufacturing process
[
Especially in critical situations and risky procedures, it is possible to train operators in safe virtual environments so that the experience gained in VR can be transferable to the real world.
Sound training applications of VR in the medical field are described in [ 4][5][6], while an overview of applications of experiential learning for construction workers is provided by [7][8].
Another interesting field of application is cultural heritage, where VR can enhance the museum visitor experience, acting as a time machine to immerse the visitors into other places and times [9]. (A. Innocenti)
To improve the efectiveness of VR applications, on one hand, the efort is towards the design of high fidelity virtual environments. The realism of the environment relies not only on the visual reconstruction of the environment itself but also on the multisensory stimulation which can be provided, including haptic feedback to perceive the forces and the material characteristics of objects [10].
For example, ElasticVR [11] is a wearable device that provides force-feedback through an elastic band, servo motors and mechanical locks. It is able to provide both passive force-feedback (e.g. simulating resistance and reaction force) and active force-feedback (e.g. simulating recoil and impact). TacTiles is a flexible haptic array to provide localized tactile feedback [ 12], that is able to render continuous contact with objects and the exploration of object surfaces and volumes. Haptic Revolver [13] is a handheld VR controller that renders fingertip haptics when interacting with virtual surfaces.
On the other hand, interactivity and more in particular tangible interaction - that is the possibility to manipulate objects and the ability of the environment to react to user’s action - is regarded to be a fundamental factor compared to the realism of the environment [14][15][16].
The current design trend is focused on intuitive interfaces by introducing natural interaction through gesture recognition and hand-tracking to interact with virtual objects (see for example: Oculus Blog, 2019).
While commercial VR devices are provided with their own controllers, some studies investigate new concepts exploiting hand-tracking sensors.
Caggianese, Gallo and Neroni [17] compared HTC Vive controllers with Leap Motion Camera in object manipulation tasks (e.g. selection, positioning and rotation). The authors found a better performance and less perceived dificulty of the Vive Controllers thanks to its stability, accuracy, and lower learning curve compared to that required by the Leap Motion sensor. Nevertheless, other dificulties occurred with both the devices, so the authors conclude that “neither of the two devices can be considered as the definitive solution to the problems relating to efective interactions in virtual environment”.
CLAW represents an augmentation of the traditional VR controller functionality [18]: it enables diferent interactions (grasping virtual objects, touching virtual surfaces, and triggering) and it generates vibrations and force-feedback in reaction to user’s action.
A brand new concept is proposed by Aguerreche, Duval and Lécuyer [19] who designed the Reconfigurable Tangible Device: it is composed of three or four arms rigidly linked together which can be compressed or stretched, to manipulate the shape and the motion of the virtual objects.
The challenge of the proposed panel was to reflect on diferent concepts of controllers to allow tangible interaction in virtual environments. Participants were engaged in a discussion on future applications of virtual controllers in virtual simulations.
The challenge of this panel was to envision future physical controller for VR applications integrating natural, playful, intriguing tangible interactions.
As said above, one of the main barriers in object manipulation in VR is the lack of realistic haptic feedback and controllers with proper afordance [ 20] to naturally execute tasks involving touching, squeezing, moving, pushing etc. as it happens in the real world [21].
Improving tangibility in VR ofers tremendous opportunities to enrich VR experiences.
To face this challenge, the panel engaged participants in the design of new hand-held haptic controllers, acting as input devices in VR applications. The expected outcomes were new concepts of VR controllers to use in virtual environments to manipulate virtual objects. Such new concepts were intended to augment or totally replace the current VR controllers.
The panel was based on a case study) developed by the recently inaugurated VR Lab managed by faculty and staf of the Department of Social, Political and Cognitive Science of the University of Siena, and funded under the Project of Excellence granted by the Italian Ministry of the University in 2018 (https://interdispoc.unisi.it/en). The VR Lab is provided with a platform for immersive virtual reality testing equipped with Oculus Quest head mounted displays (HMD from now on), HTC Vive HMDs, and an immersive CAVE room (Cave Automatic Virtual Environment - CAVE) where rear projections of computer generated 3D images are rendered for experimental simulations.
After a brief introduction focused on the two actually most used technologies in VR applications (HMDs and CAVE rooms) and a reflection about controllers and inputs in VR simulations, a case study was presented to participants.
It is actually really hard to mimic natural interaction within a VR experience. The gestures are always mediated by controllers; this results in unnatural and “crippled” interactions (with respect to the real world). Despite the fact that a lot of diferent types of controllers are developed by diferent HMDs producers (HTC, Oculus, Samsung etc.), the interaction possibilities ofered are more or less the same: pushbuttons and analog input peripherals (sticks, pads etc.).
This kind of input is not suitable when the user has to perform tasks based on interactions with specific devices.
Some examples could be: • A surgeon using a lancet • A mechanician using an electrical screwdriver
In each of these cases, developing a VR simulation introduces unnatural interactions since the controller does not have the required physical characteristics and afordance of the physical device.
Qualities like shape, weight, material afordance, handlings, are completely lost and the entire control is reduced to the use of virtual buttons and input devices which do not mimic the interaction with the physical arttefact.
To better explore the issue a case study was presented: a VR simulation of a DNA extraction procedure in a biotechnology laboratory.
The simulation, developed by the VR Lab of the University of Siena is currentl under development and will be used for learning and training activities. Wearing the HMD, the user will be projected into a biotechnology laboratory and will be asked to perform the procedure of extraction of some DNA fragments from a liquid solution. The simulation will las approximately 30 minutes. Figure 1 shows the physical pipette used for the DNA extraction task (left), Figure 2 shows the VR controller (right).
The objective of the simulation is to build a custom controller that mimics the physical properties of the real DNA extractor (shape, weight, handling, buttons etc.) and the related interactions to perform the task.
Since the panel was conducted in virtual modality, participants used Miro to support the brainstorming. The ultimate goal of the panel was to respond the following question:‘Can a physical controller for VR applications be designed to enhance multisensory experiences in simulation environments in order to increase immersivity?’.
In order to answer the question the participants were invited to: • Retrieve an image (from the web) of an object/tool (eg. a bicycle) • List the material properties and possible real multisensory experiences/feelings that can be perceived using that object (considering the bicycle: weight, shape, friction with terrain, wind, speed etc. ) • Imagine and describe a possible scenario and a VR multisensory experience (es: biking by the sea). • Reflect on ways to reproduce the actual experience in VR, integrating multisensory interactions possibilities, the material properties and real feelings in the simulated scenario
After 20 minutes of work, we gathered the results and started an open discussion about the possibilities presented by the participants trying to cluster similar results in order to produce a conceptual map of the diferent scenarios.
The map was discussed analyzing devices that are actually on the market (eg. Icaros, Tesla Suit, Virtuix Omni etc.) that can be used to recreate immersive multisensory experiences in simulated virtual reality environments.
Some interesting examples highlighted during the final analysis, were related to the possibility of integrating the perception of basic sensations useful to understand if the activity is well performed or not; for instance temperature or airflow perception (a glass-blower in front of an anvil relies on his temperature perception to understand if something is wrong; a paraglider or a skydiver relies on his wind perception to perform a correct maneuver). Participants proposed diferent design concepts to integrate such perceptions in a VR simulation using actual technologies in order to build new controllers/systems (eg. convectors and fans mounted on the HMDs to simulate temperature changes and airflow movements).
To wrap up, what emerged from the final reflections is that the actual possibility of interactions ofered by the controllers, are still not realistic enough to increase immersivity. The more we step away from the real objects trying to map their material properties into the interactions ofered by controllers, the more we decrease immersivity. Having said that, exploring the possibilities ofered by a mixed approach that combines physical and virtual objects could result in a definition of criteria and design guidelines that can strongly increase the immersivity level and the engagement of a VR simulation. [2] C. Botella, S. Quero, R. Baños, C. Perpiñá, A. Garcia-Palacios, G. Riva, Virtual reality and psychotherapy, Studies in health technology and informatics 99 (2004) 37–54. doi:1 0 . 3 2 3 3 / 9 7 8 - 1 - 6 0 7 5 0 - 9 4 3 - 1 - 3 7 . [3] C. Botella, A. García-Palacios, H. Villa, R. M. Baños, S. Quero, M. Alcañiz, G. Riva, Virtual reality exposure in the treatment of panic disorder and agoraphobia: A controlled study, Clinical Psychology & Psychotherapy 14 (2007) 164–175. URL: https: //onlinelibrary.wiley.com/doi/abs/10.1002/cpp.524. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 0 2 / c p p . 5 2 4 . a r X i v : h t t p s : / / o n l i n e l i b r a r y . w i l e y . c o m / d o i / p d f / 1 0 . 1 0 0 2 / c p p . 5 2 4 . [4] F. Mantovani, G. Castelnuovo, A. Gaggioli, G. Riva, Virtual reality training for health-care professionals, Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society 6 (2003) 389–95. doi:1 0 . 1 0 8 9 / 1 0 9 4 9 3 1 0 3 3 2 2 2 7 8 7 7 2 . [5] P. B. Andreatta, E. Maslowski, S. Petty, W. Shim, M. Marsh, T. Hall, S. Stern, J. Frankel, Virtual reality triage training provides a viable solution for disaster-preparedness, Acad Emerg Med 17 (2010) 870–876. [6] M. Reznek, P. Harter, T. Krummel, Virtual reality and simulation: Training the future emergency physician, Academic emergency medicine : oficial journal of the Society for Academic Emergency Medicine 9 (2002) 78–87. doi:1 0 . 1 1 1 1 / j . 1 5 5 3 - 2 7 1 2 . 2 0 0 2 . t b 0 1 1 7 2 . x . [7] R. Sacks, A. Perlman, R. Barak, Construction safety training using immersive virtual reality, Construction Management and Economics 31 (2013) 1005–1017. doi:1 0 . 1 0 8 0 / 0 1 4 4 6 1 9 3 . 2 0 1 3 . 8 2 8 8 4 4 . [8] Q. T. Le, A. Pedro, C. Park, A social virtual reality based construction safety education system for experiential learning, J Intell Robot Syst 79 (2014) 1–20. doi:1 0 . 1 0 0 7 / s 1 0 8 4 6 - 0 1 4 - 0 1 1 2 - z . [9] M. K. Bekele, R. Pierdicca, E. Frontoni, E. S. Malinverni, J. Gain, A survey of augmented, virtual, and mixed reality for cultural heritage, J. Comput. Cult. Herit. 11 (2018). URL: https://doi.org/10.1145/3145534. doi:1 0 . 1 1 4 5 / 3 1 4 5 5 3 4 . [10] E. Gobbetti, R. Scateni, Virtual reality: Past, present and future, Studies in health technology and informatics 58 (1998) 3–20. doi:1 0 . 3 2 3 3 / 9 7 8 - 1 - 6 0 7 5 0 - 9 0 2 - 8 - 3 . [11] H.-R. Tsai, J. Rekimoto, Elasticvr: Providing multi-level active and passive force feedback in virtual reality using elasticity, in: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–4. URL: https://doi.org/10.1145/3170427.3186540. doi:1 0 . 1 1 4 5 / 3 1 7 0 4 2 7 . 3 1 8 6 5 4 0 . [12] V. Vechev, J. Zarate, D. Lindlbauer, R. Hinchet, H. Shea, O. Hilliges, Tactiles: Dual-mode low-power electromagnetic actuators for rendering continuous contact and spatial haptic patterns in vr, in: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019, pp. 312–320. doi:1 0 . 1 1 0 9 / V R . 2 0 1 9 . 8 7 9 7 9 2 1 . [13] E. Whitmire, H. Benko, C. Holz, E. Ofek, M. Sinclair, Haptic revolver: Touch, shear, texture, and shape rendering on a reconfigurable virtual reality controller, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–12. URL: https://doi.org/10.1145/ 3173574.3173660. doi:1 0 . 1 1 4 5 / 3 1 7 3 5 7 4 . 3 1 7 3 6 6 0 . [14] M. Sanchez-Vives, M. Slater, From presence to consciousness through virtual reality,
Nature reviews. Neuroscience 6 (2005) 332–9. doi:1 0 . 1 0 3 8 / n r n 1 6 5 1 . [15] P. Zahorik, R. L. Jenison, Presence as being-in-the-world, Presence: Teleoperators and Virtual Environments 7 (1998) 78–89. URL: https://doi.org/10.1162/105474698565541. doi:1 0 . 1 1 6 2 / 1 0 5 4 7 4 6 9 8 5 6 5 5 4 1 . a r X i v : h t t p s : / / d o i . o r g / 1 0 . 1 1 6 2 / 1 0 5 4 7 4 6 9 8 5 6 5 5 4 1 . [16] M. Slater, Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments, Philos Trans R Soc Lond B Biol Sci 364 (2009) 3549–3557. [17] G. Caggianese, L. Gallo, P. Neroni, The Vive Controllers vs. Leap Motion for Interactions in Virtual Environments: A Comparative Evaluation, 2019, pp. 24–33. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 9 2 2 3 1 - 7 _ 3 . [18] I. Choi, E. Ofek, H. Benko, M. Sinclair, C. Holz, Claw: A multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 1–13. URL: https://doi.org/10.1145/ 3173574.3174228. doi:1 0 . 1 1 4 5 / 3 1 7 3 5 7 4 . 3 1 7 4 2 2 8 . [19] L. Aguerreche, T. Duval, A. Lécuyer, Reconfigurable tangible devices for 3d virtual object manipulation by single or multiple users, in: Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology, VRST ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 227–230. URL: https://doi.org/10.1145/1889863.1889913. doi:1 0 . 1 1 4 5 / 1 8 8 9 8 6 3 . 1 8 8 9 9 1 3 . [20] S. B. Schorr, A. M. Okamura, Fingertip tactile devices for virtual object manipulation and exploration, in: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 3115–3119. URL: https://doi.org/10.1145/3025453.3025744. doi:1 0 . 1 1 4 5 / 3 0 2 5 4 5 3 . 3 0 2 5 7 4 4 . [21] C. Bermejo, P. Hui, A survey on haptic technologies for mobile augmented reality (2017).