GESTURAL INTERACTION AND NAVIGATION TECHNIQUES FOR VIRTUAL MUSEUM EXPERIENCES Fabio Marco Caputo Irina Mihaela Ciortan Davide Corsi Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science University of Verona, Italy University of Verona, Italy University of Verona, Italy fabiomarco.caputo@univr.it Marco De Stefani Andrea Giachetti Dept. of Computer Science Dept. of Computer Science University of Verona, Italy University of Verona, Italy ABSTRACT techniques to create satisfactory interactive museum experi- Virtual museums are one of the most interesting applications ences. The development of core mechanics for the expected of Virtual Reality environment, but their success is strongly interactions in virtual museum and the design of effective depending on the development of effective interaction tech- solution for them is therefore mandatory to build successful niques allowing a natural and fast exploration of their con- applications. tents. In this paper a series of interaction techniques in a In this paper we focus on two of these core mechanics that full immersive Virtual Reality environment are presented. are critical for many virtual museum applications: informa- These techniques are specifically designed to cover basic tion retrieval from items in the scene such as art pieces, needs of a virtual museum experience such as navigating multimedia, etc. and navigation in the virtual museum en- in the museum space and accessing meta-information asso- vironment. For both the mechanics, we designed different ciated to displayed items. Details of the implemented meth- solutions based on mid-air gesture interaction and tested ods and preliminary results collected in user tests performed them with users in simple demo environments to evaluate to compare different navigation choices are presented and their advantages and drawbacks. The goal of the study is to discussed. derive guidelines and a wide range of solutions for an opti- mized virtual museum experience. This paper is organized as follows: In Section 2 the proposed techniques are con- Keywords textualized in the gesture-based interaction literature. In HCI; Virtual Reality; Gestural Input; Virtual Museum; Section 3 each of the techniques designed and implemented is discussed. In Section 4 the experiments performed to ana- 1. INTRODUCTION lyze and compare the techniques are is presented along with the setup details. In Section 5 preliminary results and the The recent developments on Virtual Reality (VR) and future work directions are discussed. Augmented Reality (AR) technologies made them accessi- ble and usable for general purpose applications [11] ranging from entertainment to professional tools for industrial de- 2. RELATED WORK sign or medical healthcare. Each field identifies different interactions tasks between users and system, and requires The integration of mixed reality technologies as an im- specifically designed techniques to obtain satisfactory vir- provement to the traditional museum experience has been tual experiences. an ongoing phenomenon for the last years [11], [1], [6]. This In museum applications, virtuality tools, defined as combi- phenomenon has also branched to examples of complete vir- nation of VR and AR, have been proved to be promising tualization of museums that can enhance user experience for the presentation, as well as documentation, conservation through the higher freedom in interacting with the artefacts and preservation of cultural heritage (CH) items [1, 2, 5]. [2] and can provide a series of benefits for socio-educational One of the main advantages of VR consists in the high num- purposes [6]. ber of degrees of freedom for user interactivity [12]. As shown in [1], two example criteria for defining a taxon- Different kinds of virtual museums require effective core omy of virtual museums can be the content of the exhibition (already existing structures or ex-novo reconstruction of the items in a virtual environment) and the access method to the museum. Based on the latter criterion, there are web- based virtual museums, that can be remotely-accessed by the users, such as ARCO [10] and virtual museums based on VR displays, that can be implemented on-site (interac- Copyright c 2016 for this paper by its authors. Copying permitted for tive kiosks in the museums [6], in archaeologic or excavation private and academic purposes. sites [2]) or simultaneously online and on-site [11]. In [1], it is pointed out that the interactivity with the work of art is an important pre-requisite of the multimedia rep- resentations of CH items and that a stereoscopic view con- the rest in the set and automatically display the information tributes to the immersive experience and moreover to the in the reserved space. In this method, the trivial trade-off, navigability of the environment [3]. There is a tendency to is between the available view space and the easiness of se- minimize the contribution of computers in their traditional lecting a series of different objects. form, also known as disappearing computer technology [6], where mixed reality engages the user to a high extent, so Object Selection: The user is able to select the wanted that the focus of the user is on the interaction with the vir- object by touching it with a finger. This action automati- tual objects, rather than the media or means of visualiza- cally makes the object move in front of the user for a better tion. In order to absorb the users as much as possible in the display and also triggers the view of all the text informa- virtual environment, there are several navigation paradigms tion in a similar way the Display Button method does. This [12] and gesture-based navigation is presented as an effective method is best to achieve natural interaction compared to strategy for user interaction [7] with stereoscopic environ- the Display Buttons method as the interaction with the ob- ments. The head-mounted display enables such interaction ject is direct and can also be useful for applications in which and has been used for live exhibitions in real museums in a a user can also perform other actions like touching parts of hybrid physical-digital artefact setup [6]. In the following the object, change the view angle, modify its aspect and so sections, we approach the Oculus Rift and its advantages as on. an appropriate gesture-based VR navigation technology for virtual museums. Object Picking: The technique is similar to Object Se- lection but in this case the user has to drag and drop each 3. INTERACTION TECHNIQUES object on an anchor point in the scene (which is not an- chored to the user camera) in order to trigger the display 3.1 Information Display all the information (Figure 1). This is convenient in cases where the user has to select the object while not being forced One of the enhancements a virtual museum can benefit to directly look at it. This happens when the user wants to from, compared to regular museums, is the easiness of in- still be able to look around freely or in specific applications teraction with objects and ways to retrieve information and to have a second object selected for close comparison. multimedia content from them. A typical scenario is the one in which the user has the choice to inspect different ob- jects in the scene. In a virtual museum a user can interact directly with the objects, look at them from any point of view or more in general have access to a different number of options not often possible in real museums. A common issue is however the object selection and the displaying of information, whether they should be either directly linked to the item, or a display of further possible actions to per- form on it. Displaying everything at once for each object is unpractical for different reasons including, but not limited to, visual cluttering, and it is better to allow the user to Figure 1: Object Picking technique. The selected select a specific object and enable the display of linked in- object is being dragged towards the anchor point. formation through an interaction trigger. In this work four The rest of the items in the scene remain visibile solutions are presented for a specific scenario with multiple and potentially available for further actions. objects in the scene. Each of the presented solutions features a gesture-based technique to select the wanted object and to trigger the display of its related information. Their differ- 3.2 Environment Navigation ent strengths and weak points will not only offer a different As suggested in [8] virtual museums are not necessarily range of solutions for more specific tasks but also provide bound to real space simulation. Keeping that in mind an- good insights on viability based on the user performance other critical issue in case of virtual museums featuring ex- and other data collected in the test stage. plorable space is the control of user movement. The con- sidered scenario is again one in which multiple objects are Display Buttons: The user sees a button for each ob- accessible in the museum. In 3.1 a single set of items all ject. By pressing one of the buttons, the associated object is visible in a single scene was considered while now a wider selected and becomes the focus of the view. On the right, all scenario is taken into account with multiple set of objects the text information is displayed. The button is supposed displaced around the virtual environment. This kind of sce- to give the best affordance among the different methods as nario require the user to be able to move and cover bigger it naturally suggests a trigger interaction, while also hiding distances so different solutions are presented to enable free all the information related to an object, without causing any movement around a virtual space. In all these solutions an excessive cluttering. hand interaction controls the subject’s forward speed while the steering is controlled with the HMD by gazing at the Swipe: In this mode the user sees part of the view space direction where the user wishes to go to. reserved to text information display. There is an initial se- lected object by default picked from the set of objects avail- Palm Rotation: The user can increase the movement able in the scene and all its information are also already speed by placing the palm of the hand in front of the hand displayed in the reserved space. By using a ”swipe” gesture tracker with the palm perpendicular to the ground and the the user is able to cycle the currently selected object among fingers pointing forward (Figure 2). Any other different hand positioning interrupts the movement. This gesture was chosen to be intuitive and easy to learn and remember. Palm Position: Similar to Palm Rotation but in this case the user just has to put hand in front of the hand tracker and over a fixed distance threshold from the body. Again any other different hand positioning stops the movement. This variation was implemented to test the effectiveness, in terms of performance and accuracy, of a pose trigger (Palm rotation) against a position in space kind of trigger (Palm Position). Figure 3: Forward Button navigation technique. The user is approaching a waypoint by pressing Forward Button: A widget-like button is showed in a the button to control the movement speed (azure Heads-up Display (HUD) style and by pressing it the user square). is able to move forward (Figure 3). Releasing the button stops the movement. This is less natural, takes away part of the user view on the scene and it might be hard to com- bine with the selection and inspection methods presented in 3.1. Unlike the previous two solutions in which the trigger- ing point where the gesture is recognized by the system is invisible and hard to detect, the advantage of this method is the ability for the user to clearly see the the trigger and have a more precise control on the movement speed. Mobile Control: The user has to use a mobile phone’s gyroscope as control device. By tilting it forward the in- Figure 4: Oculus Rift and Leap Motion configura- world camera begins to move. In order to stop the movement tion the phone must be brought back to the original pose. With this solution a new kind of feedback is offered to the user, compared to all the previous ones. The mobile phone serves 4.1 Data collection as a smart-object to control the speed by using the inclina- Validation of the implemented methods for both informa- tion angle and offers the possibility to use haptic feedback tion display and navigation was performed through experi- of vibration as alert for obstacles or notify the user about ments with subjects in demo environments. For information particular events taking place in the museum. display testing, users received a hint about one of the avail- able artworks (four in total for each execution) and had to figure out which one the hint was talking about by retriev- ing the available information with the different interaction techniques. This was repeated for all the different methods in a randomized order for each user. For navigation testing, users had follow with the different methods a path marked with checkpoints. The checkpoints became visible one at the time and always within the user’s field of view until the goal was reached. Again this was be repeated for all the methods in a randomized order. Execution time was used as main measure of performance to Figure 2: Palm Rotation navigation technique. The compare the efficiency of the techniques against each other. user is approaching the first waypoint on the path The randomized order prevented the presence of obvious (red door). bias in the collected data. Other time splits are measured to identify critical point and possible bottlenecks in each method. In particular: the time spent reading the informa- tion against the time spent to access each object in the first task and the number of times the user stops and resumes 4. EXPERIMENT SETUP the navigation to identify possible problems with maintain- The total immersive experience is achieved through the ing the control gesture. At the end of each session every use of a head mounted display (HMD) and an IR hand user compiled a questionnaire to rate different aspect of the tracker. Specifically low-cost technologies were chosen: Ocu- experience including, but not limited to learnability of the lus Rift DK2 [4] as HMD and Leap Motion Controller [9] for methods, easiness of execution, perceived stress and overall hand tracking. This was also to have a test setup closer to preferred technique. a real case scenario in which high-performance devices are not available. Figure 4 shows the configuration used with the hand tracker mounted directly on the HMD to ensure 5. PRELIMINARY RESULTS the hands are always in the hand tracker field of view. At this stage of the study, data was collected, for thirty sessions for both tasks. The most interesting results derive from the total time of task completion. These results al- ready show, performance-wise, relevant differences between the proposed solutions. In Figure 5 the times for the first task, selection and information retrieval, are shown. The worst performance is associated to the Object Picking tech- nique. This was expected due to the extra step of drag and drop on the anchor point. The best performances come from the Swipe and Object Selection techniques. More data are needed to prove actual significant improvement compared to the other techniques. However, this again agrees with the design for this methods, in fact Swipe relies on the re- served space on screen to offer a quick selection of the objects Figure 6: Second task completion times by just using a single gesture that doesn’t require strictly specific directions or angles to be recognize by the system. 6. REFERENCES Object Selection only requires the user to touch an object [1] F. Bruno, S. Bruno, G. De Sensi, M.-L. Luchi, which is more natural than the Display Buttons. This last S. Mancuso, and M. Muzzupappa. From 3d technique, that lies in somewhere in the middle on the per- reconstruction to virtual reality: A complete formance scale, doesn’t seem to bring any real advantage, methodology for digital archaeological exhibition. with its focus on affordance, compared to its natural coun- Journal of Cultural Heritage, 11(1):42–49, 2010. terpart (Object Selection). [2] F. Cameron and S. Kenderdine. Theorizing Digital In Figure 6 are the times for the second task, involving the Cultural Heritage: A Critical Discourse (Media in environment navigation techniques. With the current data Transition). The MIT Press, 2007. it’s only possible to say that the Forward Button technique outperforms the other techniques in terms of execution time. [3] S. E. Chen. Quicktime vr: An image-based approach This is due to the high control level on the movement granted to virtual environment navigation. In Proceedings of by the displayed button. In our solutions the navigation is the 22nd annual conference on Computer graphics and achieved with the use of hands interaction methods. Because interactive techniques, pages 29–38. ACM, 1995. of this, real natural interaction style can’t be achieved, as [4] P. R. Desai, P. N. Desai, K. D. Ajmera, and K. Mehta. the hands aren’t the part of the human body used to di- A review paper on oculus rift-a virtual reality headset. rectly move in the real surrounding space. This is one of the arXiv preprint arXiv:1408.1173, 2014. possible reasons that prevent the other implemented navi- [5] A. Gaitatzes, D. Christopoulos, and M. Roussou. gation techniques to bring any advantage in terms of time Reviving the past: cultural heritage meets virtual performance. reality. In Proceedings of the 2001 conference on In the next stage of the work more data will be collected Virtual reality, archeology, and cultural heritage, pages to be analyzed in depth. This will give a more general vi- 103–110. ACM, 2001. sion of all pros and cons of each solution, in order to offer a [6] T. Hall, L. Ciolfi, L. Bannon, et al. The visitor as wider range of methods that can be chosen for specific appli- virtual archaeologist: explorations in mixed reality cations. The experiment setup will also serve as a platform technology to enhance educational and social for new or more refined methods and their implementations, interaction in the museum. In Proceedings of the 2001 to be compared with the existing ones. In a new work, a conference on Virtual reality, archeology, and cultural simulation of a full immersive virtual museum will be de- heritage, pages 91–96. ACM, 2001. veloped to evaluate the behavior of various solutions on a [7] G. A. Satalich. Navigation and wayfinding in virtual real application prototype. The goal will be to offer better reality: Finding the proper tools and cues to enhance insights about convenient and effective combination of tech- navigational awareness. PhD thesis, University of niques for both the core mechanics, necessary for this kind of Washington, 1995. experience, that were identified and discussed in this paper. [8] W. Schweibenz. Virtual museums. The Development Acknowlegements: This work was partially supported by of Virtual Museums. ICOM News Magazine, (3), 2004. the Scan4Reco project funded by EU Horizon 2020 Frame- [9] F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler. work Programme under grant agreement no 665091. Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13(5):6380–6393, 2013. [10] M. White, N. Mourkoussis, J. Darcy, et al. Arco-an architecture for digitization, management and presentation of virtual exhibitions. In Computer Graphics International, 2004. Proceedings, pages 622–625. IEEE, 2004. [11] R. Wojciechowski, K. Walczak, M. White, and W. Cellary. Building virtual and augmented reality museum exhibitions. In Proceedings of the ninth international conference on 3D Web technology, pages 135–144. ACM, 2004. [12] J. Zara. Virtual reality and cultural heritage on the Figure 5: First task completion times web. In Proceedings of the 7th International Conference on Computer Graphics and Artificial Intelligence, pages 101–112, 2004.