Virtual museums are one of the most interesting applications of Virtual Reality environment, but their success is strongly depending on the development of e ective interaction techniques allowing a natural and fast exploration of their contents. In this paper a series of interaction techniques in a full immersive Virtual Reality environment are presented. These techniques are speci cally designed to cover basic needs of a virtual museum experience such as navigating in the museum space and accessing meta-information associated to displayed items. Details of the implemented methods and preliminary results collected in user tests performed to compare di erent navigation choices are presented and discussed.
The recent developments on Virtual Reality (VR) and
Augmented Reality (AR) technologies made them
accessible and usable for general purpose applications [
In museum applications, virtuality tools, de ned as
combination of VR and AR, have been proved to be promising
for the presentation, as well as documentation, conservation
and preservation of cultural heritage (CH) items [
Copyright c 2016 for this paper by its authors. Copying permitted for private and academic purposes. techniques to create satisfactory interactive museum experiences. The development of core mechanics for the expected interactions in virtual museum and the design of e ective solution for them is therefore mandatory to build successful applications.
In this paper we focus on two of these core mechanics that are critical for many virtual museum applications: information retrieval from items in the scene such as art pieces, multimedia, etc. and navigation in the virtual museum environment. For both the mechanics, we designed di erent solutions based on mid-air gesture interaction and tested them with users in simple demo environments to evaluate their advantages and drawbacks. The goal of the study is to derive guidelines and a wide range of solutions for an optimized virtual museum experience. This paper is organized as follows: In Section 2 the proposed techniques are contextualized in the gesture-based interaction literature. In Section 3 each of the techniques designed and implemented is discussed. In Section 4 the experiments performed to analyze and compare the techniques are is presented along with the setup details. In Section 5 preliminary results and the future work directions are discussed. 2.
The integration of mixed reality technologies as an
improvement to the traditional museum experience has been
an ongoing phenomenon for the last years [
As shown in [
One of the enhancements a virtual museum can bene t from, compared to regular museums, is the easiness of interaction with objects and ways to retrieve information and multimedia content from them. A typical scenario is the one in which the user has the choice to inspect di erent objects in the scene. In a virtual museum a user can interact directly with the objects, look at them from any point of view or more in general have access to a di erent number of options not often possible in real museums. A common issue is however the object selection and the displaying of information, whether they should be either directly linked to the item, or a display of further possible actions to perform on it. Displaying everything at once for each object is unpractical for di erent reasons including, but not limited to, visual cluttering, and it is better to allow the user to select a speci c object and enable the display of linked information through an interaction trigger. In this work four solutions are presented for a speci c scenario with multiple objects in the scene. Each of the presented solutions features a gesture-based technique to select the wanted object and to trigger the display of its related information. Their di erent strengths and weak points will not only o er a di erent range of solutions for more speci c tasks but also provide good insights on viability based on the user performance and other data collected in the test stage.
Display Buttons: The user sees a button for each object. By pressing one of the buttons, the associated object is selected and becomes the focus of the view. On the right, all the text information is displayed. The button is supposed to give the best a ordance among the di erent methods as it naturally suggests a trigger interaction, while also hiding all the information related to an object, without causing any excessive cluttering.
Swipe: In this mode the user sees part of the view space reserved to text information display. There is an initial selected object by default picked from the set of objects available in the scene and all its information are also already displayed in the reserved space. By using a "swipe" gesture the user is able to cycle the currently selected object among the rest in the set and automatically display the information in the reserved space. In this method, the trivial trade-o , is between the available view space and the easiness of selecting a series of di erent objects.
Object Selection: The user is able to select the wanted object by touching it with a nger. This action automatically makes the object move in front of the user for a better display and also triggers the view of all the text information in a similar way the Display Button method does. This method is best to achieve natural interaction compared to the Display Buttons method as the interaction with the object is direct and can also be useful for applications in which a user can also perform other actions like touching parts of the object, change the view angle, modify its aspect and so on.
Object Picking: The technique is similar to Object Selection but in this case the user has to drag and drop each object on an anchor point in the scene (which is not anchored to the user camera) in order to trigger the display all the information (Figure 1). This is convenient in cases where the user has to select the object while not being forced to directly look at it. This happens when the user wants to still be able to look around freely or in speci c applications to have a second object selected for close comparison.
As suggested in [
Palm Rotation: The user can increase the movement speed by placing the palm of the hand in front of the hand tracker with the palm perpendicular to the ground and the ngers pointing forward (Figure 2). Any other di erent hand positioning interrupts the movement. This gesture was chosen to be intuitive and easy to learn and remember.
Palm Position: Similar to Palm Rotation but in this case the user just has to put hand in front of the hand tracker and over a xed distance threshold from the body. Again any other di erent hand positioning stops the movement. This variation was implemented to test the e ectiveness, in terms of performance and accuracy, of a pose trigger (Palm rotation) against a position in space kind of trigger (Palm Position).
Forward Button: A widget-like button is showed in a Heads-up Display (HUD) style and by pressing it the user is able to move forward (Figure 3). Releasing the button stops the movement. This is less natural, takes away part of the user view on the scene and it might be hard to combine with the selection and inspection methods presented in 3.1. Unlike the previous two solutions in which the triggering point where the gesture is recognized by the system is invisible and hard to detect, the advantage of this method is the ability for the user to clearly see the the trigger and have a more precise control on the movement speed.
Mobile Control: The user has to use a mobile phone's gyroscope as control device. By tilting it forward the inworld camera begins to move. In order to stop the movement the phone must be brought back to the original pose. With this solution a new kind of feedback is o ered to the user, compared to all the previous ones. The mobile phone serves as a smart-object to control the speed by using the inclination angle and o ers the possibility to use haptic feedback of vibration as alert for obstacles or notify the user about particular events taking place in the museum.
The total immersive experience is achieved through the
use of a head mounted display (HMD) and an IR hand
tracker. Speci cally low-cost technologies were chosen:
Oculus Rift DK2 [
Validation of the implemented methods for both information display and navigation was performed through experiments with subjects in demo environments. For information display testing, users received a hint about one of the available artworks (four in total for each execution) and had to gure out which one the hint was talking about by retrieving the available information with the di erent interaction techniques. This was repeated for all the di erent methods in a randomized order for each user. For navigation testing, users had follow with the di erent methods a path marked with checkpoints. The checkpoints became visible one at the time and always within the user's eld of view until the goal was reached. Again this was be repeated for all the methods in a randomized order.
Execution time was used as main measure of performance to compare the e ciency of the techniques against each other. The randomized order prevented the presence of obvious bias in the collected data. Other time splits are measured to identify critical point and possible bottlenecks in each method. In particular: the time spent reading the information against the time spent to access each object in the rst task and the number of times the user stops and resumes the navigation to identify possible problems with maintaining the control gesture. At the end of each session every user compiled a questionnaire to rate di erent aspect of the experience including, but not limited to learnability of the methods, easiness of execution, perceived stress and overall preferred technique.
At this stage of the study, data was collected, for thirty sessions for both tasks. The most interesting results derive from the total time of task completion. These results already show, performance-wise, relevant di erences between the proposed solutions. In Figure 5 the times for the rst task, selection and information retrieval, are shown. The worst performance is associated to the Object Picking technique. This was expected due to the extra step of drag and drop on the anchor point. The best performances come from the Swipe and Object Selection techniques. More data are needed to prove actual signi cant improvement compared to the other techniques. However, this again agrees with the design for this methods, in fact Swipe relies on the reserved space on screen to o er a quick selection of the objects by just using a single gesture that doesn't require strictly speci c directions or angles to be recognize by the system. Object Selection only requires the user to touch an object which is more natural than the Display Buttons. This last technique, that lies in somewhere in the middle on the performance scale, doesn't seem to bring any real advantage, with its focus on a ordance, compared to its natural counterpart (Object Selection).
In Figure 6 are the times for the second task, involving the environment navigation techniques. With the current data it's only possible to say that the Forward Button technique outperforms the other techniques in terms of execution time. This is due to the high control level on the movement granted by the displayed button. In our solutions the navigation is achieved with the use of hands interaction methods. Because of this, real natural interaction style can't be achieved, as the hands aren't the part of the human body used to directly move in the real surrounding space. This is one of the possible reasons that prevent the other implemented navigation techniques to bring any advantage in terms of time performance.
In the next stage of the work more data will be collected to be analyzed in depth. This will give a more general vision of all pros and cons of each solution, in order to o er a wider range of methods that can be chosen for speci c applications. The experiment setup will also serve as a platform for new or more re ned methods and their implementations, to be compared with the existing ones. In a new work, a simulation of a full immersive virtual museum will be developed to evaluate the behavior of various solutions on a real application prototype. The goal will be to o er better insights about convenient and e ective combination of techniques for both the core mechanics, necessary for this kind of experience, that were identi ed and discussed in this paper. Acknowlegements: This work was partially supported by the Scan4Reco project funded by EU Horizon 2020 Framework Programme under grant agreement no 665091.