=Paper=
{{Paper
|id=Vol-1621/paper6
|storemode=property
|title= Gestural Interaction and Navigation Techniques for Virtual Museum Experiences
|pdfUrl=https://ceur-ws.org/Vol-1621/paper6.pdf
|volume=Vol-1621
|authors=Fabio Marco Caputo,Irina Mihaela Ciortan,Davide Corsi,Marco De Stefani,Andrea Giachetti
|dblpUrl=https://dblp.org/rec/conf/avi/CaputoCCSG16
}}
== Gestural Interaction and Navigation Techniques for Virtual Museum Experiences ==
https://ceur-ws.org/Vol-1621/paper6.pdf
GESTURAL INTERACTION AND NAVIGATION
TECHNIQUES FOR VIRTUAL MUSEUM EXPERIENCES
Fabio Marco Caputo Irina Mihaela Ciortan Davide Corsi
Dept. of Computer Science Dept. of Computer Science Dept. of Computer Science
University of Verona, Italy University of Verona, Italy University of Verona, Italy
fabiomarco.caputo@univr.it
Marco De Stefani Andrea Giachetti
Dept. of Computer Science Dept. of Computer Science
University of Verona, Italy University of Verona, Italy
ABSTRACT techniques to create satisfactory interactive museum experi-
Virtual museums are one of the most interesting applications ences. The development of core mechanics for the expected
of Virtual Reality environment, but their success is strongly interactions in virtual museum and the design of effective
depending on the development of effective interaction tech- solution for them is therefore mandatory to build successful
niques allowing a natural and fast exploration of their con- applications.
tents. In this paper a series of interaction techniques in a In this paper we focus on two of these core mechanics that
full immersive Virtual Reality environment are presented. are critical for many virtual museum applications: informa-
These techniques are specifically designed to cover basic tion retrieval from items in the scene such as art pieces,
needs of a virtual museum experience such as navigating multimedia, etc. and navigation in the virtual museum en-
in the museum space and accessing meta-information asso- vironment. For both the mechanics, we designed different
ciated to displayed items. Details of the implemented meth- solutions based on mid-air gesture interaction and tested
ods and preliminary results collected in user tests performed them with users in simple demo environments to evaluate
to compare different navigation choices are presented and their advantages and drawbacks. The goal of the study is to
discussed. derive guidelines and a wide range of solutions for an opti-
mized virtual museum experience. This paper is organized
as follows: In Section 2 the proposed techniques are con-
Keywords textualized in the gesture-based interaction literature. In
HCI; Virtual Reality; Gestural Input; Virtual Museum; Section 3 each of the techniques designed and implemented
is discussed. In Section 4 the experiments performed to ana-
1. INTRODUCTION lyze and compare the techniques are is presented along with
the setup details. In Section 5 preliminary results and the
The recent developments on Virtual Reality (VR) and
future work directions are discussed.
Augmented Reality (AR) technologies made them accessi-
ble and usable for general purpose applications [11] ranging
from entertainment to professional tools for industrial de- 2. RELATED WORK
sign or medical healthcare. Each field identifies different
interactions tasks between users and system, and requires The integration of mixed reality technologies as an im-
specifically designed techniques to obtain satisfactory vir- provement to the traditional museum experience has been
tual experiences. an ongoing phenomenon for the last years [11], [1], [6]. This
In museum applications, virtuality tools, defined as combi- phenomenon has also branched to examples of complete vir-
nation of VR and AR, have been proved to be promising tualization of museums that can enhance user experience
for the presentation, as well as documentation, conservation through the higher freedom in interacting with the artefacts
and preservation of cultural heritage (CH) items [1, 2, 5]. [2] and can provide a series of benefits for socio-educational
One of the main advantages of VR consists in the high num- purposes [6].
ber of degrees of freedom for user interactivity [12]. As shown in [1], two example criteria for defining a taxon-
Different kinds of virtual museums require effective core omy of virtual museums can be the content of the exhibition
(already existing structures or ex-novo reconstruction of the
items in a virtual environment) and the access method to
the museum. Based on the latter criterion, there are web-
based virtual museums, that can be remotely-accessed by
the users, such as ARCO [10] and virtual museums based
on VR displays, that can be implemented on-site (interac-
Copyright c 2016 for this paper by its authors. Copying permitted for tive kiosks in the museums [6], in archaeologic or excavation
private and academic purposes. sites [2]) or simultaneously online and on-site [11].
In [1], it is pointed out that the interactivity with the work
of art is an important pre-requisite of the multimedia rep-
resentations of CH items and that a stereoscopic view con- the rest in the set and automatically display the information
tributes to the immersive experience and moreover to the in the reserved space. In this method, the trivial trade-off,
navigability of the environment [3]. There is a tendency to is between the available view space and the easiness of se-
minimize the contribution of computers in their traditional lecting a series of different objects.
form, also known as disappearing computer technology [6],
where mixed reality engages the user to a high extent, so Object Selection: The user is able to select the wanted
that the focus of the user is on the interaction with the vir- object by touching it with a finger. This action automati-
tual objects, rather than the media or means of visualiza- cally makes the object move in front of the user for a better
tion. In order to absorb the users as much as possible in the display and also triggers the view of all the text informa-
virtual environment, there are several navigation paradigms tion in a similar way the Display Button method does. This
[12] and gesture-based navigation is presented as an effective method is best to achieve natural interaction compared to
strategy for user interaction [7] with stereoscopic environ- the Display Buttons method as the interaction with the ob-
ments. The head-mounted display enables such interaction ject is direct and can also be useful for applications in which
and has been used for live exhibitions in real museums in a a user can also perform other actions like touching parts of
hybrid physical-digital artefact setup [6]. In the following the object, change the view angle, modify its aspect and so
sections, we approach the Oculus Rift and its advantages as on.
an appropriate gesture-based VR navigation technology for
virtual museums. Object Picking: The technique is similar to Object Se-
lection but in this case the user has to drag and drop each
3. INTERACTION TECHNIQUES object on an anchor point in the scene (which is not an-
chored to the user camera) in order to trigger the display
3.1 Information Display all the information (Figure 1). This is convenient in cases
where the user has to select the object while not being forced
One of the enhancements a virtual museum can benefit
to directly look at it. This happens when the user wants to
from, compared to regular museums, is the easiness of in-
still be able to look around freely or in specific applications
teraction with objects and ways to retrieve information and
to have a second object selected for close comparison.
multimedia content from them. A typical scenario is the
one in which the user has the choice to inspect different ob-
jects in the scene. In a virtual museum a user can interact
directly with the objects, look at them from any point of
view or more in general have access to a different number
of options not often possible in real museums. A common
issue is however the object selection and the displaying of
information, whether they should be either directly linked
to the item, or a display of further possible actions to per-
form on it. Displaying everything at once for each object is
unpractical for different reasons including, but not limited
to, visual cluttering, and it is better to allow the user to Figure 1: Object Picking technique. The selected
select a specific object and enable the display of linked in- object is being dragged towards the anchor point.
formation through an interaction trigger. In this work four The rest of the items in the scene remain visibile
solutions are presented for a specific scenario with multiple and potentially available for further actions.
objects in the scene. Each of the presented solutions features
a gesture-based technique to select the wanted object and to
trigger the display of its related information. Their differ- 3.2 Environment Navigation
ent strengths and weak points will not only offer a different As suggested in [8] virtual museums are not necessarily
range of solutions for more specific tasks but also provide bound to real space simulation. Keeping that in mind an-
good insights on viability based on the user performance other critical issue in case of virtual museums featuring ex-
and other data collected in the test stage. plorable space is the control of user movement. The con-
sidered scenario is again one in which multiple objects are
Display Buttons: The user sees a button for each ob- accessible in the museum. In 3.1 a single set of items all
ject. By pressing one of the buttons, the associated object is visible in a single scene was considered while now a wider
selected and becomes the focus of the view. On the right, all scenario is taken into account with multiple set of objects
the text information is displayed. The button is supposed displaced around the virtual environment. This kind of sce-
to give the best affordance among the different methods as nario require the user to be able to move and cover bigger
it naturally suggests a trigger interaction, while also hiding distances so different solutions are presented to enable free
all the information related to an object, without causing any movement around a virtual space. In all these solutions an
excessive cluttering. hand interaction controls the subject’s forward speed while
the steering is controlled with the HMD by gazing at the
Swipe: In this mode the user sees part of the view space direction where the user wishes to go to.
reserved to text information display. There is an initial se-
lected object by default picked from the set of objects avail- Palm Rotation: The user can increase the movement
able in the scene and all its information are also already speed by placing the palm of the hand in front of the hand
displayed in the reserved space. By using a ”swipe” gesture tracker with the palm perpendicular to the ground and the
the user is able to cycle the currently selected object among fingers pointing forward (Figure 2). Any other different
hand positioning interrupts the movement. This gesture was
chosen to be intuitive and easy to learn and remember.
Palm Position: Similar to Palm Rotation but in this
case the user just has to put hand in front of the hand tracker
and over a fixed distance threshold from the body. Again
any other different hand positioning stops the movement.
This variation was implemented to test the effectiveness, in
terms of performance and accuracy, of a pose trigger (Palm
rotation) against a position in space kind of trigger (Palm
Position). Figure 3: Forward Button navigation technique.
The user is approaching a waypoint by pressing
Forward Button: A widget-like button is showed in a the button to control the movement speed (azure
Heads-up Display (HUD) style and by pressing it the user square).
is able to move forward (Figure 3). Releasing the button
stops the movement. This is less natural, takes away part
of the user view on the scene and it might be hard to com-
bine with the selection and inspection methods presented in
3.1. Unlike the previous two solutions in which the trigger-
ing point where the gesture is recognized by the system is
invisible and hard to detect, the advantage of this method
is the ability for the user to clearly see the the trigger and
have a more precise control on the movement speed.
Mobile Control: The user has to use a mobile phone’s
gyroscope as control device. By tilting it forward the in- Figure 4: Oculus Rift and Leap Motion configura-
world camera begins to move. In order to stop the movement tion
the phone must be brought back to the original pose. With
this solution a new kind of feedback is offered to the user,
compared to all the previous ones. The mobile phone serves
4.1 Data collection
as a smart-object to control the speed by using the inclina- Validation of the implemented methods for both informa-
tion angle and offers the possibility to use haptic feedback tion display and navigation was performed through experi-
of vibration as alert for obstacles or notify the user about ments with subjects in demo environments. For information
particular events taking place in the museum. display testing, users received a hint about one of the avail-
able artworks (four in total for each execution) and had to
figure out which one the hint was talking about by retriev-
ing the available information with the different interaction
techniques. This was repeated for all the different methods
in a randomized order for each user. For navigation testing,
users had follow with the different methods a path marked
with checkpoints. The checkpoints became visible one at the
time and always within the user’s field of view until the goal
was reached. Again this was be repeated for all the methods
in a randomized order.
Execution time was used as main measure of performance to
Figure 2: Palm Rotation navigation technique. The compare the efficiency of the techniques against each other.
user is approaching the first waypoint on the path The randomized order prevented the presence of obvious
(red door). bias in the collected data. Other time splits are measured
to identify critical point and possible bottlenecks in each
method. In particular: the time spent reading the informa-
tion against the time spent to access each object in the first
task and the number of times the user stops and resumes
4. EXPERIMENT SETUP the navigation to identify possible problems with maintain-
The total immersive experience is achieved through the ing the control gesture. At the end of each session every
use of a head mounted display (HMD) and an IR hand user compiled a questionnaire to rate different aspect of the
tracker. Specifically low-cost technologies were chosen: Ocu- experience including, but not limited to learnability of the
lus Rift DK2 [4] as HMD and Leap Motion Controller [9] for methods, easiness of execution, perceived stress and overall
hand tracking. This was also to have a test setup closer to preferred technique.
a real case scenario in which high-performance devices are
not available. Figure 4 shows the configuration used with
the hand tracker mounted directly on the HMD to ensure 5. PRELIMINARY RESULTS
the hands are always in the hand tracker field of view. At this stage of the study, data was collected, for thirty
sessions for both tasks. The most interesting results derive
from the total time of task completion. These results al-
ready show, performance-wise, relevant differences between
the proposed solutions. In Figure 5 the times for the first
task, selection and information retrieval, are shown. The
worst performance is associated to the Object Picking tech-
nique. This was expected due to the extra step of drag and
drop on the anchor point. The best performances come from
the Swipe and Object Selection techniques. More data are
needed to prove actual significant improvement compared
to the other techniques. However, this again agrees with
the design for this methods, in fact Swipe relies on the re-
served space on screen to offer a quick selection of the objects Figure 6: Second task completion times
by just using a single gesture that doesn’t require strictly
specific directions or angles to be recognize by the system. 6. REFERENCES
Object Selection only requires the user to touch an object [1] F. Bruno, S. Bruno, G. De Sensi, M.-L. Luchi,
which is more natural than the Display Buttons. This last S. Mancuso, and M. Muzzupappa. From 3d
technique, that lies in somewhere in the middle on the per- reconstruction to virtual reality: A complete
formance scale, doesn’t seem to bring any real advantage, methodology for digital archaeological exhibition.
with its focus on affordance, compared to its natural coun- Journal of Cultural Heritage, 11(1):42–49, 2010.
terpart (Object Selection).
[2] F. Cameron and S. Kenderdine. Theorizing Digital
In Figure 6 are the times for the second task, involving the
Cultural Heritage: A Critical Discourse (Media in
environment navigation techniques. With the current data
Transition). The MIT Press, 2007.
it’s only possible to say that the Forward Button technique
outperforms the other techniques in terms of execution time. [3] S. E. Chen. Quicktime vr: An image-based approach
This is due to the high control level on the movement granted to virtual environment navigation. In Proceedings of
by the displayed button. In our solutions the navigation is the 22nd annual conference on Computer graphics and
achieved with the use of hands interaction methods. Because interactive techniques, pages 29–38. ACM, 1995.
of this, real natural interaction style can’t be achieved, as [4] P. R. Desai, P. N. Desai, K. D. Ajmera, and K. Mehta.
the hands aren’t the part of the human body used to di- A review paper on oculus rift-a virtual reality headset.
rectly move in the real surrounding space. This is one of the arXiv preprint arXiv:1408.1173, 2014.
possible reasons that prevent the other implemented navi- [5] A. Gaitatzes, D. Christopoulos, and M. Roussou.
gation techniques to bring any advantage in terms of time Reviving the past: cultural heritage meets virtual
performance. reality. In Proceedings of the 2001 conference on
In the next stage of the work more data will be collected Virtual reality, archeology, and cultural heritage, pages
to be analyzed in depth. This will give a more general vi- 103–110. ACM, 2001.
sion of all pros and cons of each solution, in order to offer a [6] T. Hall, L. Ciolfi, L. Bannon, et al. The visitor as
wider range of methods that can be chosen for specific appli- virtual archaeologist: explorations in mixed reality
cations. The experiment setup will also serve as a platform technology to enhance educational and social
for new or more refined methods and their implementations, interaction in the museum. In Proceedings of the 2001
to be compared with the existing ones. In a new work, a conference on Virtual reality, archeology, and cultural
simulation of a full immersive virtual museum will be de- heritage, pages 91–96. ACM, 2001.
veloped to evaluate the behavior of various solutions on a [7] G. A. Satalich. Navigation and wayfinding in virtual
real application prototype. The goal will be to offer better reality: Finding the proper tools and cues to enhance
insights about convenient and effective combination of tech- navigational awareness. PhD thesis, University of
niques for both the core mechanics, necessary for this kind of Washington, 1995.
experience, that were identified and discussed in this paper. [8] W. Schweibenz. Virtual museums. The Development
Acknowlegements: This work was partially supported by of Virtual Museums. ICOM News Magazine, (3), 2004.
the Scan4Reco project funded by EU Horizon 2020 Frame- [9] F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler.
work Programme under grant agreement no 665091. Analysis of the accuracy and robustness of the leap
motion controller. Sensors, 13(5):6380–6393, 2013.
[10] M. White, N. Mourkoussis, J. Darcy, et al. Arco-an
architecture for digitization, management and
presentation of virtual exhibitions. In Computer
Graphics International, 2004. Proceedings, pages
622–625. IEEE, 2004.
[11] R. Wojciechowski, K. Walczak, M. White, and
W. Cellary. Building virtual and augmented reality
museum exhibitions. In Proceedings of the ninth
international conference on 3D Web technology, pages
135–144. ACM, 2004.
[12] J. Zara. Virtual reality and cultural heritage on the
Figure 5: First task completion times web. In Proceedings of the 7th International
Conference on Computer Graphics and Artificial
Intelligence, pages 101–112, 2004.