ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 16 Evaluation Framework for Improving 360 Virtual Tours User Experience Roman Shikhri∗ Joel Lanir Lev Poretski rshikhri@campus.haifa.ac.il ylanir@is.haifa.ac.il lporetsk@campus.haifa.ac.il University Of Haifa University Of Haifa University Of Haifa Haifa, Israel Haifa, Israel Haifa, Israel ABSTRACT studies on virtual tourism demonstrated the emotional appeal of In the context of the ongoing COVID-19 pandemic and the series virtual tours for the users, reducing their psychological stress [6] of lockdowns it introduced, virtual tourism allows individuals to and positively influencing their psychological well-being [6]. More- experience cultural places and artifacts remotely from the safety over, virtual tours increase users’ desire to visit the actual place [3] of their homes. A popular form of virtual tourism is 360-degree and improve the learning outcomes and knowledge retention [1]. virtual tours: digital experiences that render a 360-degree audio- Notwithstanding the growing popularity and usefulness of virtual visual representation of the real-world environment on the users’ tourism, there is no unified conceptual framework that governs personal device. However, despite the growing popularity of 360 the design of 360-degree virtual tour experiences. Consequently, virtual tours, there is no unified conceptual framework for designing the usability and attractiveness of these experiences for the users such experiences. In our work, we analyzed multiple existing virtual vary widely from tour to tour, depending on the particular design tours and constructed a four-dimensional framework for virtual features and affordances implemented in a specific tour. This lack tour design. We discuss this initial framework, describe how we of consistency and unified design knowledge negatively affects plan to extend the current work-in-progress, and propose future user experience and satisfaction, and more thorough research is research directions. required [3]. Motivated by the existing gap in the knowledge on virtual tour CCS CONCEPTS design, in this preliminary work, we analyzed more than 40 existing virtual tours, evaluating them in the context of their affordances, • Human-centered computing → Ubiquitous and mobile com- features, and cues designed to facilitate the interaction between puting design and evaluation methods. the system and the user. The tours varied across multiple software platforms, interaction modalities, and informational content. Based KEYWORDS on the analysis of these tours, we have created a high-level design 360 Virtual Tours, Virtual Tours, Virtual Reality Tourism, Cultural framework that can guide the creation and evaluation of virtual Heritage tours. The design framework consists of four dimensions: naviga- tion, information presentation, proactiveness, and interactivity and 1 INTRODUCTION is described in Section 2: Design Framework For Virtual Tours. We aim to provide concrete guidelines for the creation and evalu- The ever-growing network speeds and computational capabilities ation of virtual tours. This would benefit the industry practitioners of modern computers make it possible for new types of tourism- to better leverage their design and provide enriching, pleasant, and associated activities and products. One of the most popular virtual useful experiences for virtual visitors. tourism directions is 360-degree virtual tours – digital experiences We plan to build on the created framework and the newly formed that render a panoramic 360-degree audio-visual representation knowledge to extend the research in the following directions: of real-world environments, such as museums, outdoor locations, or exhibitions, on users’ personal digital devices. Usually, these (1) Validating and evaluating the design framework and its di- digital environments possess a certain level of interactivity, such mensions, as allowing the user to choose a particular location within the (2) Creating a demo 360 virtual tour that implements different place or changing one’s angle of view, looking around as if they design dimensions, are physically there. Thus, the designers’ aim is to emulate an (3) Assessing usability and user-experience of the created expe- experience of visiting the real physical place and enrich it with rience. interesting and innovative interaction techniques and modalities In the following sections, we describe the design framework and [4]. layout our proposed research direction on this topic. The ongoing COVID-19 pandemic put the traditional tourism industry in gridlock and further popularized virtual tours [6]. In the conditions where it is impossible to physically visit cultural 2 DESIGN FRAMEWORK FOR VIRTUAL locations and exhibitions, virtual tourism can partially recreate the TOURS experience by allowing the users to remotely view and interact To construct our design framework, we first identified over 40 ex- with tourist destinations from the comfort and safety of one’s home isting virtual tours for the initial review. After reviewing each tour, [2]. In addition to the convenience and cost-effectiveness of vir- we subsequently reduced the sample to the 10 most comprehensive tual tours comparing to their physical counterparts [5], previous 360-degree virtual tours, which we analyzed using the qualitative Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 17 Roman Shikhri, et al. techniques of design review and deconstruction. Based on the anal- as Google Maps. Other spatial orientation affordances com- ysis, we define a virtual tour as an information system aimed at monly used in virtual tours are the panoramic overview of presenting the user with a panoramic 360-degree view of a cul- the environment, aerial top-down view of the points of in- tural heritage site, consisting of static images and/or dynamic video terest, simple lists of available places and shortcuts, and a sequences, with multiple points of interest mapped within the re- visual showcase of key or popular points of interest using sulted virtual environment. The users can move between the points their snapshot images. of interest and browse them at their discretion. Often, the accompa- (4) User Controls and Input Modalities: the way in which the nied multimedia information is available at each point of interest: users’ input and interaction and navigation control mechan- text snippets, images, sound, and visual cues, etc. ics are implemented in a virtual tour. The controls may in- Our analysis allowed us to detect the common and unique pat- volve physical input devices, such as mice, controllers, or key- terns of interaction design used in the reviewed virtual tours. We boards to support user interaction. Another control modality were able to classify the identified cues and techniques into four is onscreen (soft) controls, for example, a set of the on-screen distinct conceptual dimensions: navigation, information presenta- directional arrow, touching or clicking on which executes a tion, pro-activeness, and interactivity. In addition, we were able to moving command. Another common control type consists identify the particular affordances that comprised each dimension. of visually marking the points of interest, with a mark serv- Below, we describe our findings in detail. ing as the interactive anchor to this point. For example, the Navigation dimension includes all affordances built into a vir- user may click the hovering dot on top of the exhibit, which tual tour that supports the user’s ability to comfortably navigate instantly transfers him or her to this exhibit. and move around the environment. In particular, the following A second conceptual dimension in our framework is Infor- affordances were identified in this conceptual dimension: mation Presentation which refers to ways in which the system presents the information available to the users in the virtual tour. (1) Movement Experience: the ability of the user to fluidly move The following particular information presentation affordances are around the space, experiencing a smooth transition between commonly present in the reviewed tours: one point of interest to another, as opposed to instantly (1) Visual Information Cues - the visual elements that let the appearing in the chosen point of interest within experiencing users know about available information, options, and interac- the transition. tion opportunities. Visual cues an important design element, (2) Freedom of Movement - how many options and places the since it supports the users’ spatial cognition, which may users can choose to move to within the tour’s environment. be significantly reduced in the virtual experiences that lack This affordance depends on the number of points of inter- physical cues. Consequently, visual cues should be paid par- est that the user can choose to move to, and the sheer size ticular attention throughout the virtual tour. Visual cues can of the tour’s reconstructed environment. The more points be implemented in the form of clickable hotspots, labels, di- are present in the environment, and the bigger is its size, alog windows, hovering media elements (2D/3D/embedded the more opportunity the designers have to increase the images), etc. The four major attributes of visual cues are users’ freedom of movement. Thus, we define the ratio of color, form, depth, and movement. POI count/space size as a key determinant of the ability to (2) Multimedia support - in addition to the visual cues, virtual explore and navigate the space. tours can include audio-visual and textual elements, for ex- In particular, the following levels of freedom of movement ample, guides, items’ descriptions, and additional imagery, are possible. The lowest freedom of movement is in the tours audio cues and examples, etc. with only a Single POI in each location (essentially mak- ing it a "no movement" tour with no transition between The third conceptual dimension in our framework is Proactive- virtual points), which can fit really small exhibitions. The ness. This concept relates to the degree to which the system aids low-medium freedom of movement has several POIs in each the user to perform or follow desired actions and mediates his or location, and that is often associated with discontinuous nav- her experience. The completely proactive system will take all the igation and unnatural transitions between points of interest. decisions and actions from the hands of the users, similarly to a Low/medium freedom of movement thus can negatively af- tourist guide managing her tourist groups in a tightly controlled fect the user experience. A high-level ratio indicates that tour around the cultural locations. Conversely, the system that there are many possible moves throughout the room, mean- lacks proactiveness will resemble the users freely sauntering up the ing that such tour will be experienced as having more or less location, completely unmediated by its personnel. In our analysis, "continuous transition". we were able to identify the two types of proactiveness: (3) Spatial Orientation: the design aspects of the virtual tours (1) Navigation proactiveness: the system mediates the users’ tran- that support one’s sense of direction and location within sitioning along with the points of interests, trying to predict the environment. One of the common ways in which spatial the users’ desired routes and movement, rather than react- orientation of the user can be supported is the inclusion of ing to their explicit commands. In terms of visual navigation the persistent mini-map, where the user sees his or her loca- guidance, virtual tour’s navigation needs to be supportive, it tion in the environment constantly updated in the top-down should motivate the visitor to proceed to the next accessible schematic representation of the environment, similar to how hotspots of the next point of interest by showing possible to the experience provided by the navigation software, such movements. Additionally, virtual tours may predict the user’s Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 18 Evaluation Framework for Improving 360 Virtual Tours User Experience next moves, based on the preferences and past movement we will examine the following measures comparing them between patterns. conditions: (2) Interaction proactiveness: the system mediates the interac- • Behavior - We will measure user behavior within the tours ex- tions between the users and the active components of the amining measures such as total time spent in a tour, number virtual tour - often clickable, such as information compo- of locations in which the user stopped, number of exhibits nents, audio-visual cues, message dialogues, and multimedia. the user interacted with, etc. The fourth dimension in our framework is Interactivity which • Usability - We will examine the perceived usability of the describes the extent to which users can interact with the system. website using an analysis of the think-aloud and of user The following types of interactions are possible. behavior as seen in the videos. • User Experience (UX) - We will examine user experience (1) Interaction with exhibits - textual annotation and augmen- using the UEQ questionnaire that will be handed out to tation, measurement of exhibits, and manipulation of 3D participants after each condition. moveable figures in the space. The results of the study will help to inform designers of the effect (2) Onboarding experience- a set of introductory screens and of the affordances stated in our framework on user behavior and flows that gradually guides the user, introducing the avail- experience in 360-degree virtual tours. Furthermore, it will be used able options, interactions, and content. Onboarding also in- to refine and validate the framework to better understand the key cludes contextual help, which appears during the tour when and important features of such tours. the particular interaction or content becomes available to the user. 4 SUMMARY Virtual tours have developed from providing motionless 360-degree 3 EXAMINING THE EFFECT OF DIFFERENT content to supporting proactive and immersive experiences. The TOUR AFFORDANCES ON USER market for virtual tours expands as technology capabilities mature EXPERIENCE and more and more people use these tours as a way to remotely experience cultural heritage sites. Still, the design of these tours is A good user experience is a central aspect of the success of inter- critical for their user experience. Better designed virtual tours will active products. Thus, Our main research question asks what are allow organizations to attract more users and provide immersive the effects of various affordances of 360-degree virtual tours on the and personalized experiences to broader and diverse crowds. Our behavior and user experience of virtual visitors? Our premise is that future research will examine key elements of 360-degree virtual a better understanding of how different features and affordances in tours and measure their influence on the user experience, behavior, virtual tours affect behavior, usability and user experience would and usability. Consequently, we will develop a unified conceptual enable better design of such tours. We plan to examine several framework for designing virtual tour experiences, as they gain key affordances listed in the framework above. We will do this by popularity in the shade of the Covid-19 pandemic and its impact designing and creating a virtual tour and comparing how users use on tourism. and experience different variants of this virtual tour differentiated by selected key features. REFERENCES To create the virtual tour we will use the 3D-VISTA virtual tour [1] Rahman Abidin, Nunuk Suryani, et al. 2020. Students’ Perceptions of 360 Degree software. We chose this platform after surveying multiple software Virtual Tour-Based Historical Learning About the Cultural Heritage Area of the options and picked this one due to its ability to easily alter the main Kapitan and Al-Munawar Villages in Palembang City. International Journal of Social Sciences and Management 7, 3 (2020), 105–112. affordances mentioned above. We will be using an Insta360 One R [2] Andreas Rio Adriyanto and Ananda Risya Triani. 2015. 360 Virtual Reality camera which enables us to shoot 360 5.3k resolution photos and Panorama Of Indonesia Tourism. Bandung Creative Movement (BCM) Journal videos which automatically stitches the footage into single files. 2, 1 (2015). [3] AONAW Mohammad and Hafiz Ismail. 2009. Development and evaluation of an The planned virtual tour will take place in the cultural heritage interactive 360 virtual tour for tourist destinations. (2009). location of Stella Maris Monastery, located on the slopes of Mount [4] M. White N. Mourkousis F. Liarokapis M. Sifniotis A. Basu Petridis, Panagiotis Carmel in Haifa, Israel. and C. Gatzidis. 2005. Exploring and interacting with virtual museums. Proc. Of Computer Applications and Quantitative Methods in Archaeology (CAA) (2005). After creating the tours, we will run a remote user study to ex- [5] M. White N. Mourkousis F. Liarokapis M. Sifniotis A. Basu Petridis, Panagiotis and amine user’s behavior and experience with virtual 360-degree tours. C. Gatzidis. 2008. Presence-centered assessment of virtual museums’ technologies. DESIDOC Journal of Library Information Technology 28, 4 (2008), 55–61. Because of the COVID-19 pandemic restrictions, conducting labora- [6] Ting Yang, Ivan Ka Wai Lai, Zhao Bin Fan, and Qing Min Mo. 2021. The impact of tory studies is challenging and somewhat problematic. However, for a 360° virtual tour on the reduction of psychological stress caused by COVID-19. an evaluation of a virtual product such as 360-degree virtual tours, Technology in Society 64 (2021), 101514. remote testing is quite natural and has the benefit of increased external validity. Thus, we will ask participants to explore the tours from the comfort of their homes. We will record the participant’s screen, and analyze all participants’ interaction within the tours using screenshots as well as the system logs. In addition, we will use the think-aloud protocol, asking the participant to comment out loud while experiencing the virtual tour. This will enable us to better understand participant’s opinions and experiences. Thus, Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).