Towards Evaluating the Impact of Recommender Systems on Visitor Experience in Physical Museums

Abstract

Recommender systems have been used in physical museums to improve visitor experience; yet to assess their impact empirically, researchers often rely on the user experience criteria alone. In this paper, we examine the multidimensionality of visitor experience, speci cally the components and factors that shape it. We present a framework that is built on existing bodies of knowledge about visitor experience and that serves as a starting point to further evaluate the transformations caused by the integration of recommender systems in museums. We then conclude that current approaches to testing the impact of recommender systems on visitor experience should be expanded beyond evaluations of user satisfaction level to include the demonstrated aspects of visitor experience. This research aims to contribute to a more in-depth understanding of the di erences between visitor and user experience types as well as to address the need for a more comprehensive set of guidelines to evaluate how recommender systems enhance visitor experience in physical museums.

Keywords

• Visitor experience
• User experience
• Recommender system

-

A meaningful "visitor experience" (VX) in a physical museum is usually one that resonates with personal worldviews, domain knowledge, past experiences and situated physical contexts [ 6,7 ]. Recommender systems (RSs) aim to enhance the VX and have gained a wide popularity in museums because they can provide a more dynamic and personalised experience by presenting a subset of items that re ect individuals' interests and needs [ 3,11,13 ]. Moreover, RSs can support guidance and group visits, e.g., see CHIP (Cultural Heritage Information Personalisation) [ 26 ]. Up to now, the impact of RSs on VX has been evaluated from the perspective of user experience (UX), which generally focuses on the "user-product" interaction [ 9,25 ]. Although UX testing can o er appropriate methodologies for examining an RS-augmented experience and user satisfaction level, we propose that current approaches should be expanded to accommodate the wider context of VX, which frames the interaction of the individual, the RS itself and the physical museum space in which the visit takes place. We question whether the impact of RSs on VX can be established through UX tests alone.

In this paper, we examine VX-related concepts and suggest that an acknowledgement of the di erences between UX and VX can contribute to gathering more comprehensive and robust evidence about the ways RSs transform VX. In Section 2, we present an analytical framework for VX that encompasses the contexts that shape the museum experience, experience generators and VX categories. In Section 3, acknowledging the multidimensionality of VX, we discuss the limitations of available studies that attempt to enhance VX with RSs. We also consider how the gap in the existing studies can be lled with an extended evaluative framework that would dwell upon UX and VX criteria as well as reveal the impact of RSs on VX. 2

Conceptualising Visitor Experience

VX is often targeted when new digital technologies are introduced into the museum setting. In this section, we survey literature that seeks to de ne and/or contextualise VX. VX is multidimensional and complex because the concepts that constitute VX are strongly intertwined and vary greatly depending on the setting and personal characteristics of visitors. There have been a few attempts to categorise VX components [ 1,8,12,6,20,14,16 ]. Among them, we have found the Doering, Pekarik and Karns' categorisation to be exemplary because of their extensive empirical tests and wide coverage of VX components [ 6,20 ]. Doering, Pekarik and Karns identi ed four experience types: { Object experience is related to the material culture objects and their associated characteristics; { Cognitive experience addresses an intellectual aspect of VX and refers to knowledge, interpretations, interests and understanding; { Introspective experience alludes to fantasy, imagination, re ection and memories; { Social experience refers to social interaction, such as spending time with family and friends.

These experience types exist on the same plane and tend to in uence each other. For instance, researchers claim that there is often a negative association between object and cognitive experiences because the emphasis can be placed either on the object itself or on the description of that object [ 20 ]. Thus, when attempting to improve one aspect of VX, we can unintentionally manipulate other VX components. This interdependency between the components of VX and the concomitant propensity for unintended consequences is important to consider when conducting VX-related evaluations.

VX is driven by the peculiarities of a setting as well as the personal characteristics of a visitor. Falk and Dierking [ 7 ] de ne VX as \a continually shifting interaction among personal, sociocultural, and physical contexts". More specifically, the visitor's motivations [ 18 ], expectations [ 23 ] and goals [ 5 ] have been linked to the quality of VX. Motivations seem to be the most distant from the VX in uence as they carry subjective and situational characteristics from the Falk and Dierking's contexts [ 18 ], though goals have a tendency to change as the VX develops [ 5 ]. Based on this, the peculiarities of personal, sociocultural and physical contexts should be taken into consideration to have a more in-depth understanding of VX in a given setting. For instance, researchers who developed the adaptive guide HyperAudio had to reconsider the scope of functionality after their empirical tests because, although based on extensive literature review and conversations with curators, the visitors' needs in the targeted museum did not fully match the proposed functionality [ 21 ].

Considering the concepts associated with VX, we have populated an analytical framework for VX (see Fig. 1) based on the scholarship of those discussed above. Packer [ 17 ] presented an early synthesis of theoretical frameworks and

CONTEXTS THAT SHAPE MUSEUM EXPERIENCE

by Falk and Dierking, 2012 personal context e.g., age, gender, attitude, worldview, interests, background knowledge, past experiences MUSEUM EXPERIENCE sociocultural context e.g., language, culture, beliefs, community motivations by Packer and Ballantyne, 2002 learning and discovery, SR passive enjoyment, TO restoration, RA social interaction, EEN self-fulfilment G E C IEN expectations R EP by Sheng and Chen, 2012 EX easiness and fun, cultural entertainment, personal identification, historical reminiscences, escapism initial / emerging goals by Corredor, 2006 physical context e.g., architecture, spatial layout, museum collection, type of museum

VISITOR EXPERIENCE (VX) CATEGORIES

by Pekarik, Doering and Karns, 1999 object experience material culture objects and their associated characteristics cognitive experience gaining new knowledge, curiosity, interest, clarity and understanding, cognitive interpretations introspective experience reminiscence, reflection, imagination, memories social experience interaction with other visitors, spending time with companions, spending time alone Fig. 1. Analytical framework that demonstrates the origin and components associated with visitor experience showed the relations between situational characteristics, VX categories and bene cial outcomes. We want to revisit and extend Packer's framework to accommodate other ndings in the literature including the importance of personalised experience. Until validated, we cannot con rm that our framework is exhaustive; however, it does give some insights into available knowledge in the eld and the concepts associated with VX. When new technologies are introduced to improve VX, we need to make sure that the visitor is either accommodated on the same level or their experience is improved. 3

Evaluations of the Recommender System In uence on Visitor Experience With some insight into the VX categories and related components, we can now examine and identify potential limitations of projects aimed at improving VX with the RSs. When assessing RSs and similar personalised multimedia guides in physical museums, researchers often rely on questionnaire-based surveys about system usability, e.g., [ 27,15,24 ], and user satisfaction level, e.g., [ 10,24 ]. Although these studies are valuable for system performance evaluations [ 25 ], they capture little information about VX and its potential enhancement. For example, SMARTMUSEUM, a mobile recommender system, was evaluated on a modi ed System Usability Scale (SUS) [ 4 ] and was found easy to use [ 22 ]. However, the survey participants suggested integrating functionality that would support tourplanning and navigation within museums. This suggests that the usability scale can have certain limitations in indicating the required level of VX support. Indeed, a poor system performance may impose a barrier to a positive VX, yet it is not evident from available studies. Nevertheless, it can be misleading to assume that user satisfaction level with the system can reveal the full impact on the VX.

Following [ 25 ], UX evaluations are aimed at improving a product and thus relevant studies revolve around the experiences caused by the product. If we take a step further, we can analyse not only a new experience (UX) associated with the RS but also the RS-driven changes in the original experience (VX). This is important for understanding how RSs contribute to VX and ultimately gathering strong evidence that the RSs' impact is positive. To evaluate the UX with museum technologies, Pallud and Monod [ 19 ] used six phenomenological criteria related to VX. We believe that it is necessary to distinguish UX from VX, acknowledging that VX focuses on the \visitor-context" interaction [ 7 ] and that UX refers to the \user-product" interaction [ 9 ]. Furthermore, it is not enough to examine only one aspect of VX at a time because VX components interact and are a ected by each other. For instance, a personalised guide recommendation system, PGR, was aimed at mitigating information overload in the museum setting [ 10 ]. Although the PGR received positive feedback in the user satisfaction level survey, the studies would have bene ted from more extensive evaluations on how the resolution of the information overload issue with the PGR helped visitors improve other aspects of their experience.

There is clearly a lack of comprehensive evaluations of RSs and their in uence that would entail both UX and VX criteria. Othman et al. [ 16 ] presented two measurement scales, on UX and VX, and showed that multimedia guides can have a positive impact on four identi ed VX components, i.e., Engagement, Knowledge/Learning, Meaningful Experience, and Emotional Connection. However, some prominent impact could only be seen on the Engagement component. Similar studies need to be conducted with the RSs to show their actual in uence on VX. Moreover, to collect some statistical evidence that VX is improved with RSs, a comparative analysis should be based on two groups of visitors, those who browse the museum collection with and without technologies, to establish the objective e ectiveness of the system integration [ 19 ]. This was also mentioned by [ 10 ] who agreed that their studies were limited by the self-reported perceptions of the survey participants. In addition, Pallud and Monod [ 19 ] proposed that to capture VX more fully, researchers need to conduct semi-structured interviews rather than rely on the questionnaires alone. This reveals the scope of work that needs to be done in order to have some strong evidence that VX is improved with RSs.

From the available set of evaluations, we can conclude that the multidimensionality and mutual interrelatedness of the VX components is often ignored or used interchangeably with UX. This poses a question about the actual impact of RSs on VX. It is also important to present an expanded evaluative framework to clarify the di erences between UX and VX. Moreover, the participants' feedback should be collected from semi-structured interviews to explore potential changes in the VX categories with the integration of RSs in a physical museum as well as from questionnaires to show the in uence of RSs on the existing VX criteria. 4

Discussion and Future Work

This research paper aims to show the importance of discerning VX from UX in physical museums and related limitations of available evaluations of the impact of RSs on VX. It is challenging to understand VX and its transformations that occur with the integration of RSs by focusing on UX criteria alone. This is because the multidimensionality of VX includes factors not accounted for in the "user-product" interaction. In addition, from a more technical perspective, the distinction of UX and VX might also be bene cial. Considering that the user's bene t can change depending on the UX criteria utilised in the RS algorithms [ 2 ], there is a potential discussion to be raised on how VX components can be transferred into algorithms to provide more accurate and diverse recommendations in a physical museum.

The framework, presented in this paper (see Fig. 1), has been derived from an extensive literature review and aims to capture the complexity of VX. Until validated, the framework cannot be considered exhaustive. At the initial stage towards evaluating the impact of RSs on VX, we will carry out a survey to measure the prominence of VX categories before and after the RS prototype is integrated in the museum setting. Following Pallud and Monod [ 19 ], we also aim to conduct semi-structured interviews as a means of re ning the adequacy of the proposed framework and expanding the scope of a digitally augmented experience. Based on these studies, we hope to reveal the transformations that occur when VX is exposed to the in uence of technology, and to present our results in one extensive framework that would re ect a potential overlap of VX and UX components when the RSs are integrated.

References

1. Annis , S.: The museum as a staging ground for symbolic action . Museum International 38 ( 3 ), 168 { 171 ( 1986 )

2. Bohmer, M. , Ganev , L. , Kruger, A.: Appfunnel: A framework for usage-centric evaluation of recommender systems that suggest mobile applications . In: Proceedings of the 2013 International Conference on Intelligent User Interfaces . pp. 267 { 276 . ACM , New York, NY ( 2013 )

3. Bohnert , F. , Zukerman , I. : Personalised viewing-time prediction in museums . User Modeling and User-Adapted Interaction 24 ( 4 ), 263 { 314 ( 2014 )

4. Brooke , J.: SUS - A quick and dirty usability scale . In: Jordan, P. , Thomas , B. , Weerdmeester , B. , McClelland , I.L . (eds.) Usability evaluation in industry , pp. 189 { 194 . Taylor and Francis, London ( 1996 )

5. Corredor , J.: General and domain-speci c in uence of prior knowledge on setting of goals and content use in museum websites . Computers & Education 47 ( 2 ), 207 { 221 ( 2006 )

6. Doering , Z.D. : Strangers, guests, or clients? Visitor experiences in museums . Curator: The Museum Journal 42 ( 2 ), 74 { 87 ( 1999 )

7. Falk , J.H. , Dierking , L.D. : The Museum Experience Revisited . Left Coast Press, Walnut Creek, CA, 1st edn. ( 2012 )

8. Graburn , N.: The Museum and the Visitor Experience . Roundtable Reports pp. 1 { 5 ( 1977 )

9. Hellweger , S. , Wang , X. : What is User Experience Really: Towards a UX Conceptual Framework . CoRR ( 2015 )

10. Huang , Y.M. , Liu , C.H. , Lee , C.Y. , Huang , Y.M. : Designing a Personalized Guide Recommendation System to Mitigate Information Overload in Museum Learning . Journal of Educational Technology & Society 15 ( 4 ), 150 { 166 ( 2012 )

11. Jannach , D. , Zanker , M. , Felfernig , A. , Friedrich , G.: Recommender systems: an introduction . Cambridge University Press, Cambridge ( 2010 )

12. Kaplan , S. , Bardwell , L.V. , Slakter , D.B. : The restorative experience as a museum bene t . Journal of Museum Education 18 ( 3 ), 15 { 18 ( 1993 )

13. Konstan , J.A. , Riedl , J.: Recommender systems: from algorithms to user experience . User Modeling and User-Adapted Interaction 22 ( 1-2 ), 101 { 123 ( 2012 )

14. Kotler , N. , Rentschler , R.: Creativity and Interactivity: New Ways to Experience, Market and Manage Museums . Kenneth Myer Lecture in Art & Entertainment Management , Deakin University ( 2003 )

15. Ku ik, T., Stock , O. , Zancanaro , M. , Gor nkel , A., Jbara , S. , Kats , S. , Sheidin , J. , Kashtan , N.: A visitor's guide in an active museum: Presentations, communications, and re ection . Journal on Computing and Cultural Heritage (JOCCH) 3 ( 3 ) ( 2011 )

16. Othman , M.K. , Petrie , H. , Power , C. : Engaging visitors in museums with technology: scales for the measurement of visitor and multimedia guide experience . In: Campos, P. , Graham , N. , Jorge , J. , Nunes , N. , Palanque , P. , Winckler , M. (eds.) IFIP Conference on Human-Computer Interaction. pp. 92 { 99 . Springer Berlin Heidelberg ( 2011 )

17. Packer , J.: Beyond learning: Exploring visitors' perceptions of the value and bene ts of museum experiences . Curator: The Museum Journal 51 ( 1 ), 33 { 54 ( 2008 )

18. Packer , J. , Ballantyne , R.: Motivational factors and the visitor experience: A comparison of three sites . Curator: The Museum Journal 45 ( 3 ), 183 { 198 ( 2002 )

19. Pallud , J. , Monod , E. : User experience of museum technologies: the phenomenological scales . European Journal of Information Systems 19 ( 5 ), 562 { 580 ( 2010 )

20. Pekarik , A.J. , Doering , Z.D. , Karns , D.A. : Exploring satisfying experiences in museums . Curator: The Museum Journal 42 ( 2 ), 152 { 173 ( 1999 )

21. Petrelli , D. , Not , E.: User-centred design of exible hypermedia for a mobile guide: Re ections on the HyperAudio experience . User Modeling and User-Adapted Interaction 15 ( 3-4 ), 303 { 338 ( 2005 )

22. Ruotsalo , T. , Haav , K. , Stoyanov , A. , Roche , S. , Fani , E. , Deliai , R. , Makela, E., Kauppinen , T. , Hyvonen, E.: SMARTMUSEUM: A mobile recommender system for the Web of Data . Web Semantics: Science, Services and Agents on the World Wide Web 20 , 50 { 67 ( 2013 )

23. Sheng , C.W. , Chen , M.C. : A study of experience expectations of museum visitors . Tourism management 33(1) , 53 { 60 ( 2012 )

24. Tesoriero , R. , Gallud , J.A. , Lozano , M. , Penichet , V.M.R. : Enhancing visitors' experience in art museums using mobile technologies . Information Systems Frontiers 16 ( 2 ), 303 { 327 ( 2014 )

25. V aananen-Vainio- Mattila , K. , Roto , V. , Hassenzahl , M. : Towards practical user experience evaluation methods. Meaningful measures: Valid useful user experience measurement (VUUM ) pp. 19 { 22 ( 2008 )

26. Van Hage , W.R. , Stash , N. , Wang , Y. , Aroyo , L. : Finding your way through the Rijksmuseum with an adaptive mobile museum guide . In: Aroyo, L. , Antoniou , G. , Hyvonen, E., ten Teije , A. , Stuckenschmidt , H. , Cabral , L. , Tudorache , T. (eds.) The Semantic Web: Research and Applications . pp. 46 { 59 . Springer Berlin Heidelberg ( 2010 )

27. Wang , Y. , Aroyo , L.M. , Stash , N. , Rutledge , L. : Interactive user modeling for personalized access to museum collections: The Rijksmuseum case study . In: Conati, C. , McCoy , K. , Paliouras , G . (eds.) International Conference on User Modeling. pp. 385 { 389 . Springer Berlin Heidelberg ( 2007 )