Gamifying the museological experience António Coelho*1,3, Pedro Cardoso1,2,3, Maria van Zeller1, Liliana Santos1, José Raimundo1,3, Roberto Vaz1 1 FEUP – Faculty of Engineering, University of Porto, Rua Dr. Roberto Frias, 4200-465 Porto, Portugal 2 FBAUP – Faculty of Fine Arts, University of Porto, Avenida Rodrigues de Freitas, 265, 4049-021 Porto, Portugal 3 INESC TEC – Institute for Systems and Computer Engineering, Technology and Science, Rua Dr. Roberto Frias, 4200-465 Porto, Portugal Several museums are already using virtual and augmented ABSTRACT reality technologies to provide new experiences to their visitors. Museums continue to exert fascination in their visitors. However, The Smithsonian museum’ AR Bone Hall Exhibit [1] and the the new generation of visitors expects museological experiences project Modigliani VR: The Ochre Atelier [2] from the Tate that promote their active participation. It is in this context that Modern Museum, are some examples. Nonetheless, one of the games and the gamification of such experiences capitalize on digital challenges museums face today in this context is the experiential learning by experimenting and enacting with in-game simultaneous navigation and exploration of both these virtual and embedded artefact surrogates and know-how. physical spaces. Mixed reality technologies aim at bridging such In this article, we present four distinct projects that aim to enhance gap. However, every context has its very particular characteristics the visitors’ experience in museums and green spaces, and also that need to be addressed with specific tools and designs in order their effectiveness in informal learning. In the first project, to promote participation and stimulate creative practices in gamification is used in combination with Augmented Reality to museological spaces. provide a more engaging experience in a boat museum. The drive In the broad field of the Ludification of Culture, we can focus of this experience is the metaphor of the stickers album collection on the area of games or the area of playful interaction [3]. In this to unleash the relevant information of the key-artefacts of the particular work, we focus on the more structured use of games, museum collection. The second and third projects focus on the use both full-fledged games and gameful design. of pervasive games, more specifically location-based games, to This document is divided into four main sections, presenting enhance the visitors’ experience and informal learning in a natural four examples of applications in museums. They were developed park and a botanical garden, respectively. The second project by members of the Graphics, Interaction and Games research presents the concept of a mobile app for outdoor nature laboratory, under the same supervision. The projects took place experiences. The drive for the experience in the third project is the between 2016 and 2018. The first example shows an application narrative that intertwines specific locations in the botanic garden of gameful design, which is commonly named Gamification, that and a story inspired by the same place. Finally, in the fourth is the use of game design elements or game design techniques in project, we focus on the potential of technology to provide non-game contexts. For the second and third examples, we use accessibility in museums for people with special needs or games that extend the players’ limits of time and space, i.e., the disability, focusing more specifically on blind visitors. Huizinga’s “magic circle” [4], to the context of the user, on what is called a pervasive game. In the fourth and last example, we approach the need of the museums to be inclusive, that is, Keywords: Gamification, Augmented Reality, Pervasive games, accessible to all visitors including those with particular needs or/ Location-based games, Accessibility. and disabilities. 2 EXAMPLE 1 - GAMIFICATION IN MUSEUMS Index Terms: •Human-centered computing~Ubiquitous and mobile Museums consolidate the heritage of a country or a given region, computing~Empirical studies in ubiquitous and mobile computing interlinking history, art, science and the territory. As an •Applied computing~Education~Interactive learning inalienable landmark in the history of Portugal, the first example environments is about Maritime Museums, and most particularly, the exploration of traditional boats. We propose a gamified approach based on the concept of 1 INTRODUCTION sticker album collection and its integration in an Augmented The use of digital technology can undoubtedly assist and create Reality (AR) mobile application. The concept of sticker album new experiences for visitors in museums while promoting their collection is quite familiar to most people, mainly from their active participation. The use of mobile devices during museum youth, and is the central dynamic of the gamification design, visits has great potential for informal learning, playing an engaging the learner to collect more stickers and progress in the important role in finding, reading and interpreting museological exploration of the museum. As a pervasive solution, we do not use artefacts while being non-invasive. However, the continually physical support, but instead, a mobile application to provide the evolving hardware asks for a constant search for new and learning experiences by uncovering the stickers using AR over the alternative ways to use such devices, which in turn asks for the museum collection, in order to enhance the knowledge transfer design and development of new software. An example is the and rewarding. In 2018, we developed a prototype for a boat museum guide being assisted and sometimes replaced by diverse museum where digital stickers are obtained by overcoming smartphone applications. challenges in the context of the exploration of the boats in the museum rooms [5]. One of the main principles of gamification is the engagement * acoelho@fe.up.pt cycle. Thus, we have developed a progressive activity that Copyright © 2020 for this paper by its authors. Use permitted under engages the player into small iteration cycles in order to provide Creative Commons License Attribution 4.0 International (CC BY 4.0). mastery. Each boat has a predefined number of stickers that, being the epic meaning of the experience, and informal learning is initially drawn as empty slots, show the number of challenges to fostered on the challenges associated with the locked stickers. At solve (this number can vary) in order to collect the boat. Figure 1 the same time, progression is achieved by collecting the missing shows the typical iteration cycles for a specific boat. stickers. The AR layer is the key element that provides the connection between this heritage and the gamified learning experience. 3 EXAMPLE 2 – LOCATION-BASED GAMES IN PARKS: THE CASE OF GAIA’S BIOLOGICAL PARK Digital mobile games can offer engaging experiences and increase communication effectiveness concerning cultural and natural heritage. Location-based serious games allow us to overcome the barriers of space and enables their players to explore the real world around them while playing a game with educational or cultural purposes. A prototype of a mobile app was developed for the Gaia’s Biological Park [6], in Portugal, to help in making science communication engaging for visitors and in improving the overall outdoor visitor experiences. The concept is to have a mobile app to promote and disseminate the natural heritage in a tourism context, with a map, the user’s location in real-time, points of interest with geolocalized information (GPS), and small serious games about the green spaces visitors are experiencing. This project started in 2016. Figure 1: Engagement cycle – iteration on a boat. One of those games, the virtual animal detector [7] (figure 3) is The first sticker provides a brief description of the boat and a “treasure hunt” and collection game about the animals that live challenges the user to identify a specific component of the boat in freely in the park. Some of them are difficult to see in the wild (figure 2). The user will need to find it using the AR tool. In the on account of their behavioural habits. With this game, visitors case of difficulty, a “help button” can be pressed, and a text hint can learn more about these species. Players must explore the park will be provided (about its location). If the user is still not able to to find all the virtual game animals and collect pictures of them all find it, another more detailed hint (like a drawing of the boat until they complete the collection. An in-game radar detects the component) will be additionally provided. This provides mastery supposed location of a virtual animal when the player is near that to proceed. location and displays the information on that species. When discovered, a species gets registered in a collectables board, and some information about it can be consulted on-demand. Another example is the game of the jay [6], that tries to communicate the dynamics of an oak tree forest ecosystem through its gameplay. The player controls a jay that collects and plants acorns. To make the sprouts grow, the player must collapse clouds to make it rain. That also puts down wildfires. To win the player must completely reforest the scenery. Figure 2: First sticker with brief description of the boat. As the user identifies the boat component, additional information is provided in the AR layer, showing specificities and details on top of the original museum boat. This launches the second challenge, a simple quiz that assesses some information provided previously in AR or on the site. After collecting this second sticker, the boat image is complete in the album, and a new multimedia content is available as a reward: a three- dimensional model of the boat and a small animation of how the Figure 3: The virtual animal detector [7]. boat is constructed. Thus, the user progresses through each level by surpassing each challenge, gaining access to digital stickers These games’ goal is to amplify and enrich the visitors’ (and associated multimedia content) as a reward, making the experience, augmenting our perception of the natural world so we exploration of the museum and the real apprehension of can better understand it and protect it. Although the prototype was knowledge more attractive. The AR layer has a significant role in designed for the Gaia’s Biological Park, its concept can be making the connection with the heritage of the museum. adapted and applied to other parks, zoos, botanical gardens and In conclusion, we propose the concept of the sticker album other green spaces. Presently, the AR implementation into this collection integrated with AR to improve the museological app concept is being studied, with some promising preliminary experience. The museum heritage is the anchor point that provides results. 4 EXAMPLE 3 – LOCATION-BASED GAMES IN MUSEOLOGICAL interactor/ player from the locale itself, but one that promotes a SPACES: THE CASE OF THE UNIVERSITY OF PORTO’S stronger bond between them. We found that one way to do that is BOTANICAL GARDEN to use that mobile application to focus their attention on the The University of Porto’s Botanical Garden is a place interspersed physical world and not on their smartphones (or other devices), be by Literature, Botany and History. The garden was home of the it for navigation, information seeking activities or for moments of celebrated Portuguese writer Sophia de Mello Breyner Andresen, contemplation. hence a place with significant influence on her work. In our 5 EXAMPLE 4 – ACCESSIBILITY FOR THE BLIND VISITORS discernment, a visit to the Botanical Garden is not complete without acknowledging these different facets as complementary Despite the growing concern about increasing accessibility for dimensions that characterise this site. visitors with visual impairments [8], [9], [10], [11], only 5,5% of With this in mind, we developed an experience for visitors to be blind and visually impaired people visit museums in Europe [12]. acquainted with those facets in the form of a location-based game Patrons with total and partial visual impairments represent a that articulates educational content in Botany with Sophia’s significant potential market for museums [13], and positive famous story “O Rapaz de Bronze” (the bronze boy). In this game economic effects are expected with the implementation of (that resorts to GPS), players progress as they encounter particular assistive technologies to promote access to exhibitions [14], Points of Interest (POI), moving in a specific sequence between besides enhancing their experience of the visit. them by solving challenges hinted from previous successful steps. Haptic interfaces can take advantages of the dynamic nature of In each POI, players are confronted with challenges that can be the kinesthetic sense and generate forces, allowing the exploration solved by inspecting their surroundings, ergo creating a of virtual copies of museum and gallery collections [15] [16]. The relationship between Literature and Botany, all found in situ. We Museum of Pure Form project allowed visitors to touch and found this relationship to be crucial for developing player experience the properties of sculptures belonging to the collection engagement with the locale, and to instigate location-awareness in of multiple European Museums using an arm exoskeleton and a players. haptic interface for fingers [17], [18]. Visitors are required to pick Exploratory visits with the garden staff were very welcome and up a pen of the device in order to interact with the Geomagic necessary to build a proper experience. During the day of testing,1 Touch System [19] and to allow haptic experiences of objects we resorted to participant-observers for collecting information from the National Museum of Transylvanian History of Cluj- from participants while they were playing, acting as facilitators Napoca. Another project, developed in the context of the for solving technical issues and identifying unexpected problems AMBAVis project and, in collaboration with Manchester Museum in situ. Through these methods and by analysing player surveys, and Gallery Belvedere, allows patrons to explore artefacts, we concluded that players were engaged with the experience, that augmenting the experience by providing vocal information about was well-accepted amongst families and groups of children and the object material and characteristics [14]. teenagers. For them, learning through a location-based game Other museums are using 3D printing technologies with proved to be a fun and enriching. However, specific interaction embedded sensors to provide digitally augmented touch replicas. design issues arose for our consideration in future work. They offer tactile and hearing multisensory experiences that help the visually-impaired visitors formulate mental images of the objects while providing crucial contextual information, such as the Tooteko project [20], developed for the Correr Museum. Those technologies are being used as well to reconstruct the object at the date of its origin, so visitors can learn about how an Egyptian Cat Sarcophagus was 2500 years ago [21]. Contrary to digitally augmented touch replicas, gesture-based interactive tactile reliefs do not have physical sensors attached to detect users’ activities. Instead, their interaction is tracked by computer-vision or other non-intrusive systems, in order to enrich tactile reliefs with audio or additional multisensory information. Researchers developed a gesture-based interactive audio guide for the interpretation of the painting The Kiss [22] and to assist the tactile exploration of the painting Madonna with Child and Angels [23]. Figure 4: Location-based game in the botanical garden. Besides helping to create new ways for disabled visitors to know about museum collections interactively, technology is also In conclusion, navigating in the Botanical Garden as a being used to provide them with better physical access, indoor museological space requires visitors to be attentive to their navigation and wayfinding in such environments. The result is an surroundings. It is in those surroundings that is the motive for enhancement in their ability to move independently through their visit, for their presence. Operating a mobile application in different galleries, without the aid of guides or escort. As an this context needs to be an experience that does not alienate the example, a mobile application was developed at the National Science Center to assist sightless visitors’ navigation by 1 presenting step-by-step audio instructions [24]. A similar interface This pilot was developed within the scope of the project was proposed for the Museum of the Lighthouse for Blind [25]. BEACONING: Breaking Educational Barriers with Contextualised, Pervasive and Gameful Learning. This case study 6 CONCLUSION was tested on the 10th of February, 2018. Beaconing was an Museums maintain the same epic meaning that attracts visitors. H2020 project involving 16 universities and partner companies, New media and technology can complement and not replace the INESC TEC included, led by Coventry University mission of these institutions. The presented projects follow (http://beaconing.eu/). different lines of research for creating frameworks for the design and the development of AR applications and games. These key studies have contributed with insights on how to make better or turn. Disability Studies,” Disabil. Stud. Q., vol. 33, no. 3, pp. 1– more enriching museological experiences. The visitors’ 20, 2013. experience should be the main drive for technological [11] European Commission, “Memo: 19 destinations across Europe breakthroughs in museums, and games, gamification, and mixed reality technologies can provide improvements in such win awards for accessible and sustainable tourism,” no. experience. However, these technologies must be balanced, so November, 2013. they will not distract the visitors from the museum collection but, [12] K. Dash and G. Grohall, “Economic impact of creating and instead, enhance their experience with these artefacts. exhibiting 3D objects for blind and visually impaired people in Furthermore, these new technologies promote the inclusive museums,” pp. 1–81, 2016. museum where all visitors can have a great experience regardless of any disability or particular needs. [13] S. Mesquita and M. J. Carneiro, “Accessibility of European museums to visitors with visual impairments,” Disabil. Soc., ACKNOWLEDGMENTS vol. 31, no. 3, pp. 373–388, 2016. The BEACONING research project, has received funding from [14] Project AMBAVis, “Inclusive technologies in museums,” 2017. the European Union's Horizon 2020 - The EU Framework [15] J. P. Fritz and K. E. Barner, “Fritz, Jason P.; Barner, Kenneth, E. Programme for Research and Innovation 2014-2020, under grant (1999). "Design of a haptic data visualization system for people agreement No. 687676. The researchers are supported by the Operation NORTE-08-5369- with visual impairments,” IEEE Trans. Rehabil. Eng., vol. 7, no. FSE-000049 supported by Norte Portugal Regional Operational 3, pp. 372–384, 1999. Programme (NORTE 2020), under the PORTUGAL 2020 [16] F. Biocca, K. Jin, and Y. Choi, “Visual touch in virtual Partnership Agreement, through the European Social Fund (ESF). environments: An exploratory study of presence, multimodal The work was also supported by FCT-Austin grant Ref. interfaces, and cross-modal sensory illusions,” Presence PD/BD/142893/2018 and by FCT, reference SFRH/BD/137169/2018, financed by the Human Capital Teleoperators Virtual Environ., vol. 10, no. 3, pp. 247–265, Operational Programme (HCOP), under the European Social Fund 2001. and national funds of the MCTES. [17] G. Jansson, M. Bergamasco, and A. Frisoli, “A new option for the visually impaired to experience 3D art at museums: Manual REFERENCES exploration of virtual copies,” Vis. Impair. Res., vol. 5, no. 1, pp. [1] D. Marques, Realidade Aumentada em Exposições de Museus - 1–12, 2003. Experiências dos Utiliadores. Caleidoscópio - Edição e Artes [18] M. Carrozzino and M. Bergamasco, “Beyond virtual museums: Gráficas, SA, 2018. Experiencing immersive virtual reality in real museums,” J. [2] Tate Modern, “Modigliani VR: The Ochre Atelier – Behind The Cult. Herit., vol. 11, no. 4, pp. 452–458, 2010. Scenes | Tate.” [Online]. Available: [19] R. Comes, “Haptic Devices and Tactile Experiences in Museum https://www.tate.org.uk/whats-on/tate- Exhibitions,” J. Anc. Hist. Archaeol., vol. 3, no. 4, 2016. modern/exhibition/modigliani/modigliani-vr-ochre-atelier. [20] F. D’Agnano, C. Balletti, F. Guerra, and P. Vernier, “Tooteko: [Accessed: 16-Apr-2020]. A case study of augmented reality for an accessible cultural [3] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From Game heritage. Digitization, 3D printing and sensors for an audio- Design Elements to Gamefulness : Defining ‘ Gamification ,’” tactile experience,” Int. Arch. Photogramm. Remote Sens. Spat. 2011. Inf. Sci. - ISPRS Arch., vol. 40, no. 5W4, pp. 207–213, 2015. [4] J. Huizinga, Huizinga, J. (1949) Homo Ludens: A study of the [21] A. Reichinger, S. Schröder, C. Löw, S. Sportun, P. Reichl, and play-element in culture. London, Boston and Henley: Routledge W. Purgathofer, “Spaghetti, sink and sarcophagus: Design & Kegan Paul Ltd. Original edition, 1944. explorations of tactile artworks for visually impaired people,” [5] A. Coelho and L. M. Costa, “The integration of augmented ACM Int. Conf. Proceeding Ser., vol. 23-27-Octo, no. October, reality and the concept of sticker album collection for informal 2016. learning in museums,” Commun. Comput. Inf. Sci., vol. 725, pp. [22] A. Reichinger, A. Fuhrmann, S. Maierhofer, and W. 107–115, 2017. Purgathofer, “Gesture-based interactive audio guide on tactile [6] L. Santos, D. Pereira, P. Beça, and A. Coelho, “Aplicação móvel reliefs,” ASSETS 2016 - Proc. 18th Int. ACM SIGACCESS Conf. para divulgação do património natural no turismo,” Tur. Comput. Access., pp. 91–100, 2016. Desenvolv., vol. 27/28, pp. 1461–1473, 2017. [23] Y. Buonamici, F., Furferi, R., Governi, L., & Volpe, “Designing [7] L. Santos, D. Pereira, R. Nóbrega, P. Beça, and A. Coelho, the architecture of a preliminary system for assisting tactile “Jogos sérios digitais para a promoção do património natural – o exploration of bas-reliefs,” J. Des. Res., vol. 15, no. 2, pp. 110– caso de um jogo de exploração de um parque na busca dos seus 127, 2017. animais [poster + demo] {BEST DEMO},” Vj 2016, vol. 33, no. [24] D. Jain, “Pilot evaluation of a path-guided indoor navigation 2015, p. 2991, 2016. system for visually impaired in a public museum,” ASSETS14 - [8] F. Candlin, “Blindness, art and exclusion in museums and Proc. 16th Int. ACM SIGACCESS Conf. Comput. Access., pp. galleries,” Int. J. Art Des. Educ., vol. 22, no. 1, pp. 100–110, 273–274, 2014. 2003. [25] A. Meliones and D. Sampson, “Indoor blind navigator: A use [9] Instituto Português de Museus, Museus e Acessibilidade. 2004. case for self-guided tours in museums,” ACM Int. Conf. [10] A. Cachia, “Talking blind: disability, access, and the discursive Proceeding Ser., vol. Part F1285, pp. 17–20, 2017.