Our experiences of artworks in galleries and museums are generally passive. While this is a widely adopted practice for preservation purposes, it hinders the engagement potential with younger visitors. However, we could adopt novel technologies to change this, and augmented reality (AR) is one of the most promising because of its' capacity to increase the engagement and add value to the experience. In this paper, we present the development of a mobile augmented reality application designed based on "Google Tango" technology. The playful experience provides by the AR application is divided into three steps, which allow users to colour a contour of a 3D object on a paper, scan it with the AR application and embark on a treasure hunt to find a 3D object (statue) somewhere in the gallery on which the coloured contour would wrap. The paper demonstrates that such a concept is feasible on a modern camera phone with Google Tango technology creating a playful experience in the gallery.
Galleries and museums can be defined as informal learning settings providing various artworks,
objects, exhibits, and programs developed around subjects of art, history, science and
archaeology [
The use of augmented reality (AR) to display artworks or museum content is a well-established
concept within the ICT and Human Computer Interaction (HCI) research community. A number
of applications aimed at addressing the needs of the cultural sector has emerged predominantly
as either guides for cultural sites [
In this paper we development an AR concept Playing with the artwork we designed base on our previous publication [9]. The game concept aims to enhance the user experience within the existing physical gallery exhibition through a mobile augmented reality treasure hunt style game.
The game starts by assigning users a contour drawing which they are asked to colour on a physical paper. Users do not know what object they are colouring, but are made aware that these segments, if placed correctly, should wrap up on a 3D object (statue) (see Figure 2).
Once they have their contour coloured and scanned, each user embarks on a treasure hunt through the gallery/museum exhibition or city and tries to fit the coloured contour to various artwork (e.g. sculptures, buildings, paintings, etc.) by pointing their mobile phone camera at the object and pressing a scan button. After scanning, active objects—the ones that have an appropriate contour available and are incorporated into the game—are highlighted with an orange mesh (see Figure 2a). If the user thinks the object matches the coloured contour, they tap on the object.
If the contour does not match, the user is given extra information to lead them to the artwork or simply encourage them to keep exploring. If the matching artwork is identified, the coloured patches from the contour wrap onto the artwork creating an unique personalised version of the artwork (see Figure 2). User also will be able to capture and save their personalised version of the artwork to the image gallery (Figure 1).
After the matching object is been found, users are also able to claim another contour drawing and embark on the next mission within the treasure hunt game.
This paper presents a feasibility study if the proposed game concept can generate a convincing and playful experience when we use of-the-shelf mobile phone with Google Tango technology. We do this through the development of the proposed game for which we provide insights, highlight key challenges and present key methods that made the implementation possible.
To implement the proposed game and create a convincing illusion of curating users’ own artwork the system needs to be capable of: (i) overlaying 3D models over a predefined objects in the gallery space with high precision, (ii) generate a texture map from a coloured contour drawing where uncoloured segments of the paper are considered as transparent, and (iii) operate in environments which can not be modified (the active 3D objects cannot be moved after being placed as well as markers cannot be placed on objects and around them).
The precision is very important for a good experience, because any misalignment between the semi-transparent coloured texture and a real object is very noticeable breaking the illusion (a) The object is highlighted with a mesh when scans. (b) Coloured texture wrapped around the object. and aesthetics of creation. Supporting transparency when capturing the texture is also vital because we build the illusion of our own version of the artwork by mixing the textures of physical objects with the one coloured by the user (digital texture).
In order to overlay a 3D model over a physical object (e.g. sculpture) the system needs to perform camera pose tracking that is able to localise based on predefined environment maps. Due to the fact that it is not desirable to modify the environment of a gallery, tracking needs to be performed using existing natural features only. As we would like to track camera pose in relatively large indoor environments we decided to use Google Tango tracking system in order to curate area descriptors (3D maps of environment), localise the camera and track the camera pose.
Our application works with absolute coordinate system that is selected at the time of capturing a 3D map of the environment. Every active statue in the gallery requires careful manual placement of a 3D model at the exact position of the statue by the curator of the game or a person that sets up the system. This needs to be done only once at the time of setting up the system. To position the models, the user taps on the surface where the model should be placed. Using depth cloud from depth camera the system identifies the dominant base plan and places the object at the place where this plan intersects with ray cast sent from the touch position. Using touch screen interface the user can fine tune the position, orientation and size of the model so that it best fits to the object.
Once the system is set up we can start using it in the game mode. Before the colored puzzle can be applied to the 3D object we need to digitise it and create a texture map from it. The coloured puzzle is scanned using a phone camera. The key challenge is to generate a texture map where the uncolored parts of the puzzle (white parts of the paper) become transparent in the texture map. We accomplish this by replacing all grayscale pixels in RGB mode with transparent pixels with static threshold.
Currently, we are investigating methods to improve the accuracy of how well the 3D content would fits the real object, by using information from depth camera. Based on the camera pose information we could predict the point cloud our depth camera should see. Then we could make small adjustments to the position of the object in order to minimise the error between the predicted point cloud and actual point cloud data.
Our idea and prototype have been built on the concept of personalised playful learning engaging users in active participation rather than passive consumption [10]. The aim of our prototype is to give users the opportunity to actively explore and design expanded art forms rather than being passive observers. There are several ways in which the prototype can achieve this. With colouring contours users (i) explore their creativity, (ii) try to figure out the appropriating object that would fit their creation (as has been observed in [ 9]), and (iii) think about underlying geometry, shapes, and space that form statues through their own exploration and understand how these concepts build up the artwork. Additionally, the game could be coupled with audio/video feedback for scanned objects and short context aware quizzes in order to allow users progress in the game.
In the future, we aim to explore tangible (e.g., paper, chalk on sidewalk) and digital colouring (e.g., mobile devices), as well as individual (e.g., paper) and collaborative treasure hunting game play. Overall, we believe this research is a positive first step in developing more engaging experience of gallery and museums content for a broader audience including young population.
The authors acknowledge the European Commission for funding the InnoRenew CoE project (Grant Agreement 739574) under the Horizon2020 Widespread-Teaming program and the Republic of Slovenia (Investment funding of the Republic of Slovenia and the European Union of the European regional Development Fund). This research was also supported by Slovenian research agency ARRS (program no. P1-0383 and J1-9186). [5] M. Weerasinghe, A. Nisansala, D. Sandaruwan, N. Kodikara, C. Keppitiyagama, K. Dias, N. Dammika, I. Buddhika, C. Perera, Poster: Virtual eye-a smarter mode of interaction to virtual tours, in: Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services Companion, 2016, pp. 86–86. [6] P. Coulton, R. Smith, E. Murphy, K. Č. Pucihar, M. Lochrie, Designing Mobile Augmented Reality Art Applications: Addressing the Views of the Galleries and the Artists, Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services (2014) 177–182. URL: http://doi.acm.org/10.1145/2676467.2676490. doi:10.1145/2676467.2676490. [7] A. Angelopoulou, D. Economou, V. Bouki, A. Psarrou, L. Jin, C. Pritchard, F. Kolyda, Mobile Augmented Reality for Cultural Heritage, volume 65, 2013. URL: http://link.springer.com/ 10.1007/978-3-642-36660-4. doi:10.1007/978-3-642-36660-4. [8] N. Moorhouse, M. C. tom Dieck, T. Jung, An experiential view to children learning in museums with augmented reality, Museum Management and Curatorship 34 (2019) 402–418. [9] K. Čopič Pucihar, M. Kljun, P. Coulton, Playing with the Artworks, in: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems CHI EA ’16, ACM Press, New York, New York, USA, 2016, pp. 1842–1848. doi:10.1145/ 2851581.2892322. [10] J. S. Bruner, The process of education, Harvard University Press, 2009.