In recent years, there is a constant tendency in integrating modern technologies into mobile guides and applications in Cultural Heritage (CH) domain, aiming in enriching cultural user experience. Amongst them, Virtual Reality (VR) has widely been used in digital reconstruction or restoration of damaged cultural artifacts and monuments, allowing a deeper perception in their characteristics and unique history. This work presents a VR environment that takes into account the diverse needs and characteristics of visitors and digitally immerses them into paintings, giving them the ability to directly interact with their characteristics with the Leap Motion controller. To test our proposed system, a mobile prototype application has been designed, focused on the famous painting “Children Concert” created by Georgios Iakovidis, which also integrates the User Personas and the different scenarios depending on users' profile.
(audio guides) or walked freely without any kind of guidance. A VR guide can boost mental and visual focus on exhibits, achieving a level of flow (Mihaly Csikszentmihalyi, 1975), which motivates user to seek more knowledge and extend his visit.
Meanwhile, personalization methods in User Experience (UX) and Cultural User
Experience (CUX) appear to give a new perspective to mobile guides and applications
in Cultural Heritage (CH). Personalization
Current work presents a Virtual Reality interface that represents digitally the world
of paintings, allowing users to interact with the aspects of the painting in a 3D
environment. The presented framework also integrates personalization, user personas
ARTISTS is a mobile application that brings to life famous paintings, by digitally construct its aspects in a Virtual Reality environment, where users can interact with its 3D models. Users immerse into the VR world by using their own devices mounted on a VR headset (Google Cardboard), and then interact with the 3D environment using gestures that are captured by the Leap Motion controller, that’s attached on the headset. The proposed interface not only puts user inside a painting, allowing them to observe and interact with the 3D models in many angles, but also uses various methodologies (context-awareness, personalization, and gesture-recognition) in order to enhance user’s cultural experience.
ARTISTS prototype has been designed based on the famous painting “Children’s Concert” by Greek painter George Iakovidis, which can be found in Athens National Gallery – Greece, and in a digital format in “George Iakovidis” digital gallery, in Hidira village – Lesvos. For this painting, seven 3D human models were created, along with their animations and sounds, in accordance with the 7 characters found in the original painting. Painting’s surrounding space (a bright room having some furniture) has been digitally reconstructed in a VR environment, taking into consideration the limited resources of mobile devices.
ARTISTS prior version was a mobile application in which users were also able to interact with the 3D version of a painting by just tapping on mobile device’s screen, thus without totally immersion to the VR environment. Application settings like sound, running scenarios, animations etc were depending on user’s profile and interests, a functionality that still stands in ARTISTS, but with the use of more accurate methodologies.
In ARTISTS design, we take into consideration parts of the context like the ambient
noise level, processing power of the mobile device and screen resolution, trying to
improve users’ experience regardless of environmental conditions. In particular, in a
quite noisy environment (to the noise level of 50dB), sound volume can be increased up
to 50%, whilst in extremely noisy conditions (noise level more than 70dB), application
audio volume mutes to avoid Lombard effect
In our case, we use the User Personas method, which categorize users based on their
profile during a museum visit. User Personas
Gesture recognition refers to computers’ ability to understand gestures involving
physical movements of multiple body parts (fingers, arms, hands, head, feet, etc) and
execute commands based on the corresponding gesture, thus allowing interaction with
the computer environment. Many gesture recognition approaches suggest that gestures
used as interaction methods between humans, can also been successfully applied as a
natural and intuitive way to interact with machines [
The design of personas as ‘fictional’ characters is considered as a very consistent and representative way to define actual users and their goals. However, it is important to clarify the exact number of personas in each occasion in order to focus on the visitor profiles to be examined. On ARTISTS, we take into consideration these UPs and their characteristics and we create more Personas by splitting Followers and Browsers into 3 Levels. Searchers and Researchers are combined and split into 2 Levels. These Levels have a quantitative meaning. For example, Level 2 Researcher has done more research and shows more of the initial Researcher characteristics than Level 1 Researcher.
In order to match each museum (or any other cultural site) visitor to an ARTIST persona, the system collects and process various data about visitors. Data mining is ARTISTS involves no user interference or preparation and it’s a 3-stages process: 1. Face recognition: Using Microsoft Cognitive Services, user age and emotions are calculated by their face picture taken from the device’s front camera that is sent over network. In addition, a database of visitors is created, turning every possible upcoming visit into a more successfully personalized experience. 2. Social networks data mining: Using data mining algorithms, visitor’s data (profile and prior experience) are extracted from user social profiles (Facebook, Twitter or Instagram). Fully compatible with GDPR rules, algorithms can only use data that users expose as public. 3. Behavior study: Sensors embedded into the visiting area monitor visitors’ path and behavior into space, providing ARTISTS more personalization data.
ARTISTS is a Client – Server system, as shown in Image 1. Core of the system is a server, located either in a museum (or any cultural site) or in a remote position. Server supports communication between database, application and sensors network (installed in museum). Furthermore, more server tasks are responsible for matching visitors to predefined personas, or displaying multimedia for the VR environment.
The mobile application creates the appropriate interface between user and ARTISTS system. Depending on visitors’ profile, the system shows a different scenario and service. Server also is responsible for handling sensors’ and Smart Objects (SO) input that can alter applications’ content.
Image 1: System architecture in ARTISTS
After getting necessary visitor data and assigning one persona from Table 1, one of the 19 usage scenarios may initiate. These scenarios are 19 in total and matching a visitor to a scenario is a dynamic process. For example, user can start visiting a museum as a Level 3 Follower, but after a while, his behavior can turn him into Level 1 Browser and then Level 2 Browser. This happens because behavior monitoring is an ongoing process that gives feedback data which can eventually change the flow of user experience. Each one of the scenarios in Table 2 is different in functionality, interactivity, display quality and load, audio (Table 2).
Image 2: The VR representation famous painting “Children Concert” created by Georgios Iakovidis
In this work, we describe the ARTISTS framework, a mobile application that displays a VR reconstructed environment of a painting, and immerses users allowing them to interact with its 3D aspects. We used the Leap Motion controller as a sensor for detecting gestures, alongside with Unity, Microsoft’s Azure Cognitive Services and Android Studio for the implementation of the application and the MySQL database that stores the 3D environment and painting’s data. Our next step includes the ARTISTS evaluation stage, in which we will test our framework to evaluate user’s experience and the efficiency of our integrated technologies.
The research of this paper was financially supported by the General Secretariat for Research and Technology (GSRT) and the Hellenic Foundation for Research and Innovation (HFRI). John Aliprantis has been awarded with a scholarship for his PhD research from the “1st Call for PhD Scholarships by HFRI” – “Grant Code 234”.