The last two years have been dificult for museums and cultural sites since the Covid-19 pandemic changed and limited people's cultural experiences. This paper aims at exploring the impact of the virtual assistant metaphor to guide people vising a virtual museum and improve the accessibility to information about artworks in the National Archeological Museum of Taranto. We present the cultural context, the application and the visitors' feedback. To this aim, we created two virtual agents, one for welcoming people and explaining the visiting experience, staying at the reception desk of the museum, and another acting as a guide in a room with artworks that interacts with users in natural language. A preliminary evaluation regarding the visitor acceptance of the system was conducted and its results show, in general, a good acceptance especially as far as the user experience is concerned, while some problems emerged from the usability point of view, control in particular. Then, we evaluated the interaction with the two virtual assistants. Results show that all the users were satisfied with the interaction with the two assistants even if some errors in the speech understanding occurred. Participants appreciated, in particular, the possibility to make questions about the artworks to the virtual guide. These results encourage us to go on the idea of integrating VR environments with AI-based virtual assistants.
The last two years, due to the Covid-19 pandemic, have been dificult for museum and cultural
sites. Many museums, to favor remote visiting and cultural experiences, used digital
technologies, such as virtual reality, to let people visiting places remotely and immersively. Recently,
the concept of metaverse [
This research aims at exploring the impact of the virtual assistant metaphor to guide people
visiting the virtual representation of a museum and improving the accessibility to information
about art- works in the National Archeological Museum of Taranto (MArTA) [
The application of VR to the museum provides a new dimension to the visitors’ experience
and many museums ofer VR guided tours. During the lockdown due to Covid-19 pandemic,
the use of technologies enabling remote visits has become a requirement to allow visitors
continuing enjoying the art [
There are several examples of museums that enriched their exhibitions by providing to visitors VR experiences. For instance, in 2019, Paris’ Louvre launched ‘Mona Lisa: Beyond the Glass’, a VR experience that through interaction allows users to discover details about the painting. The experience can be enjoyed not only directly at the Louvre but also on VR app store, iOS, and Android. Also the Smithsonian National Museum of Natural History ofers several virtual museum tours on the web. It is possible to browse their permanent, current or past exhibitions as virtual journeys through time and history. The New York’s Metropolitan Museum of Art, the Florence’s Ufizi Gallery and the Van Gogh Museum, thanks to a successful Google Arts Culture partnership ofers diferent 3D virtual tours of their collection.
Recently, with the introduction of the concept of metaverse, VR appears to be intended for use
more widely by many people. Then, extending the museum experience in the metaverse seems
appropriate. In this context, to provide a likelihood experience with a real museum visit, besides
the VR representation of the museum, it seems successful to integrate it with AI technologies.
Avatar of people visiting the museum and virtual assistant, that are 3D versions of chatbots
[
The idea of implementing such a system is not new. Several studies were already discussing
the challenges posed by the development of an intelligent museum guiding system [
Many chatbots have been developed to guide people in a museum and receive information
about artworks [
These works and examples indicate that integrating VR visits with AI-based interaction
can be a good solution to enhance the visitors’ experience. However, to be successful, when
designing these types of applications, we should take into account the visitor’s feedback and
their acceptance of the technology using a human-centered perspective [
To test our approach, we re-created one of the rooms of the MArTA museum of Taranto: the Ricciardi Room, that takes the name from Mons. Giuseppe Ricciardi, who donated his precious collection of paintings to the museum.
The 3D virtual environment consisted of a reception room, aiming at welcoming and helping
visitors with useful information about the experience, and the Ricciardi room. For the 3D
reconstruction of the rooms, the Amazon Sumerian platform has been used [
Amazon Sumerian allows creating the 3D scenes that are made up of components and entities, organized into projects. A scene is a 3D space that contains objects (i.e. paintings, lights, etc.) and behaviors (animations, timelines and scripts) that together define the VR environment. The scene, once ready, can be exported as a static website that can be opened in a browser. Two virtual assistants were developed as Sumerian hosts. An host is an asset provided by Sumerian that has built in animation, speech, and behaviors for interacting with users. Hosts use Amazon Polly as a text-to-speech. The virtual assistant in the VR museum were created with two diferent aims: • Sam, that welcomes people at the reception and provides information about the visit. It explains also that is possible to select a language diferent from Italian by clicking on the flag on the reception desk. In addition, the user may also see the content of a short tutorial video on the back of Sam (Figure 1a). • Marta, that guides the visitor through the paintings presenting and describing each painting in the room; on request Marta can answer to visitor’s questions about a painting of interest (i.e. ”Who is the author of this painting?” or ”When it was painted?”) (Figure 1b).
The evolution of the VR environment and virtual assistant’s behavior as a consequence of an event is managed through a finite state machine. The information about the paintings are structured and stored in the AWS DynamoDB so as to let Marta describing the artworks and answer to questions. The experience has been designed for an immersive VR visit, however it is possible to visit the Ricciardi room also on a PC without VR headset. In the immersive VR setting, the interaction through the headset allows looking around, moving the head to see what is in the room. To move around it is suficient to point the motion controller toward the destination point on the floor. Then, to speak with the virtual assistants it is necessary to point and click with the motion controller on its body to open its listening channel and then, using the microphone of the VR headset, it is possible to talk with the assistant. The assistant, using the Amazon Lex NLU service, will process the input and provide the answer.
As mentioned previously, we evaluated the usability and the user experience (UX) of the VR application and, according to the human-centered approach, we performed several formative evaluation phases, using the thinking aloud approach, until we reached a version of the system that seemed acceptable for the testers. Then, we performed a user test on the resulting application and collected both subjective and objective measures regarding usability and UX.
Fifteen users evaluated the system (7 males and 8 women from 23 to 40 years old). They asked to participate voluntarily and none of them previously had an immersive VR experience.
We selected a set of tasks to be accomplished by the participants. Some of these tasks concerned the interaction with Sam the receptionist and some others with Marta the virtual guide. The set of tasks are listed in Table 1.
To get subjective measures of the general perceived usability and UX the UEQ (User Experience
Questionnaire) questionnaire has been employed. The UEQ allows a quick assessment of the
user experience in a very simple and immediate way [
The questionnaires were made available through the Google Forms platform. As quantitative measures, we collected the task success rate. We defined as a success a task executed correctly without the help of the facilitator, as a partial success the case in which the task was executed after the help of the facilitator, as a failure when, even with the help of the facilitator, the participant did not complete the execution of the task. The devices supported by Amazon Sumerian are many, for the evaluation we used the Oculus Quest 2 device.
Before starting the experiment, participants were given an overview regarding the VR technology, the experiment, and the type of data collected. After signing an informed consent, participants were trained on how to wear and use the headset and the application by the test facilitator. The headset was sanitized according to the Covid-19 protocol. The experiment was conducted in a room of a research lab in our Department. Each user has been invited to the lab and, after wearing the VR headset, started the experiment by executing the tasks indicated by the facilitator (Table 1). During the test the observer took note of every problem occurring during the interaction. After completing the test, each participant was invited to fill-out an on-line form containing general demographic questions, the UEQ and CUQ questionnaires.
From the analysis of the questionnaires results, in general, participants did not have a high dificulty in interacting with the VR environment and with the virtual assistants. The graphic in Figure 2 shows that the scores related to the UX (attractiveness, stimulation and novelty) are very high and denotes that participants had a pleasant experience with the system. Surprisely, the perspicuity score is high, meaning that it was easy for participants to understand how to interact with the application and executed the assigned tasks. Even if above average, the eficiency and dependability are lower than the other scores, showing that there were some problems related to usability and control of the interaction. Globally these results are quite encouraging considering that none of the participants had a previous experience with VR technology.
As far as quantitative measures are concerned, all the users completed the tasks with a success rate of 90,06%. Only three users, the older ones, asked for the help of the facilitator 10 times. All the times the task was related to speech input toward the virtual assistants. As far as the CUQ is concerned we asked to evaluate the interaction with Sam and Marta. Sam received an evaluation on average of 77/100 and Marta a slightly higher one (79/100). These scores can be considered a good result. Looking at the notes of the observer, the main problems concerned error handling, since three times the speech input was not recognized correctly by the assistants and for two times, even if the speech was recognized correctly Marta did not answer. However, from the ifnal interview all the user stated to be satisfied by the user experience and appreciated the possibility to interact vocally with the assistants. In particular, the majority of the participants appreciated the possibility of asking information about a painting directly to Marta. They suggested to change the answers of the virtual assistant according to the available time of the user by providing short or long descriptions.
This study is valuable from the applied research point of view, as we present the preliminary evaluation of a VR museum experience integrated with AI-based virtual assistants. The proposed system uses Amazon Sumerian and it is flexible enough to be used in multiple scenarios. Even if the study has been performed only with 15 users, its results indicate that the proposed system provides a good interactive experience. However, some problems, related to control and speech based input, were detected indicating that a better errors handling strategy needs to be implemented.
In the near future, we aim to conduct a new user study on several categories of visitors based on a quantitative and qualitative methodology that takes into account also some aspects related to the engagement and emotional experience. A more believable behavior of the two virtual assistants in terms of eye gaze and turn taking could be set to track the person who is actively involved in the conversation. As for the “intelligence” of the virtual assistant is concerned, the museum’s staf could expand the database with thousands of questions. One particularly useful comment received as a feedback from visitors is to allow them to choose among two diferent forms of answers: small (brief resumes) and large (more descriptive) or to adapt the answer to the age of the user. All these developments can be quickly addressed and implemented, as proof that technologies used are mature enough to support the museum experience in the metaverse.
We thanks the staf of the MArTa museum for the permission of reproducing the paintings and also for their descriptions.