Adaptation and content personalization in the context of multi user museum exhibits

Abstract

Two dimensional paintings are exhibited in museums and art galleries in the same manner since at least three centuries. However, the emergence of novel interaction techniques and metaphors provides the opportunity to change this status quo, by supporting mixing physical and digital Cultural Heritage experiences. This paper presents the design and implementation of a technological framework based on Ambient Intelligence to enhance visitor experiences within Cultural Heritage Institutions (CHIs) by augmenting two dimensional paintings. Among the major contributions of this research work is the support of personalized multi user access to exhibits, facilitating also adaptation mechanisms for altering the interaction style and content to the requirements of each CHI visitor. A standards compliant knowledge representation and the appropriate authoring tools guarantee the effective integration of this approach in the CHI context.

Keywords

• J
• 5 (Fine arts)
• H
• 5
• 1 (Artificial
• augmented
• and virtual realities)

1. Introduction

Ambient Intelligence (AmI) presents a vision of a technological environment capable of reacting in an attentive, adaptive and active (sometimes proactive) way to the presence and activities of humans and objects in order to provide appropriate services to its inhabitants [ 3 ]. In the context of AmI, the need to adapt a Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.

2. Background

Nowadays CHIs strive to design and implement interactive exhibitions that offer enjoyable and educational experiences. However, designing such an exhibition is not an easy task, because most visitors might visit only once, and a typical visit only lasts for a very short time [ 4 ], [ 5 ]. To address such issues interactive exhibits are often employed as a means of providing alternative experiences. Such exhibits can be broadly classified in four categories: (a) hybrid exhibits which aim at augmenting an artifact with graphics [ 6 ] or audio commentaries [ 7 ]; (b) side exhibits which are placed adjacent to a real exhibit, providing indirect exploration of, and interaction with it [ 8 ]; (c) isolated, but linked, exhibits having “a conceptual affinity with the original artwork”; they are related to a real exhibit but installed in separate, dedicated, locations [ 7 ], [ 9 ]; and (d) stand-alone exhibits containing content related to an exhibition, but not directly linked to an artifact [ 10 ].

One of the main challenges of interactive exhibits is the need to cope with the requirements of diverse users. These requirements may affect both desired interaction and content. A possible solution to address these requirements could be the integration of some form of intelligence in the way that UIs are built and information is presented. Intelligent user interfaces are characterized by their capability to adapt at run-time and make several communication decisions concerning ‘what’, ‘when’, ‘why’ and ‘how’ to communicate, through a certain adaptation strategy [ 16 ]. The provision of these qualities within CHIs entails the need to address design issues far more complex than those faced by traditional HCI. To address similar needs, a user interface adaptation methodology has been proposed as a complete technological solution for supporting universal access of interactive applications and services [ 17 ]. This methodology conveyed a new perspective into the development of user interfaces, providing a principled and systematic approach towards coping with diversity in the target user requirements, tasks and environments of use [ 18 ]. Several UI adaptation frameworks have been proposed implementing the aforementioned development methodology, such as for example the EAGER framework [ 19 ] that allows Web developers to build adaptive applications. In these prior approaches knowledge about users was either statically represented or acquired through formal specifications using special purpose programming languages [ 20 ]. These ad-hoc approaches are currently replaced through the usage of knowledge modelled with the help of a web ontology language such as OWL [ 21 ]. Such models store the appropriate information in the form of semantic web rules and OWL-DL [ 15 ] ontologies. At the same time, rule engines are employed to facilitate adaptation logic and decision making while mature UI frameworks are employed to ensure a smooth user experience [ 14 ].

In terms of technology, mobile devices have currently achieved the greatest amount of penetration within CHIs. Existing mobile applications for CHIs fall into the following categories [ 11 ]: (a) 45% provide guided tours of the CHIs in general; (b) 31% provide guided tours of temporary exhibitions; (c) 8% provide combinations of the first two; (d) 8% are applications devoted to a single object; (e) 4% offer content creation or manipulation; and (f) 3% are games.

Although much work has been done to date, there are several limitation to the approaches currently followed for facilitating CH within CHIs. Major improvements are considered: (a) the support of multi user interaction, (b) content personalization, (c) facilitation of structured knowledge (based on existing domain standards) and (d) scalability and extensibility. To provide the above, an augmented digital exhibit should be designed and implemented to be: (a) generic, built on top of an ontology metamodel (extending CIDOC-CRM) to present two dimensional paintings including the appropriate tools to support the integration, annotation, and preparation of knowledge, (b) available to a large number of visitors concurrently (using smart phones, digital projections, interactive captions and hand held tablet devices), (c) personalizable using mobile devices for information displays through a user profile so as to adapt content and presentation and (d) adaptable facilitating a rule engine to execute UI adaptation rules resulting to the optimum UI variation for each user.

3. Scenario of use

One of the personas [ 22 ] used during the conceptual design of the exhibit was Anna, who has a non-professional interest in art, but is an art lover enjoying visiting museum, galleries, etc. Anna decides to take a visit to the local Museum of Art. While entering the museum towards the exhibition, a notification appears on her mobile device prompting her to download the mobile client. She also takes a minutes to fill in an anonymous profile (see figure 2). Within the museum her mobile device is used as a navigator allowing her to access information by scanning QR codes (see figure 4-3). When Anna approaches an exhibit, she notices that information is projected on the periphery of the painting, while a tablet is unobtrusively located in front as an interactive caption (see figure 4-2). Anna can use touch for navigating and browsing the vast collection of information available for the specific exhibit using the tablet. She also shows the QR code representation of her profile to the caption (or any other component of the exhibit) so as to access personalised information (Anna has painting as a hobby and loves learning about materials and techniques used by the old masters). She also notices that the UI of the caption is altered allowing her to slide through representations (as an expert user of mobile devices see figure 4-6).

When she stands in front of the digital painting, an interactive menu appears allowing her to start interacting with the specific exhibit. She can use her hands to indicate points of interest within the painting to get additional information (see figure 4-5). She can also use gestures for zooming in and out specific regions of the painting and therefore accessing details that are typically lost when digitized artefacts are presented in their entirety at low resolution. Anna also wonders what happens when more than one person is accessing the same exhibit. In the room she sees several people standing in front of a large painting and all seem to be actively engaged while also noticing that an elderly user is required only to locate himself in front of a painting so as to get information. Alternatively, when approaching a physical exhibit, she gets informed that she can use one of the tablets located on a stand on each side of the exhibit to access personalised information based on her location in front of the painting.

4. A Distributed Architecture to support content and UI adaptation in CHIs

Four main goals are addressed in the proposed architecture (see figure 1): (a) model the knowledge facilitated by the system (artefacts, users and context), (b) provide facilities within a distributed environment (consisting of applications, devices and sensors), (c) provide personalised information to users based on their preferences and (d) perform task and UI adaptation. The Content Personalisation Engine (figure 1-A) employs the Art meta-model, which is an extension of the CIDOC CRM [ 13 ], to represent two dimensional paintings. The model is populated with the help of a purposefully developed authoring tool and currently contains 300 paintings by 30 world known artists. Additionally, the User Profile model of the engine contains attributes used to personalise information to visitors. These models are exported to the higher levels of the architecture through a set of programming language classes (c#, java protégé data export facilities) and two sparql query (c# using SemWeb.Net and java using Jena and Pellet). A number of alternative implementations were created to support multiple development platforms and thus ensure the reusability of the Content Personalisation Engine. Finally, the multi-scale image repository stores and serves through an IIS web server images in extremely large resolutions and their representation in xml to be used for deep zooming into digital artefacts.

The Computer Vision Infrastructure (figure 1-B) is built on top of the Microsoft Kinect SDK to support a number of alternative interaction styles (hand - skeleton tracking, gestures and postures recognition). At the same level lies the zxing library for generating and scanning of QR codes.

The service oriented communication protocol (figure 1-C) built on top of the FORTH’s Famine middleware [ 12 ] (a distributed service oriented middleware that supports all popular programming languages exposing a common event model and service discovery and invocation mechanism), provides a common dialect for applications to coexist and communicate in the context of the developed application scenarios while using sensing for decision making. The existence of a common communication protocol was essential in order to allow a number of standalone and heterogeneous applications running on alternative devices (desktop pc, Windows phone device, Windows tablet) to communicate (exchange messages and events) at runtime using a commonly understood dialect.

The UI Adaptation engine (figure 1-D) has the responsibility of producing adaptation decisions using the Windows Workflow Foundation Rules. WWF rules engine was selected both for simplicity of implementation and because it is light weight in conjunction to other rules engines. Furthermore it allows the separation of the adaptation logic with the UI functionality that implements adaptations in each UI instance. For each application a set of rules has been defined. These rules are modeled separately from the interface itself and the adaptation engine carries out the task of chaining an interactive application with its rules and user profile to perform adaptation.

Finally, the Applications (figure 1-E), which extract functionality from services, are targeted to different devices and application frameworks and are interconnected at runtime to form personalized application scenarios.

5. The Augmented Personalised Exhibit

The Augmented Personalized Exhibit provides interaction where no interaction exists (making physical artefacts interactive) and provides interactive digital artefacts where no artefacts exist (importing both an artefact and the means to interact with it within the CHI experience). The exhibit comprises a number of devices for content provision as well as a number of modalities for interaction. As shown in Figure 4, the main section of the exhibition wall is occupied by a digital representation of an exhibit in two variations. The first variation is a fully digital exhibit where the exhibit itself is projected through the usage of a short throw projector, while the second one is an actual physical painting. In both cases skeletal tracking technology is installed on the exhibit for tracking the location and distance of visitors. The installed tracking technology supports the presentation of information about points of interest using body tracking (two visitors supported on the body tracking mode while three are supported for the hand tracking). On the rear sides of the exhibit two tablets are mounted on the wall or on two portable stands to act as the captions of the painting. The captions based on the visitor profiles present a multitude of information such as description, videos, points of interests, deep zoom representation of the painting, full artefact info and information from external sources. These tablets are also equipped with embedded web cameras for QR code recognition. Visitors’ mobile phones are used for accessing information about the exhibit by scanning the QR codes (from the captions). Portable tablets, rented or carried by visitors, can be also be employed as information displays. Currently each installation supports a single digital or physical exhibit and a variation of devices (project, mobile phones, tablets etc.). Each visitor can select the device to be used for interaction but there is no control over the artefact to interact with.

5.1 Content Personalization

The content personalisation workflow is initialised by the installation of the mobile client to a visitor’s cell phone. When the application launches, the user is prompted to fill-in an anonymous user profile (see figure 2). User selections are stored in the smart phone’s local storage to ensure that no confidential information is transmitted over the web (although the profile is anonymous malicious software may be possible to relate other services running on the mobile phone e.g. GPS and social media with the transmitted profile data and thus infer the identity of the user). This profile is used for presenting personalised information from the smart phone. All queries formed by the mobile application to the ontology model carry with them the required profile attributes and the QR code of the exhibit scanned by the user. Users can use the mobile client to generate a QR code representation of the profile that is in turn scanned by other interactive applications so as to identify user preferences. For example, the user can shows the QR code generated from his mobile phone to the mounted caption or the exhibit itself, and the exhibit personalises the information to the profile selections of the user. The overall workflow is presented in figure 3.

5.2 UI adaptation

Each interactive application comes to its initialisation state by retrieving and executing default application specific rules from the rules store. A QR recognition service is initiated and runs on the background. Each of the users can in turn use their Smartphone to generate the QR code representation of their profile, and point this representation to the application so as to transfer their preferences to the application. The transmitted preferences are used to alter several application properties. This results in the re-evaluation of the rules by the rule engine and the generation of adaptation decisions that are directly transferred from the Rules Engine to the application. The result is the generation of an adapted UI that matches the user preferences as recorded to the profile.

The tablet browser used to access information based on current location of the user in front of the physical exhibit One of the alternative views of the caption showing a relevant video

The digital exhibit shows information about POIs based on the location of the visitor’s hand The mobile client used to access information using QR code scanning

The mobile client used to personalize the caption of a painting (Top: filling the profile; bottom: QR code representation of the profile)

Figure 4. The interactive digital exhibit An example of this process is shown in figure 5. On the top left of the picture is the screen from the mobile emulator where the user is entering his profile. On the top right of the picture is the QR code generated based on the user’s profile and on the bottom left side is the QR code profile scanning mechanism that is running on the artefact caption. The resulted adapted caption is shown on the bottom right side of the same picture. Another example is shown on figure 4-1 where the user is not experienced with technology so skeletal tracking is employed to automatically identify his/her position and present information inline. On the contrary in figure 4-5 the user is expert so hand tracking is employed to allow him to fully explore the exhibit.

In the case of multiple users a mixed adaptation process is followed. The profiles of all users are merged and the most appropriate representation of the exhibit is presented to cover possibly all users. Further research is required so as to mark with computer vision algorithms each user and thus allow the per user adaption of the interactive exhibits.

6. Evaluation

The evaluation exhibit has been tested with usability experts and subsequently with visitors. The expert based evaluation was conducted by three usability experts. A scoring scale from 0 (not a usability problem) to 4 (usability catastrophe) was used [ 14 ]. Thirty issues were identified in total, and twelve of them were considered major usability problems. The user-based evaluation session was performed with the participation of ten users. Concerning the participants’ gender, it came out that more male users participated in the evaluation, in percentage 60%. Regarding the age of the participants the majority (7 users) was between 20 and 29 years old, also having 2 users within the age group of 3039 and another one on the 40-49 age group. Five users where experts regarding the usage of ICT (both desktop and mobile devices) while the remaining where moderately experienced and one user had limited experience. Users were requested to fill in a pre-test questionnaire containing demographic information and questions to collect data regarding the usage of ICT technology within CHIs. Upon completion of this process, users were requested to carry out a number of interaction scenarios and then fill in a post-test questionnaire. The user based evaluation was conducted within a room in the AmI facility of ICS-FORTH that was appropriately set up to host the implemented interactive digital exhibit.. User interaction was recorded for offline processing.

The results gathered through the post-test questionnaire were used to calculate four factors, namely the overall user satisfaction, the satisfaction of users when using the system, the quality of the provided information and the satisfaction regarding the interface provided by the system. Regarding overall user satisfaction, ~87% of the users are within the range 5 to 7, while 30.56% of the users provided a grade of 7 to all questions. However, ~5% of the users stated that they were not satisfied. Regarding user satisfaction when using the system, ~85% of the users are within the range 5 to 7, while ~37% of the users provided a grade of 7 to all questions. However, ~14% of the users stated that they were little to medium satisfied. Regarding information quality, ~88% of the users are within the range 5 to 7, while ~25% of the users provided a grade of 7 to all questions. However, ~43% of the users scored 6, which implies that there is a substantial amount of users who faced some form of difficulty understanding the presented information. Finally, the user interface of the system, ~83% of the users are within the range 5 to 7, while ~35% of the users provided a grade of 7 to all questions. However, ~25% of the users scored 5 and ~24% scored 6, which implies the existence of some form of usability barriers. The results of the aforementioned quality factors provided some initial indications about potential areas of improvement. To identify those areas more clearly further post processing was conducted. The questions where grouped into four categories, analysed both individually and by category: • General User Satisfaction: Analyzing the comments provided by users in the questions used to calculate general user satisfaction several new research directions became prominent. In some cases users may require specialized curation for some digital assets, especially in the case where the digital asset is linked to a myth or a historic event. In such cases, the system should support the curators into the process of revealing the myth out of the artefact, providing extra historic information or even building a story to be told. These new directions highlight the need for concrete strategies towards curating digital assets. • Interaction techniques: The hand tracking interaction technique scored lower grades in relation to body tracking and touch (~55% of the users scored 5 regarding hand-mirrored hand synchronizations and ~44% scored 5 for hand-based content navigation). On the contrary, body tracking and touch have better results. • Information representation & extraction: Users were in general very satisfied (~85% scored from 5 to 7 in all questions of this group). Nevertheless there, is a percentage of ~55% who are not fully satisfied regarding the way that information is browsed in general. In this sense, 33% scored 5 • the way that information is presented using body tracking, ~44% scored 6 for the mobile client, while ~55% scored 6 in the caption.

UI Adaptation: Regarding the ways that the UI of the system are adapted, users were in general satisfied (~70% scored from 5 to 7 in all questions), but there was a substantial number of users that were not fully satisfied with the way that the system was adapted to map their selected profile. In their comments, some of the users documented that for example they preferred to slide the different screens of the digital caption but based on their profile next and previous buttons appeared. Such cases are typical examples when performing profile based adaptation and are typically restored by integrating an additional personalisation layer to the system. In this layer the user overrides the default decisions made by the system to fine tune the interface to best suit his/her personal preferences. Especially in the case of Heritage Institutions where visitors have limited time to configure a provided interface integrating such a layer does not seams a good idea. The usability experts proposed a more intelligent way of solving such issues by introducing the possibility of runtime adaptation based on user input. For example in the case of navigation buttons a message could appear to the user: “Switch to slide by just sliding your finger over the screen”. In such a case the user can perform the personalisation part while browsing information.

7. Discussion and future work

This work expands the current state of the art in the context of augmented exhibits within CHIs in a number of directions. The proposed digital exhibit integrates a number of alternative devices and interaction metaphors to facilitate simultaneous multi user access to paintings. Moreover, focus is put back to art itself rather than providing just another exhibit in the CHI. In the same context visitor’s interaction capabilities, technology expertise and art knowledge are used for applying content personalisation and UI adaptation coping with the diversity of the target user population within CHIs. User acceptance and satisfaction factors were measured by conducting a user based evaluation within an in-vitro installation of the proposed approach. Practical exploitation of the concept within CHIs is currently being considered.

Regarding future research directions the user based evaluation of the produced significant input regarding how this research work can be improved and what are the aspects that should be improved. A possible direction further to the ones identified during the evaluation is the introduction of social features to the interactive digital exhibit thus being able to capture user feedback. Such feedback could be exploited through of line processing to enhance the provided information with user extracted info thus producing a more pluralistic view on art.

Acknowledgments

This work is supported by the FORTH-ICS internal RTD Programme 'Ambient Intelligence and Smart Environments'.

References

[1] Stephanidis , C. ( 2001 ). The concept of Unified User Interfaces . In C. Stephanidis (Ed.), User Interfaces for All - Concepts , Methods, and Tools. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 371 - 388 (ISBN 0-8058-2967-9).

[2] Savidis , A. , & Stephanidis , C. ( 2004 ). Unified user interface development: the software engineering of universally accessible interactions . Universal Access in the Information Society , 3 ( 3-4 ), 165 - 193 .

[3] Stephanidis , C. ( 2012 ). Human Factors in Ambient Intelligence Environments . In G. Salvendy (Ed.), Handbook of Human Factors and Ergonomics (4th Edition) , Chapter 49 (pp. 1354 - 1373 ). USA: John Wiley and Sons.

[4] Falk , J. H. , Koran , J. J. , Dierking , L. D. , & Dreblow , L. ( 1985 ). Predicting visitor behavior . Curator: The Museum Journal , 28 ( 4 ), 249 - 258 .

[5] Serrell , B. ( 1998 ). Paying Attention: Visitors and Museum Exhibitions . American Association of Museums, Washington, DC.

[6] Bimber , O. , Coriand , F. , Kleppe , A. , Bruns , E. , Zollmann , S. , & Langlotz , T. ( 2006 , July) . Superimposing pictorial artwork with projected imagery . In ACM SIGGRAPH 2006 Courses (p. 10 ). ACM.

[7] Kortbek , K. J. , & Grønbaek , K. ( 2008 , October) . Interactive spatial multimedia for communication of art in the physical museum space . In Proceedings of the 16th ACM international conference on Multimedia (pp. 609 - 618 ). ACM.

[8] Hornecker , E. and Stifter , M. , 2006 . Learning from interactive museum installations about interaction design for public settings . In Proceedings of OZCHI '06 , 135 - 142

[9] Ferris , K. , Bannon , L. , Ciolfi , L. , Gallagher , P. , Hall , T. , Lennon , M. ( 2004 ). Shaping experiences in the hunt museum: a design case study . In: Proceedings of DIS 2004 , pp. 205 - 214 .

[10] Robertson , T. , Mansfield , T. , and Loke , L. , 2006 . Designing an immersive environment for public use . In Proceedings of PDC '06. ACM , New York, NY, 31 - 40 .

[11] Economou , M. , & Meintani , E. ( 2011 ). Promising beginnings? Evaluating museum mobile phone apps . In Rethinking Technology in Museums Conference Proceedings (pp. 26 - 27 ).

[12] Georgalis , Y. , Grammenos , D. , Stephanidis , C. : Middleware for Ambient Intelligence Environments: Reviewing Requirements and Communication Technologies . In: Stephanidis, C . (ed.) UAHCI 2009 , Part II . LNCS, vol. 5615 , pp. 168 - 177 . Springer, Heidelberg ( 2009 )

[13] Definition of the CIDOC Conceptual Reference Model : http://www.cidoc-crm.org/docs/cidoc_crm_ version_5 .1- draft-2013May.pdf

[14] Michou , M. , Bikakis , A. , Patkos , T. , Antoniou , G. , Plexousakis , D. ( 2009 ). A Seman-tics-Based User Model for the Support of Personalized, Context-Aware Navigational Ser-vices . In First International Workshop on Ontologies in Interactive Systems , 2008 . ONTORACT ' 08 . pp 41 - 50

[15] OWL Web Ontology Language Reference . W3C Recommendation , 10 February 2004 . http://www.w3.org/TR/owl-ref/ ( 2004 ).

[16] Stephanidis , C. , Karagiannidis , C. , & Koumpis , A. ( 1997 , January) . Decision making in in-telligent user interfaces . In Proceedings of the 2nd international conference on Intelligent user interfaces (pp. 195 - 202 ). ACM.

[17] Stephanidis , C. ( 2001 ) The concept of Unified User Interfaces . In C. Stephanidis (Ed.), Us-er Interfaces for All - Concepts, Methods, and Tools . Mahwah, NJ: Lawrence Erlbaum As-sociates, pp. 371 - 388 (ISBN 0-8058-2967-9).

[18] Stephanidis , C. ( 2001 ). New Perspectives into Human - Computer Interaction . In C. Stephanidis (Ed.), User Interfaces for All - Concepts , Methods, and Tools (pp. 3 - 20 ).

Mahwah , NJ: Lawrence Erlbaum Associates (ISBN 0-8058- 2967-9 , 760 pages).

[19] Doulgeraki , C. , Partarakis , N. , Mourouzis , A & Stephanidis , C. ( 2008 ). A Development Toolkit for Unified Web-Based User Interfaces . Computers Helping People with Special Needs , 2008 , pp 346 - 353

[20] Savidis , A. , Antona , M. , & Stephanidis , C. ( 2005 , in print). A Decision-Making Specifica-tion Language for Verifiable User-Interface Adaptation Logic . International Journal of Software Engineering and Knowledge Engineering.

[21] Partarakis , N. , Doulgeraki , C. , Leonidis , A. , Antona , M. , & Stephanidis , C. ( 2009 ). User interface adaptation of webbased services on the semantic web . In Universal Access in Human-Computer Interaction. Intelligent and Ubiquitous Interaction Environments (pp. 711 - 719 ). Springer Berlin Heidelberg.

[22] Zimmermann , G. , & Vanderheiden , G. ( 2008 ). Accessible design and testing in the application development process: considerations for an integrated approach . Universal Access in the Information Society , 7 ( 1-2 ), 117 - 128 .