=Paper=
{{Paper
|id=Vol-2299/paper9
|storemode=property
|title=Modeling User Interface Adaptation for Customer-Experience Optimization
|pdfUrl=https://ceur-ws.org/Vol-2299/paper9.pdf
|volume=Vol-2299
|authors=Christian Märtin,Christian Herdin,Bärbel Bissinger
|dblpUrl=https://dblp.org/rec/conf/fmt/MartinHB18
}}
==Modeling User Interface Adaptation for Customer-Experience Optimization==
https://ceur-ws.org/Vol-2299/paper9.pdf
Modeling User Interface Adaptation for Customer-
Experience Optimization
Christian Märtin, Christian Herdin, and Bärbel Bissinger
Augsburg University of Applied Sciences
Faculty of Computer Science
Augsburg, Germany
{Christian.Maertin, Christian.Herdin}@hs-augsburg.de; baerbelbissinger@web.de
Abstract—The customer journey in digital marketing defines With SitAdapt [14], [15] we have developed a software
several touch points, where interested users can directly interact architecture for situation analytics and for integrating adaptive
with an e-business platform. In order to convert a user into a behavior into web- or app-based interactive applications.
buyer, persona-based a priori adaptations of the user interface SitAdapt fulfills the requirements for automating essential parts
can be combined with dynamic adaptations at runtime with the of the customer experience optimization process as well as for
goal to optimize individual customer experience and guide task various other domains from medical monitoring to driver
accomplishment. This paper examines customer experience assistance systems. Possible adaptations are modeled within the
optimization for scenarios from a cosmetics industry e-business PaMGIS MBUID framework [5], [6]. They are triggered by
portal with the SitAdapt 2.0 system. Dynamic adaptations are
situation rules and generated by activating and exploiting
triggered by situation rules based on the continuous analysis of
the users’ varying cognitive and emotional situations during a
domain-dependent and independent HCI-patterns. In this paper
session. The model-based adaptation process exploits models and we present our preliminary lab-based results for using the
patterns for the rapid generation of user interface modifications. current implementation SitAdapt 2.0 with a new rule editor and
an advanced situation interpreter within the e-commerce
Keywords—customer journey; user experience; customer domain1.
experience; situation analytics; situation rules; emotion The paper includes the following main contributions:
recognition; eye-tracking; HCI-patterns
• Discussion of a new model-based approach [17] for
I. INTRODUCTION AND RELATED WORK automating customer experience optimization
Digitalization in marketing can be seen as a straightforward
approach to designing and implementing IT-based solutions for • Defining the potential for software adaptation [24],
[12], [19], [20] based on situation analytics [3], context-
the generic steps of the customer journey. A customer journey
is a customer’s interaction at several touch points with a awareness [22], and situation-awareness [7]
service or several services of one or more service providers in • Demonstrating the suitability of emotion recognition
order to achieve a specific goal [9]. More focused on and bio-signal tracking for triggering user interface
purchasing a product, the customer journey can be defined as modifications [8], [19], [21], [23].
an iterative process that includes touch point based interactions
with a provider or a business during a pre-purchase, a purchase, • Detailing the adaptation process and workflow for the
and a post-purchase phase [13]. The journey could include e-business domain
experiences from earlier purchases and affect future purchases. The remainder of the paper is structured as follows:
In this view no fixed a priori purchase goal is necessary, but the
service provider would try to arouse the interest of potential Chapter II introduces the SitAdapt 2.0 system with its new
customers in the pre-purchase phase. At all touch points rule editor. Chapter III first introduces possible adaptation
between the provider and the customer, one has to distinguish features and defines example scenarios for generic and
between the customer view and the provider view. It must be individual situations that are occurring in different phases of
the provider’s goal at every touch point, to create a situation the customer journey when visiting a cosmetics business portal.
that leads to optimum user experience (UX) for the potential Some of the possible SitAdapt 2.0 use-cases are demonstrated.
customer. After this, the chapter discusses the modeling and generation of
adaptations. Chapter IV concludes the paper.
UX during the customer journey is often described as
customer experience. As an extract and synthesis of earlier
research efforts customer experience can be seen as “a
multidimensional construct” that focuses on “a customer’s 1
Part of this work was carried out in cooperation with Dr.
cognitive, emotional, behavioral, sensorial, and social” Grandel GmbH, Augsburg, Germany. We greatly
reactions to the offerings of a provider or a business “during acknowledge the opportunity to run the SitAdapt 2.0 tools and
the customer’s entire purchase journey” [13]. user tests on their enterprise e-business platform.
68
Modeling User Interface Adaptation for Customer-Experience Optimization
II. SITADAPT 2.0 we used results from our cosmetics industry user study
The SitAdapt 2.0 runtime environment is integrated into the [1] for finding plausible situation rules. Fig. 2 shows the
PaMGIS (Pattern-based Modeling and Generation of creation of a simple situation rule with two conditions
Interactive Systems) development framework. The framework and one action. In this case only a dialog with the user
allows for modeling and generating responsive behavior in the is created. However, situation rules can also activate
user interface and has now been enhanced towards dynamic HCI-patterns in the PaMGIS pattern repository. These
adaptation by situation interpretation at runtime. patterns are exploited at runtime to generate user
interface adaptations from predefined UI-, task-, or
The architecture (Fig. 1) consists of the following parts: domain-model fragments.
• The data interfaces from the different devices (Tobii • The situation analytics component analyzes the
eye-tracker2, Empatica wristband3, Noldus Facereader4, sequences of raw situations with their parameters
metadata from the application) varying over time and condenses them to a situation
profile holding the most significant information about
the currently applying situations. Typical situations can
be described in the form of situation patterns. The
situation analytics component matches the raw
sequences to such situation patterns. A set of typical
domain-dependent and independent situation patterns is
available in the PaMGIS pattern repository. Such
situation patterns can serve as templates for creating
situation rules with the rule editor, where an action part
with one or more actions is added. New situation
patterns can be discovered by running offline data
mining tools, e.g., RapidMiner5, on the raw situation
sequences recorded during multiple sessions.
Fig. 1. SitAdapt 2.0 Architecture
• The recording component synchronizes the different
input records with a timestamp, records the eye- and
gaze-tracking signal of the user and tracks the emotional
video facial expression as a combination of the six basic
emotions (happy, sad, scared, disgusted, surprised, and
angry) based on Ekman’s model [4]. Other recorded
data about the user are, e.g., age-range and gender [15].
The stress-level and other biometric data are recorded in
real-time by a wristband. In addition, mouse movements
and keyboard logs are protocolled [11].
• The database writer stores the data from the recording
component and from the browser in the database in the
form of discrete raw situations and manages the
communication with the rule editor. Raw situations are
generated at each tick of a predefined time frame Fig. 2. SitAdapt 2.0 Rule Editor with an example rule exploiting visual
varying from 1/60s to 1s. emotions provided by Facereader
• The rule editor allows the definition and modification • The evaluation and adaptation component uses the
of situation rules, e.g. for specifying the different user situation profile provided by the situation analytics
states (e.g. if a happy state is observed, it will only component to decide whether an adaptation of the user
become relevant, if the state lasts more than five interface is meaningful and necessary at a specific
seconds and the grade of the emotion surpasses a certain moment. For this purpose the component evaluates
activation level). For experimenting with rule heuristics given situation rules. Whether an adaptation is
and observing users we built a prototypical web meaningful depends on the predefined purpose of the
application for long distance travel booking. In addition situation-aware target application. Goals to meet can
range from successful marketing activities in e-
2
www.tobii.com business, e.g. having the user buying an item from the
3
https://www.empatica.com/en-eu/research/e4/ e-shop or letting her or him browse through the latest
4
www.noldus.com/human-behavior-
5
research/products/facereader https://rapidminer.com
69
Modeling User Interface Adaptation for Customer-Experience Optimization
special offers, to improved customer experience levels, situation rules. Due to the limited space we can only discuss
or to meeting user desires defined by the hidden mental some of the most interesting findings in this section.
states of the user. The adaptation component finally
generates the necessary modifications of the interactive
target application.
These architectural components are necessary for enabling
the PaMGIS framework to support automated adaptive user
interfaces. In the user interface construction process, the
SitAdapt 2.0 evaluation and adaptation component cooperates
with the models of the interactive application (abstract,
concrete and final user interface model, context of use models, Fig. 3. Emotional reaction after finding the ideal product
task and concept model) and can also access the HCI-patterns
(not to be confused with the situation patterns) residing in the In order to illustrate the potential of situation-aware
PaMGIS repositories to build the necessary modifications of adaptation we present some real-world situation examples and
the user interface at runtime. possible adaptive reactions. In the first example (Fig. 3) a test
person is searching for a specific winter skin cream. Upon
III. AUTOMATING CUSTOMER EXPERIENCE OPTIMIZATION IN E-
reading the detailed description of the product Winter Silk
BUSINESS
Crème, the user’s emotional state significantly changes to
As a promising candidate domain for exploring situation happy. A situation rule could now exploit this knowledge to
analytics and situation-aware adaptation we have selected the give additional information about other winter products. The
e-business and e-commerce fields. In our current project we improved customer experience near the purchase touch point
focus on a commercial cosmetics e-business portal. can directly lead to a purchase of this and similar products.
A. Dynamic Adaptation Features In the next example (Fig. 4 and 5), the system has gathered
We have implemented dynamic adaptation features for pre- a priori knowledge about the varying gaze behavior of test
session, first session and recurring session adaptation. Typical persons, who are known customers of the business or who are
adaptation features are related to the following areas: here for the first time, by distinguishing between the lab-
created heat maps. The gaze behavior with respect to this
Visual appearance of the application image can be used to categorize anonymous users. The
• Gender or age specific coloring customer experience during the pre-purchase phase can be
improved. When the system assumes a returning customer, the
• Gender or age specific image selections
focus of her further customer journey will be put on showing
• Soothing image or color selection
aesthetic images, while in the other case more descriptive
• Age specific element size information will be given during the rest of the customer
• Element ordering or widget selection dependent on age journey.
or emotional state
• Screen contrast dependent on clock time, bio-physical
or emotional user state
• Font type, font size dependent on age, clock time, bio-
physical or emotional user state
New user interface or content elements
• Tutorial-offering at first session or dependent of user
age
• Help functionality, e.g. chat window, help menu item,
tool tips, UI element tips dependent on user behavior Fig. 4. Heatmap for customers of the business
• Personalized fields and panes (user- and behavior-
specific advertisement
Content-based adaptation
• Personalized product offers or suggestions
• Voucher offering dependent on user behavior
• User feedback functionality dependent on user behavior
B. Complex Situation Examples for a Cosmetics Portal
In a comprehensive user study with 9 female test persons
we tested the usability, user experience and emotional behavior Fig. 5. Heatmap for first time users of the website
for several scenarios when interacting with a real-world
cosmetics industry web-portal [1]. These tests served as the Another application area for using situation analytics in the
basis for finding domain-dependent situations and formulating e-business field is the evaluation and fine-tuning of pre-defined
70
Modeling User Interface Adaptation for Customer-Experience Optimization
customer personas, which are used for pre-runtime adaptations
and configurations of an application. Focusing on personas for Desktop_PC
a priori adaptation of the cosmetics portal can e.g. affect the ...
visual appearance, the product content structure, the level of
the product description language, the appearance of special product_view
advertisements, or the gaming and social media orientation of
model_AUI_product_view_1
the website. Are test persons behaving like their respective
personas or are there significant deviations from the expected model_CUI_product_view_1
behavior? This can be evaluated by comparing the situation product_view
profiles that come up during persona-adapted user tests with product_view
the typical situation profiles specified during the persona
definition process. Vice-versa SitAdapt 2.0 can classify model_concept_Product_view_1
unknown customers or first time visitors into one of the given
persona categories by analyzing the situations appearing during Product_Textbox_Product_1(10 s)
the session and by analyzing the users’ behavior after situation- 30-50
rule triggered adaptations. female
happy
All of these user observations and behavior evaluations as
well as the adaptations of the interactive software are currently
done in our situation analytics lab environment. The rapid normal
evolution of visual and biophysical user tracking and green
monitoring technology will enable situation-aware individual ...
adaptations for the end user in the near future.
C. Adaptation Modeling and Adaptation Process
By applying our MBUID approach, the modeling, The evaluation and adaptation component examines the
generation and adaptation of the target website is done with the situation profile to decide, if an adaptation can take place. This
help of the PaMGIS framework and the integrated SitAdapt 2.0 is usually achieved by activating the responsible sub-set of
system (Fig. 1). The PaMGIS framework is based on the situation rules in the rule editor (Fig 2.). Alternatively, the
Cameleon Reference Framework (CRF) [2], [18]. In the programmer or web designer can directly provide code for
construction process first of all, the abstract user interface interpreting the situation profile in the web application client or
model (AUI) is generated from the information contained in server, which is triggered when the user interacts with specific
the domain model of the application that includes a task model elements of the user interface
and the concept model (i.e. business model) and defines
abstract user interface objects that are still independent of the In the concrete example, the situation rules specify that
context of use. This AUI model can then be transformed into a additional information about other winter cosmetic products
concrete user interface model (CUI) that already exploits the should be displayed in this particular situation. The decision
context of use model that includes the user, device, UI-Toolkit can be refined by also taking into account the user persona, if it
and environment model, and the dialog model, which is is already known. For a strictly goal-oriented persona, a new
responsible for the dynamic user interface behavior. In the next window with additional product information may be shown.
step, the final user interface model (FUI) is generated by For a more cautious persona, a question text may appear,
parsing the CUI model and by exploiting the context of use whether additional product information about winter products
model and the layout model [15]. is welcome.
The first displayed version of the product e-commerce The evaluation and adaptation component now starts the
website is already adapted to the user. For example by using adaptation process, which leads to the generation of a modified
the age and target device information from the context of use user interface. A new CUI and subsequently a FUI is generated
model. The SitAdapt 2.0 system permanently monitors the user and displayed to the user. The PaMGIS modeling environment
and recognizes the situations she or he is experiencing while must provide all the necessary models, model variants and
viewing the webpage and interacting with the user interface. model fragments necessary for user interface modifications.
The evaluation component recognizes in the first example (Fig. User interface models may contain links to HCI-patterns that
3), that the user reads the text attentively and that the level of can facilitate user interface code generation. More complex
the happy emotion surpasses a given minimum duration (e.g. adaptations may also activate different tasks specified in the
more than 5 seconds). These data come from the raw situation task model and require the activation of non-UI service code.
sequences stored in the database by the recording component. By observing the users’ emotional behavior after such
The various inputs from the Facereader (emotion) and eye adaptations, the quality of the situation rules and the respective
tracking (text screen field) and the metadata of the website adaptations can be evaluated and rated. Such information can
(URL) were evaluated just in time by the situation analytics be used offline for refining the situation rule set for later use.
component that has created the following situation profile of
the current situation:
71
Modeling User Interface Adaptation for Customer-Experience Optimization
IV. CONCLUSION Inferring Emotion Through Human Computer Interaction Devices," MIS
Quarterly, (41: 1) pp.1-21.
SitAdapt 2.0 is an advanced architecture for automating [12] Hudlicka, E. and M. D. Mcneese, “Assessment of user affective and
user interface adaptation for responsive web-applications that belief states for interface adaptation: Application to an Air Force pilot
were constructed with the PaMGIS MBUID framework. This task,” User Model. User-Adapt. Interact., vol. 12, no. 1, pp. 1–47, 2002.
paper has demonstrated the flexibility and versatility of this [13] Lemon, K.N., Verhoef, P.C.: Understanding Customer Experience
new approach by testing it with different scenarios and touch Throughout the Customer Journey, J. of Marketing: AMA/MSI Special
points of the customer journey in a commercial e-business Issue, Vol. 80 (Nov. 2016), pp. 69-97
platform for cosmetics products. It could be demonstrated that [14] Märtin, C., Herdin, C.: Enabling Decision-Making for Situation-Aware
Adaptations of Interactive Systems, Proc. 10th Forum Media
emotion recognition combined with eye- and gaze-tracking can Technology, FMT 2017, 29-30 Nov., St. Pölten, Austria, (2017)
be a powerful method for assessing situations and finding [15] Märtin, C., Herdin, C., Engel, J.: Model-based User-Interface
possible adaptations at application runtime. By the lab-based Adaptation by Exploiting Situations, Emotions and Software Patterns,
observation of users through multiple visual and physical Proc. CHIRA 2017, Funchal, Madeira, Portugal, 31 October-2
channels we could establish a basis for improving the customer November, SCITEPRESS (2017), pp. 50-59
experience in the pre-purchase and after purchase phases, [16] Märtin, C., Rashid, S., Herdin, C.: Designing Responsive Interactive
because the gathered knowledge can be used for optimizing a Applications by Emotion-Tracking and Pattern-based Dynamic User
Interface Adaptation. Proc. of HCII 2016, Toronto, 17-22 July, Vol. III,
priori and persona-based adaptations and can lead to improved Springer LNCS, 2016, pp. 28-36
situation rules. A follow-up study that is currently under way
[17] Meixner, G., Calvary, G., Coutaz, J.: Introduction to model-based user
will evaluate the effectiveness of the persona-based a priori and interfaces. W3C Working Group Note 07 January 2014.
situation-aware runtime adaptations for the perceived http://www.w3.org/TR/mbui-intero/. Accessed 27 May 2015
individual customer experience. [18] Melchior, J., Vanderdonckt, J., Van Roy, P.: A Model-Based Approach
for Distributed User Interfaces, Proc. EICS ‘2011, ACM (2011), pp. 11-
In the future we will combine SitAdapt 2.0 functionality 20
with web and big-data analytics to further improve the [19] Meudt, S. et al.: Going Further in Affective Computing: How Emotion
customer experience of e-business applications and to evaluate, Recognition Can Improve Adaptive User Interaction, in: A. Esposito and
which of the applied user monitoring technologies can be L.C. Jain (eds.), Toward Robotic Socially Believable Behaving Systems
helpful in situation-aware end-user environments. – Vol. 1, Intelligent Systems Reference Library 105, Springer Int.
Publishing Switzerland (2016), pp. 73-103
REFERENCES [20] Nasoz, F.: “Adaptive intelligent user interfaces with emotion
recognition, University of Central Florida Orlando, Florida, 2004.
[1] Bissinger, B.C.: Messung und Analyse von bio-physischen und visuellen
Daten zur Optimierung der User Experience und des Digitalmarketings, [21] Picard, R.:”Recognizing Stress, Engagement, and Positive Emotion”,
Master Thesis, Business Information Systems, Augsburg University of Proc. IUI 2015, March 29-April 1, 2015, Atlanta, GA, USA, pp. 3-4
Applied Sciences, 2018 [22] Schilit, B.N., Theimer, M.M.: Disseminating Active Map Information to
[2] Calvary, G., Coutaz, J., Bouillon, L. et al., 2002. “The CAMELEON Mobile Hosts, IEEE Network, vol. 8, no. 5, pp. 22–32, (1994)
Reference Framework”. Retrieved August 25, 2016 from [23] Schmidt, A. Biosignals in Human-Computer Interaction, Interactions
http://giove.isti.cnr.it/projects/cameleon/pdf/CAMELEON%20D1.1RefF Jan-Feb 2016, (2016), pp. 76-79
ramework.pdf [24] Yigitbas, E., Sauer, S., Engels, G.: A Model-Based Framework for
[3] Chang, C.K.: Situation Analytics: A Foundation for a New Software Multi-Adaptive Migratory User Interfaces. In: Proceedings of the HCI
Engineering Paradigm, IEEE Computer, Jan. 2016, pp. 24-33 2015, Part II, LNCS 9170, Springer (2015), pp. 563-572
[4] P. Ekman, “An argument for basic emotions,” Cogn. Emot., vol. 6, no.
3–4, pp. 169–200, 1992.
[5] Engel, J., Märtin, C., Forbrig, P.: Practical Aspects of Pattern-supported
Model-driven User Interface Generation. Proc. HCI International 2017,
Vancouver, Canada, 9-14 July, Vol. I, Springer LNCS, 2017, pp. 397-
414
[6] Engel, J., Märtin, C., Forbrig, P.: A Concerted Model-driven and
Pattern-based Framework for Developing User Interfaces of Interactive
Ubiquitous Applications, Proc. First Int. Workshop on Large-scale and
Model-based Interactive Systems, Duisburg, (2015), pp. 35-41
[7] Flach, J.M., Mulder, M, Van Paassen, M.M.: The Concept of the
Situation in Psychology, in: Banbury, S. and Tremblay, S. (eds): A
Cognitive Approach to Situation Awareness: Theory and Applications,
Ashgate Publisching, Oxon (UK), (2004), pp. 42-60
[8] Galindo, J. et al.: Using user emotions to trigger UI adaptation, Proc.
12th Int. Conf. on Research Challenges in Information Science (RCIS),
(2018)
[9] Halvorsrud, R., Kvale, K., Følstad, A.: Improving Service Quality
through Customer Journey Analysis, J. of service theory and practice,
vol. 26, 6, (2016), pp. 840-867
[10] Herdin, C., Märtin, C., Forbrig, P.: SitAdapt: An architecture for
situation-aware runtime adaptation of interactive systems. Proc. HCI
International 2017, Vancouver, Canada, 9-14 July, Vol. I, Springer
LNCS, 2017, pp. 447-455
[11] Hibbeln, Martin; Jenkins, Jeffrey L.; Schneider, Christoph; Valacich,
Joseph S.; and Weinmann, Markus. 2017. "How Is Your User Feeling?
72