-The customer journey in digital marketing defines several touch points, where interested users can directly interact with an e-business platform. In order to convert a user into a buyer, persona-based a priori adaptations of the user interface can be combined with dynamic adaptations at runtime with the goal to optimize individual customer experience and guide task accomplishment. This paper examines customer experience optimization for scenarios from a cosmetics industry e-business portal with the SitAdapt 2.0 system. Dynamic adaptations are triggered by situation rules based on the continuous analysis of the users' varying cognitive and emotional situations during a session. The model-based adaptation process exploits models and patterns for the rapid generation of user interface modifications.
Digitalization in marketing can be seen as a straightforward
approach to designing and implementing IT-based solutions for
the generic steps of the customer journey. A customer journey
is a customer’s interaction at several touch points with a
service or several services of one or more service providers in
order to achieve a specific goal [9]. More focused on
purchasing a product, the customer journey can be defined as
an iterative process that includes touch point based interactions
with a provider or a business during a pre-purchase, a purchase,
and a post-purchase phase [
UX during the customer journey is often described as
customer experience. As an extract and synthesis of earlier
research efforts customer experience can be seen as “a
multidimensional construct” that focuses on “a customer’s
cognitive, emotional, behavioral, sensorial, and social”
reactions to the offerings of a provider or a business “during
the customer’s entire purchase journey” [
With SitAdapt [
The paper includes the following main contributions:
Discussion of a new model-based approach [
Detailing the adaptation process and workflow for the e-business domain
Chapter II introduces the SitAdapt 2.0 system with its new rule editor. Chapter III first introduces possible adaptation features and defines example scenarios for generic and individual situations that are occurring in different phases of the customer journey when visiting a cosmetics business portal. Some of the possible SitAdapt 2.0 use-cases are demonstrated. After this, the chapter discusses the modeling and generation of adaptations. Chapter IV concludes the paper. 1 Part of this work was carried out in cooperation with Dr. Grandel GmbH, Augsburg, Germany. We greatly acknowledge the opportunity to run the SitAdapt 2.0 tools and user tests on their enterprise e-business platform.
The SitAdapt 2.0 runtime environment is integrated into the PaMGIS (Pattern-based Modeling and Generation of Interactive Systems) development framework. The framework allows for modeling and generating responsive behavior in the user interface and has now been enhanced towards dynamic adaptation by situation interpretation at runtime.
The architecture (Fig. 1) consists of the following parts: •
The data interfaces from the different devices (Tobii eye-tracker2, Empatica wristband3, Noldus Facereader4, metadata from the application)
The recording component synchronizes the different
input records with a timestamp, records the eye- and
gaze-tracking signal of the user and tracks the emotional
video facial expression as a combination of the six basic
emotions (happy, sad, scared, disgusted, surprised, and
angry) based on Ekman’s model [4]. Other recorded
data about the user are, e.g., age-range and gender [
The database writer stores the data from the recording component and from the browser in the database in the form of discrete raw situations and manages the communication with the rule editor. Raw situations are generated at each tick of a predefined time frame varying from 1/60s to 1s.
The rule editor allows the definition and modification of situation rules, e.g. for specifying the different user states (e.g. if a happy state is observed, it will only become relevant, if the state lasts more than five seconds and the grade of the emotion surpasses a certain activation level). For experimenting with rule heuristics and observing users we built a prototypical web application for long distance travel booking. In addition 2 www.tobii.com 3 https://www.empatica.com/en-eu/research/e4/ 4 www.noldus.com/human-behaviorresearch/products/facereader • • we used results from our cosmetics industry user study [1] for finding plausible situation rules. Fig. 2 shows the creation of a simple situation rule with two conditions and one action. In this case only a dialog with the user is created. However, situation rules can also activate HCI-patterns in the PaMGIS pattern repository. These patterns are exploited at runtime to generate user interface adaptations from predefined UI-, task-, or domain-model fragments.
The situation analytics component analyzes the sequences of raw situations with their parameters varying over time and condenses them to a situation profile holding the most significant information about the currently applying situations. Typical situations can be described in the form of situation patterns. The situation analytics component matches the raw sequences to such situation patterns. A set of typical domain-dependent and independent situation patterns is available in the PaMGIS pattern repository. Such situation patterns can serve as templates for creating situation rules with the rule editor, where an action part with one or more actions is added. New situation patterns can be discovered by running offline data mining tools, e.g., RapidMiner5, on the raw situation sequences recorded during multiple sessions.
The evaluation and adaptation component uses the situation profile provided by the situation analytics component to decide whether an adaptation of the user interface is meaningful and necessary at a specific moment. For this purpose the component evaluates given situation rules. Whether an adaptation is meaningful depends on the predefined purpose of the situation-aware target application. Goals to meet can range from successful marketing activities in ebusiness, e.g. having the user buying an item from the e-shop or letting her or him browse through the latest special offers, to improved customer experience levels, or to meeting user desires defined by the hidden mental states of the user. The adaptation component finally generates the necessary modifications of the interactive target application.
These architectural components are necessary for enabling the PaMGIS framework to support automated adaptive user interfaces. In the user interface construction process, the SitAdapt 2.0 evaluation and adaptation component cooperates with the models of the interactive application (abstract, concrete and final user interface model, context of use models, task and concept model) and can also access the HCI-patterns (not to be confused with the situation patterns) residing in the PaMGIS repositories to build the necessary modifications of the user interface at runtime.
III. AUTOMATING CUSTOMER EXPERIENCE OPTIMIZATION IN E
BUSINESS
As a promising candidate domain for exploring situation analytics and situation-aware adaptation we have selected the e-business and e-commerce fields. In our current project we focus on a commercial cosmetics e-business portal.
We have implemented dynamic adaptation features for presession, first session and recurring session adaptation. Typical adaptation features are related to the following areas: Visual appearance of the application • Gender or age specific coloring • Gender or age specific image selections • Soothing image or color selection • Age specific element size • Element ordering or widget selection dependent on age or emotional state • Screen contrast dependent on clock time, bio-physical or emotional user state • Font type, font size dependent on age, clock time, biophysical or emotional user state New user interface or content elements • Tutorial-offering at first session or dependent of user age • Help functionality, e.g. chat window, help menu item, tool tips, UI element tips dependent on user behavior • Personalized fields and panes (user- and behaviorspecific advertisement
• Personalized product offers or suggestions • Voucher offering dependent on user behavior • User feedback functionality dependent on user behavior
In a comprehensive user study with 9 female test persons we tested the usability, user experience and emotional behavior for several scenarios when interacting with a real-world cosmetics industry web-portal [1]. These tests served as the basis for finding domain-dependent situations and formulating situation rules. Due to the limited space we can only discuss some of the most interesting findings in this section.
In order to illustrate the potential of situation-aware adaptation we present some real-world situation examples and possible adaptive reactions. In the first example (Fig. 3) a test person is searching for a specific winter skin cream. Upon reading the detailed description of the product Winter Silk Crème, the user’s emotional state significantly changes to happy. A situation rule could now exploit this knowledge to give additional information about other winter products. The improved customer experience near the purchase touch point can directly lead to a purchase of this and similar products.
In the next example (Fig. 4 and 5), the system has gathered a priori knowledge about the varying gaze behavior of test persons, who are known customers of the business or who are here for the first time, by distinguishing between the labcreated heat maps. The gaze behavior with respect to this image can be used to categorize anonymous users. The customer experience during the pre-purchase phase can be improved. When the system assumes a returning customer, the focus of her further customer journey will be put on showing aesthetic images, while in the other case more descriptive information will be given during the rest of the customer journey.
Another application area for using situation analytics in the e-business field is the evaluation and fine-tuning of pre-defined customer personas, which are used for pre-runtime adaptations and configurations of an application. Focusing on personas for a priori adaptation of the cosmetics portal can e.g. affect the visual appearance, the product content structure, the level of the product description language, the appearance of special advertisements, or the gaming and social media orientation of the website. Are test persons behaving like their respective personas or are there significant deviations from the expected behavior? This can be evaluated by comparing the situation profiles that come up during persona-adapted user tests with the typical situation profiles specified during the persona definition process. Vice-versa SitAdapt 2.0 can classify unknown customers or first time visitors into one of the given persona categories by analyzing the situations appearing during the session and by analyzing the users’ behavior after situationrule triggered adaptations.
All of these user observations and behavior evaluations as well as the adaptations of the interactive software are currently done in our situation analytics lab environment. The rapid evolution of visual and biophysical user tracking and monitoring technology will enable situation-aware individual adaptations for the end user in the near future.
By applying our MBUID approach, the modeling,
generation and adaptation of the target website is done with the
help of the PaMGIS framework and the integrated SitAdapt 2.0
system (Fig. 1). The PaMGIS framework is based on the
Cameleon Reference Framework (CRF) [2], [
The first displayed version of the product e-commerce website is already adapted to the user. For example by using the age and target device information from the context of use model. The SitAdapt 2.0 system permanently monitors the user and recognizes the situations she or he is experiencing while viewing the webpage and interacting with the user interface. The evaluation component recognizes in the first example (Fig. 3), that the user reads the text attentively and that the level of the happy emotion surpasses a given minimum duration (e.g. more than 5 seconds). These data come from the raw situation sequences stored in the database by the recording component. The various inputs from the Facereader (emotion) and eye tracking (text screen field) and the metadata of the website (URL) were evaluated just in time by the situation analytics component that has created the following situation profile of the current situation: <SituationProfile> <TargetApplication>Desktop_PC ... <Situation_product_view> <FUI_link> product_view <AUI_link> model_AUI_product_view_1 <CUI_link> model_CUI_product_view_1 <Dialog_link> product_view <Task_link> product_view <Concept_link> model_concept_Product_view_1
<Eye_Tracking> Product_Textbox_Product_1(10 s) <USRUA_Age_Range>30-50 <USRUA_Gender>female <EmotionalState> happy <UserPsychologicalState> <BiometricState> <Pulse> normal <StressLevel> green ...
</Situation_product_view> </SituationProfile>
The evaluation and adaptation component examines the situation profile to decide, if an adaptation can take place. This is usually achieved by activating the responsible sub-set of situation rules in the rule editor (Fig 2.). Alternatively, the programmer or web designer can directly provide code for interpreting the situation profile in the web application client or server, which is triggered when the user interacts with specific elements of the user interface
In the concrete example, the situation rules specify that additional information about other winter cosmetic products should be displayed in this particular situation. The decision can be refined by also taking into account the user persona, if it is already known. For a strictly goal-oriented persona, a new window with additional product information may be shown. For a more cautious persona, a question text may appear, whether additional product information about winter products is welcome.
The evaluation and adaptation component now starts the adaptation process, which leads to the generation of a modified user interface. A new CUI and subsequently a FUI is generated and displayed to the user. The PaMGIS modeling environment must provide all the necessary models, model variants and model fragments necessary for user interface modifications. User interface models may contain links to HCI-patterns that can facilitate user interface code generation. More complex adaptations may also activate different tasks specified in the task model and require the activation of non-UI service code.
By observing the users’ emotional behavior after such adaptations, the quality of the situation rules and the respective adaptations can be evaluated and rated. Such information can be used offline for refining the situation rule set for later use.
SitAdapt 2.0 is an advanced architecture for automating user interface adaptation for responsive web-applications that were constructed with the PaMGIS MBUID framework. This paper has demonstrated the flexibility and versatility of this new approach by testing it with different scenarios and touch points of the customer journey in a commercial e-business platform for cosmetics products. It could be demonstrated that emotion recognition combined with eye- and gaze-tracking can be a powerful method for assessing situations and finding possible adaptations at application runtime. By the lab-based observation of users through multiple visual and physical channels we could establish a basis for improving the customer experience in the pre-purchase and after purchase phases, because the gathered knowledge can be used for optimizing a priori and persona-based adaptations and can lead to improved situation rules. A follow-up study that is currently under way will evaluate the effectiveness of the persona-based a priori and situation-aware runtime adaptations for the perceived individual customer experience.
In the future we will combine SitAdapt 2.0 functionality with web and big-data analytics to further improve the customer experience of e-business applications and to evaluate, which of the applied user monitoring technologies can be helpful in situation-aware end-user environments.