=Paper=
{{Paper
|id=Vol-3222/paper2
|storemode=property
|title=Psychological User Characteristics and Meta-Intents in a Conversational Product Advisor
|pdfUrl=https://ceur-ws.org/Vol-3222/paper2.pdf
|volume=Vol-3222
|authors=Yuan Ma,Timm Kleemann,Jürgen Ziegler
|dblpUrl=https://dblp.org/rec/conf/recsys/MaK022
}}
==Psychological User Characteristics and Meta-Intents in a Conversational Product Advisor==
https://ceur-ws.org/Vol-3222/paper2.pdf
Psychological User Characteristics and Meta-Intents
in a Conversational Product Advisor
Yuan Ma, Timm Kleemann and Jürgen Ziegler
University of Duisburg-Essen, Duisburg, Germany
Abstract
We present a study investigating psychological characteristics of users of a GUI-style conversational
recommender system in a real-world application case. We collected data of 496 customers of an online
shop using a conversational product advisor (CPA), using questionnaire responses concerning decision-
making style and a set of meta-intents, a concept we propose to represent high-level user preferences
related to the decision process in a CPA. We also analyzed anonymized data on users’ interactions in
the CPA. Concerning general decision-making style, we could identify two clusters of users who differ
in their scores on scales measuring rational and intuitive decision-making. We found evidence that
rationality and intuitiveness scores are differently correlated with the proposed meta-intents such as
efficiency orientation, interest in detail, and openness for guidance. Relations with interaction data could
be observed between rationality/intuitiveness scores and overall time spent in the CPA. Trying to classify
users’ decision style from their interactions, however did not yield positive results. Despite the limitation
that only a single CPA was studied in a single domain, our results provide evidence that the proposed
meta-intents are linked to the general decision-making style of a user and can thus be instrumental
in translating general decision-making factors into more concrete design guidance for CPA and their
potential personalization.
Keywords
conversational UI design, interactive behavior analysis, decision making, influence of psychological
factors on interaction
1. Introduction
Conversational recommender systems (CRS[1]) have been gaining increased attention in re-
search and industry in recent years [2, 3]. Generally, conversational techniques can provide
users with strong guidance to achieve their goals combined with a high level of flexibility in
expressing their needs. Jannach et al. [4] distinguish between natural language-based, form-
based, and critiquing approaches. Due to the advances in NLP techniques in recent years,
natural language-based CRS have become the subject of extensive research. However, despite
various advantages, such as human-like dialog and a high level of flexibility, their capabilities
are still limited and users may need to adapt to the underlying (invisible) vocabulary and domain
coverage to interact successfully.
Despite the recent surge in NLP-based CRS, GUI-based forms of conversation, leading users
IntRS’22: Joint Workshop on Interfaces and Human Decision Making for Recommender Systems, September 22, 2022,
Seattle, US (hybrid event).
$ yuan.ma@uni-due.de (Y. Ma); timm.kleemann@uni-due.de (T. Kleemann); juergen.ziegler@uni-due.de
(J. Ziegler)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
through a pre-defined branching structure of form-based questions and response options,
constitute an alternative that has a number of advantages from a user perspective. They provide
a high level of transparency and user guidance, minimize errors, and incorporate expert domain
knowledge relevant to decision-making. GUI-based conversational product advisors (CPA),
have thus become frequently used tools in e-commerce websites. Designing CPA to support
users in finding a suitable item efficiently and with a positive user experience, however, involves
a number of critical challenges. Questions need to be formulated at an appropriate level of
abstraction, for example, asking either about the intended use of the product or about some
specific technical features. Dialog flow should follow the user’s likely mental decision process,
providing sufficient flexibility without becoming overly complex, and recommendations should
be presented in appropriate numbers and with an appropriate level of detail.
To address these design challenges for CPA, a thorough understanding of user needs and
their decision-making style is needed. Little research, however, has investigated the influence
of psychological user characteristics and intents in the context of CPA thus far [5]. In this
paper, we explore psychological characteristics of CPA users under two different objectives.
First, we aim at obtaining a deeper understanding of psychological characteristics of CPA
users, based on responses from questionnaire instruments. Here, we distinguish between stable
individual traits including personality factors [6] and decision-making style [7], and, second,
task-oriented characteristics that represent general user preferences when interacting with a
CPA, such as obtaining detailed information about items or comparing products. We call the
latter characteristics meta-intents since they describe user goals that are more general and
high-level than the search goals typically extracted through intent detection methods in CRS.
As a second objective, we investigate whether psychological user characteristics can be
predicted from their behavior in a CPA. Meta-intents such as interest in detail might influence,
for example, the time spent in different steps of an advisor dialog. If such user characteristics
could be derived from their interaction, this information would provide insights for optimally
designing the dialogs, or for personalizing them.
In this paper, we describe a first study analyzing these questions and present the results
of an online study conducted with users of a real CPA in productive, commercial use. Our
contribution is twofold: We provide insights about CPA users’ decision-making style (rationality
vs. intuitiveness) and the relevance of different meta-intents we propose for CRS. Furthermore,
we discuss the possibility of predicting psychological characteristics from users’ interaction
behavior in CPA in the light of our case study data.
2. Related work
Conversational Recommender Systems (CRS) have become a rapidly growing and popular
research area because they provide a flexible, human-like multi-turn dialog for preference
elicitation, which is essential for generating personalized recommendations [8]. Jannach et al.
[4] distinguish three types of CRS differing in the style and structure of the interaction used:
natural language-based, form-based, and critiquing-based. NLP-based CRS have received
considerable interest recently due to the advancements in natural language processing. They
typically use a question-answer format [9], sometimes mixing language and GUI elements
in mixed-mode systems [10]. Form-based CRS present questions and answers in a GUI style,
leading users through a predefined dialog structure. This type of CRS has many advantages
as they provide guidance to the users, avoid errors, and can incorporate domain knowledge.
Especially usage-related questions, asking users about the tasks they want to accomplish with
a product, are important for users who have only limited knowledge about technical item
properties [11]. CPA are thus often used as product advisors in e-commerce sites. An early
example of such knowledge-based CPA is ADVISOR Suite [12]. Despite their relevance, very
limited research has as yet studied users’ interaction needs in CPA and related design questions.
For example, Papenmeier et al. [13] investigated human advisory dialogs, identifying some
recurring strategies such as funneling to successively narrow down the space of potential items.
Kleemann et al. [14] investigated user behavior when using a CPA in combination with other
decision aids.
To provide design guidance for CPA and to potentially adapt them to the individual user,
a deeper understanding of the psychological factors influencing users’ decision-making and
interaction behavior in CRS and CPA is required. For recommender systems in general, the
influence of psychological characteristics on users’ preference construction and decision making
has been shown repeatedly [15]. Lex et al. [16] distinguish between factors related to cognition,
personality, and emotion. The influence of psychological characteristics such as the Big Five
personality factors (e.g. [17, 18]), Need for Cognition [19], or cognitive biases [20] has been
studied in several works. However, these studies mostly aim at better understanding user
preferences with respect to the recommended items and at improving their accuracy. In contrast,
the relationship between psychological factors and the design of advisory dialogs in CPA remains
an underexplored area. Especially theories related to human decision-making styles appear to
be promising points of departure for studying this relation. The distinction between rational
and intuitive decision-making styles [21] or cognitive styles such as the need for cognition may
influence users’ assessment of CPA dialogs. More domain-specific theories such as Shopping
Orientation [22, 23], distinguishing between task-focused and experiential shopping are also of
interest. However, none of these approaches has yet been applied to CPA.
User goals and preferences when interacting with a CRS may be located on different levels of
abstraction. Low-level preferences refer to concrete properties of the desired item (often called
Intents in CRS, specifically Add Details [24]). On a more abstract level, meta-level preferences
[25]. [26] may represent high-level product dimensions (such as economy and safety). Yet, they
also refer to product-related aspects. Meta-level preferences that relate to the conversation style
and type of questions in a CPA have, to our knowledge, not been studied yet.
Inferring psychological user characteristics and needs from different sources, such as email
[27] social media posts [28], or smartphone use [29] has been subject of a considerable body of
work, giving indications for the feasibility of such methods. However, it is an open question
whether psychological characteristics can also be derived from users’ interaction behavior in
CPA where only a very limited number of signals can be extracted from the GUI dialog to be
used as predictors. Investigating this research question is one of the goals of this paper.
3. User characteristics and meta-intents
To investigate differences in CPA users’ psychological properties, we hypothesized that decision-
making style might influence users’ usage and interaction in CPA. Accordingly, we applied
instruments to measure these properties, using the Decision Styles Scale (DSS) [21] for distin-
guishing rational and intuitive decision-making styles.
While general decision-making styles, e.g. rationality and intuitiveness, apply to arbitrary
decision contexts, we also aimed at capturing users’ preferences at a more specific, yet still
abstract level. These meta-intents should bridge the gap between item-level intents and general
decision-making style, and should also relate to the design and question-asking style in CRS and,
specifically, CPA. They might also be relevant for more general recommendation scenarios. We
postulated the following set of meta-intents (with sample questionnaire items in parentheses),
partly related to general usage factors such as efficiency, effectiveness, and user guidance. We
see this list as a first step towards defining factors relevant for users’ decision-making process
in CPA which is neither complete nor final.
• Efficiency orientation (For me, finding a suitable product quickly is more important
than exploring all options.)
• Diversity orientation (When shopping online, I tend to explore a diverse range of
products that might interest me.)
• Goal focus (I usually have a clear idea of what I want before visiting an online shop. I
often only make up my mind once I see the available choices.)
• Openness for guidance (I appreciate it if a shop recommends products I might like.)
• Interest in detail (I usually gather as much information as possible about products that
I want to buy.)
• Brand awareness (The brand of a product is an important factor for my decision.)
• Comparison orientation (Comparing the features of different candidate products is
important for me.)
• Scope of choice (When the system recommends products, I rather like to see a longer
list rather than a short one.)
4. Experiment
To investigate CPA users’ psychological characteristics, both at the level of decision-making
style and CPA-specific meta-intents , as well as possible relations with their interaction behavior,
we performed a long-term online experiment with users of a CPA embedded in an online shop.
We hypothesized that there are relations between general traits (decision-making style) and
individual preferences with respect to the meta-intents formulated. We also assumed that
decision-making style and meta-intents would influence user interaction behavior, for example,
the time spent on the dialog as a whole as well as for different categories of advisor questions.
4.1. Method
We assessed the behavior of real users rather than relying on an artificial scenario in a labo-
ratory experiment in order to obtain realistic data. For this purpose, we applied existing and
self-developed questionnaires and linked them to a real-world CPA, designed for supporting
customers when purchasing a new mattress. The domain was chosen both for reasons of
opportunity (it is notoriously difficult for academics to get access to real e-commerce sites to
perform experiments) and because mattress selection is a sufficiently complex decision process,
involving a variety of decision criteria. Furthermore, the site offers a large variety of products,
not limited to a single brand or product type and the embedded CPA also comprises a range of
different question types.
Our study involves real users who enter and browse an online mattress shop and who freely
choose to engage with a GUI-based CPA. In the CPA dialog, the user is first confronted with a
series of six predetermined questions, after answering all questions (note that different question
paths will appear according to different answers, but all the paths have the same depth of 6
questions), the recommended products will appear, at this point they can freely select, compare
and view products. All visitors who had completed the advisory process were then invited to
participate in our study with a pop-up on the results page.1 We only asked users who had fully
gone through the advisor if they wanted to participate in the study, firstly to ensure that we
could conduct our analyses mainly with complete user interaction data. Second, we assumed
that users might be more motivated to participate in a study and provide meaningful responses
if they did not drop out of the advisory process early.
Once they clicked to participate in our study, a questionnaire opened in a separate browser
tab, avoiding distracting them from interacting with the online store or the results of their
advisory process. Thus, the participants could answer the questions at their discretion, either
directly or at a later time. There was no compensation for participation. The questionnaire was
presented using a self-hosted instance of the tool LimeSurvey.2 All items had 5-point Likert
response scales.
The questionnaire had two parts. First, we measured participants’ decision style using
Decision Styles Scale (DSS) [21]. Next, we assessed the participants’ meta-intents , discussed in
Section 3. Due to strict data protection requirements imposed by the online store and advisor
provider, no demographic data was collected. Furthermore, participants were provided with
the questionnaire in the language of the online store and the advisor (Dutch). Through an
identification code that was automatically passed to the questionnaire, we were able to match
the participants who took part in the study to their respective advisor’s interaction data. Also,
the interaction data of the advisor were completely anonymized.
4.2. Participants
The study ran over a period of five months. During this time, 18.914 visitors of the online store
finished the CPA process. Of these, 3.782 visitors opened the questionnaire. Overall, a total of
1
“Before you go... Would you like to spend 3 minutes answering a questionnaire so we can better understand your
needs and provide you with higher-quality services?”
2
https://www.limesurvey.org/
506 visitors, subsequently referred as participants, responded to all questions. After cleaning
the data by removing outliers (users with less then 10 advisor interactions) and participants
with implausible response combinations (whose decision-making scale scores or meta-intents
scores have all the same value), we were left with both questionnaire and advisor interaction
data from 496 participants.
4.3. Advisor and Interaction Data
The advisor is composed out of 12 questions, which involves 4 question types (with a real CPA
example in parentheses):
• Purpose-related. What is the intended use of this product? (What feeling do you want
when you lie down? )
• Feature-related. Questions about product features. (What size do you want? )
• Context-related. What is the environment of using the product. (I’m sleeping . . . ? (prefer
not to say; alone; with a partner))
• User-related. Questions about user preferences. (Select your weight for the perfect sleeping
comfort)
Table 1
Interactive features and descriptions. A total of 14 interaction features will be further used to train and
test the classification model that attempts to predict users’ decision-making styles.
Feature groups Interaction features Description
Time spent . . .
duration .. . . in the advisor.
Nth-Q-duration . . . on N-th question. (N is ordinal from 2 to 6)
user-related-duration . . . on user related questions.
Durations
feature-related-duration . . . on feature related questions.
context-related-duration . . . on context related questions.
purpose-related-duration . . . on purpose related questions.
Total number of . . .
Advisor actions advisor-interactions . . . events the user generates while interacting with the advisor.
Total number of . . .
Product actions clickouts . . . clicks to view product details.
Total number of . . .
. . . events the user interact with the online-shopping website.
Overall events including active (i.e. selecting answers, switch between questions) and passive
(i.e. webpage focus deactivation) events.
webpage-focus-deactivation . . . deactivating this online-shopping website.
Note that different question paths will appear according to different answers, but all the paths
have the same depth of 6 questions. This is due to the fixed design of the advisor which we did
not alter. For each question step, besides choosing one answer, users have various interaction
options: They may navigate back and forward, change their answers, or completely restart the
advisory process. Once users have completed the CPA process, suitable products are displayed
on a results page. Here, users have the option to browse through further results and to change
the sort order of the results. For each product shown, a summary of the product features that
match the given answers is displayed, by clicking, it opens a separate detailed product page.
We logged all interactions that occurred in the advisor and on the results page. This included
data on the total time spent during the advisory process, as well as the time spans between
each interaction. We also logged which products were clicked on by users in the results list
(referred to as clickouts in Table 1). In addition, we recorded whether and for how long the
browser window containing the advisor was inactive or merely opened in the background.
All collected event types and the features subsequently generated from the data in a feature
engineering process are summarized in Table 1. These features were used for classifying the
decision-making styles of the participants. For this attempt, we used three popular machine
learning models: support vector machine (SVM), multi-layer perceptron (MLP) and decision
tree (DT). Limited by the number of datasets we did not use deep neural networks.
5. Results
Figure 1: K -means clustering of rationality and intuitiveness factors. The X-axis is the intuitive score,
Y-axis is the rational score. The blue-filled circle indicates the user is clustered to the rationality group
(200 participants), red indicates the intuitiveness group (296 participants), and black crosses are centroids
of groups. Due to data overlapping, we present each user with a filled circle with 10% transparency,
light colors mean sparse, and dark colors mean dense.
Rationality vs. Intuitiveness Clustering With respect to decision making, we observed a
higher degree of rationality (M = 4.11, SD = 0.32), whereas participants seemed to be less intuitive
(M = 3.05, SD = 0.79). Note that most participants scored high on the rationality dimension,
larger differences were seen for the intuitiveness dimensions. To assign participants to either
the intuitiveness or rationality group, we applied k-means clustering. Thus, 200 participants
could be assigned to the rationality group, while 296 participants were more likely to belong
to the intuitiveness group as shown in Figure 1. To validate the clustering quality, besides
visualization we also calculate the Silhouette score (0.40) and Calinski-Harabasz score (388.46)
which indicates the clustering of the rationality group and the intuitiveness group are OK. Based
on these results, we further analyzed the data for differences in meta-intents between the two
groups and for potential relations with interactive behavior.
Figure 2: Scores obtained for the different meta-intents. Whiskers indicates the 95% confidence interval.
Meta-Intents Factors We first present descriptive statistics for meta-intents scores for the
two groups in Figure 2 based on our self-constructed meta-intents items. In Figure 2, there
were no overlaps between the whiskers of the rationality and intuitiveness group bar in three
meta-intents factors: efficiency orientation, interest in detail and comparison orientation, which
suggests there are significant differences between the two groups on these factors (at 0.05 level).
While in Openness for guidance factor there is a small amount of overlap, in order to further
rigorously verify its significance, we conducted a t-test.
Table 2 shows the t-test results which indicated that besides the aforementioned three factors
in the last paragraph, another significant difference was observed in the openness for guidance
factor between the rationality and the intuitiveness group (𝑝 = .017). In addition, we point
out that since there are 8 comparisons in our experiment, there is a possibility of statistical
coincidence, so we apply a Benjamini-Hochberg (BH) Procedure, under which the (𝑝 = .034) of
this openness for guidance factor still meet the significance, as shown in the Table 2. We noticed
these four factors are directly or indirectly related to the duration of the interaction behavior.
For example, efficiency orientation means users tend to spend less time, and interest in detail
Table 2
Results from the independent samples t-test with Benjamini-Hochberg (BH) Procedure for meta-intents
factors between the rational group and intuitive group. Values marked with * are significant at a level
of 𝑝 < .05. df is equal to 496 means equal variances are assumed, otherwise not.
Rational Intuitive
n M SD n M SD df T p BH adjusted p d
Efficiency orientation 200 2.410 1.023 298 3.057 1.032 496 -6.884 <.001* <.001* -0.629
Diversity orientation 200 3.755 0.900 298 3.674 0.840 496 1.005 .315 .420 0.093
Openness for guidance 200 3.275 0.987 298 3.480 0.846 381.344 -2.402 .017* .034* -0.226
Interest in detail 200 4.160 0.683 298 3.782 0.797 467.419 5.489 <.001* <.001* 0.502
Brand awareness 200 2.88 1.015 298 2.940 0.944 496 -0.670 .503 .575 -0.061
Comparison orientation 200 4.145 0.811 298 3.910 0.704 385.442 3.396 .001* .003* -0.319
Scope of choice 200 3.350 0.976 298 3.310 0.894 496 0.487 .627 .627 0.044
Goal focus 200 2.858 0.479 298 2.928 0.458 496 -1.650 .100 .160 -0.151
Figure 3: Population’s distribution (duration feature, N=16543). The X-axis indicates the duration
features value (the unit is seconds), Y-axis indicates the counts.
would lead to spending more time, therefore we are motivated to investigate whether rationality
and intuitiveness group people spend a significantly different time on the CPA process. To be
concrete, the duration feature is the total time spent that the user interacts with CPA. We set
1000 seconds as a threshold to filter out outliers leaving us with 420 participants (A total of
496). This filtering left us with a still sufficiently large number of samples, and the shape of
the distribution is also well preserved, as illustrated in Figure 3. We applied a Mann-Whitney
U test (non-parametric test) since the distribution of the population does not follow a normal
distribution. Results showed in Table 3 indicate that participants within the rationality group
spent significantly more time in the advisor than participants within the intuitiveness group.
Based on these results, subsequently we address on the correlations between meta-intents,
decision-making style, and interactive behaviors.
Table 3
The results from Mann–Whitney U test (U test). Duration feature is time spent in the advisor. Mean rank
indicates the rank of 2 groups (higher duration scores get higher rank number). Mann-Whitney U is the
test statistic of U test. Values marked with * are significant at a level of 𝑝 < .05.
n M SD Mean Rank Mann-Whitney-U Z p
Rational 163 356.98 209.44 227.85
18118 −2.332 .020*
Intuitive 257 316.68 209.63 199.50
Correlations between Decision style and Meta-Intents Since our questionnaire data are
not continuous, but ordinal, we calculated the Spearman’s rank correlation coefficient between
decision-making style and meta-intents, and show the results in Table 4. We observed the
rationality factor highly correlates with diversity orientation (.314), interest in details (.582)
and comparison orientation (.524) with a positive value, and has a negative correlation with
efficiency orientation (-.284), goal focus (-.156), and the intuitiveness factor (-.192). In contrast,
the intuitiveness factor is negatively correlated with interest in details (-.222), comparison
orientation (-.145) and the rationality factor (-.192), and has a positive correlation with efficiency
orientation (.351), openness for guidance (.153), goal focus (.108). The factors rationality and
intuitiveness are highly correlated with some of the meta-intents, indicating that decision-
making style as a more stable personality factor can provide insights into PCA users’ high-level
intentions. Given the observation that the degrees of rationality and intuitiveness of a person
have an effect on meta-intents, we further analyzed whether decision style and meta-intents also
manifest themselves in certain features of the interaction with the CPA, that decision. To study
potential effects, we defined a set of interaction features that were derived from interaction logs
and performed further correlation analyses.
Table 4
Spearman’s rank correlation coefficient of decision-making style and meta-intents factors. Bold font
indicates the relatively high correlation which is greater than 0.3. **Correlation is significant at the .01
level (2-tailed). *Correlation is significant at the .05 level (2-tailed).
Meta-Intents Factors Decision Style
Efficiency Diversity Openness Interest Brand Comparison Scope Goal
Rational Intuitive
Interactive Features orientation orientation for guidance in detail awareness orientation of choice focus
Efficiency orientation –
Diversity orientation −.130** –
Openness for guidance .158** .191** –
Interest in detail −.290** .290** .105* –
Brand awareness .082 .061 .202** .067 –
Comparison orientation −.222** .026** .091* .485** .041 –
Scope of choice .060 .188** .224** .143** .117** .200** –
Goal focus .156** −.119** −.037 −.105* .013 −.133** −.011 –
** ** ** **
Rational −.284 .314 .084 .582 −.013 .524 .085 −.156** –
Intuitive .351** −.031 .153** −.222** .055 −.145** .018 .108* −.192** –
Correlations between Interaction Features, Decision style and Meta-Intents To identify
correlations between interactive features, decision styles and meta-intents, we also performed a
Spearman’s Rank correlation test, and the results are shown in Table 5. Here we identified only
weak correlations, thus leading to the assumption that there is no strong linear relationship
between interactive behaviors and decision-making style and meta-intents at an individual
level. The rationality factor has a significant (𝑝 < .001) but small correlation with overall dialog
duration (𝜌 = .132) and webpage-focus-deactivation (𝜌 = .145), the latter feature is the number of
times users switched focus to another webpage. A significant negative correlation between the
intuitiveness factor and duration is observed (𝜌 = −.130). We noticed the largest difference in
correlation values for the feature duration with the factors rationality and intuitiveness. This
result gave us the motivation to investigate further the possibility of decision style classification
based on interaction features in such short dialogs.
Table 5
Spearman’s rank correlation coefficient of decision-making style, meta-intents factors and interactive
features. **Correlation is significant at the .01 level (2-tailed). *Correlation is significant at the .05 level
(2-tailed). Bold font indicates the largest difference (correlation value) between rational and intuitive
factors.
Meta-Intents Factors Decision Style
Efficiency Diversity Openness Interest Brand Comparison Scope Goal
Rational Intuitive
Feature Groups Interactive Features orientation orientation for guidance in detail awareness orientation of choice focus
duration −.139** .035 −.015 .128** .026 .079 −.032 −.033 .132** −.130**
2nd-Q-duration −.008 .053 −.005 .039 .004 .038 .063 .063 .069 −.016
3rd-Q-duration .048 −.066 .037 −.033 .010 −.041 .030 .019 −.037 .041
4th-Q-duration −.033 .088 .051 .026 .026 .013 .062 .028 .084 −.021
5th-Q-duration .042 .040 .007 .086 −.020 .066 .032 .036 −.003 −.016
Durations
6th-Q-duration .047 .030 .005 .064 .042 −.010 .048 −.009 .006 .024
user-related-duration .027 .008 .000 .053 .012 .056 .061 .067 .012 −.015
feature-centric-duration .022 .016 .003 .043 .035 −.018 .052 .021 −.001 −.002
context-related-duration .047 −.007 .070 −.034 −.007 −.076 .078 .018 −.059 .021
purpose-related-duration −.055 −.047 −.086 .055 −.052 .073 −.053 −.012 −.016 −.005
Advisor actions advisor-interactions −.058 −.014 −.083 .057 −.029 .089* −.011 −.011 .069 .066
Product actions clickouts −.112 .043 .000 .073 .035 .053 .029 −.045 .107* −.068
events −.055 −.017 −.073 −.097* −.002 .083 −.026 −.003 .102* −.078
Overall
webpage-focus-deactivation −.104* .031 −.079 .114 .020 .058 −.031 −.024 .145** −.056
Prediction of Decision-making style To train a classification model, group sizes were
first aligned to obtain a balanced dataset. Therefore, 200 of the 296 participants previously
assigned to the intuitive group were randomly selected, to avoid sampling errors, this procedure
was repeated 10 times for both sampling and testing. Thereby, the average accuracy was
calculated. To filter outliers, we checked all 14 interactive feature histograms and defined
the threshold to keep a good shape of data distribution (keep most samples, remove outliers),
e.g. as aforementioned duration smaller than 1000 seconds. Finally, we obtained a balanced
dataset with 226 samples (75 % training set, 25 % testing set), and fitted them in three established
classification models: support vector machine (SVM), multi-layer perceptron (MLP) and decision
tree (DT). The results presented in Table 6 show that the performance of all classification models
is very weak (𝑆𝑉 𝐶 = 0.46, 𝑀 𝐿𝑃 = 0.46, 𝐷𝑇 = 0.53).
Table 6
The total of 3 classification models with parameters and their performance. Bold font indicate the test
accuracy greater than 0.5. Value marked with * indicates the best test accuracy.
Classification model Parameters Training accuracy Test accuracy
SVC C=1, gamma=2, kernel=’rbf’ 1.00 0.46
MLPClassifier alpha=0.1, max_iter=1000 1.00 0.46
DecisionTreeClassifier max_depth=5 0.83 0.53*
6. Discussion
Our results provide further evidence that indeed people differ in their general decision-making
style as substantial previous research has shown. Most users in our study score themselves high
on the rationality dimension while their scores vary more with respect to intuitiveness. We
further find correlations between the decision-making style dimensions and the factors that we
call meta-intents. More rational users seem to be more concerned with details of recommended
products (and possibly more detailed questions in the CPA) while they seem less interested
in efficient and goal-focused dialogs. They also appreciate diverse sets of products and the
possibility to compare items. For the intuitiveness group, efficiency is more important and they
seem to be more open for guidance (which is one of the strong features of a CPA). Those MI
factors that are significantly correlated with rationality factors did not show a strong correlation
with intuitiveness factors (no significant correlations observed). These findings provide some
insights for the design of CPA conversations with respect to dialog structure, question design,
and the presentation of recommendations. If data on the user’s decision style were available,
the findings can also provide a basis for personalizing the CPA.
However, deriving users’ decision style only from their interaction in a CPA seems difficult.
We could only find small, yet significant and reverse correlations of rationality and intuitiveness
with the overall time taken in the conversation (where rationality group takes longer). The
attempt to classify users into rationality or intuitiveness group and to predict meta-intents from
interaction features did not yield positive results. While this does not sound promising for the
goal of developing user-adaptive conversations, we have to point out a number of limitations of
this study. First, we only studied a single CPA in a single domain, which also had a relatively
simple dialog structure. Furthermore, due to the specific use case where the CPA is operated
as an external service, no data on users’ interaction in the online shop itself were available,
considerably limiting the number of signals by which user characteristics could be derived. Also,
a single user mostly used the CPA only once which also prevents us from collecting a more
comprehensive interaction history. As another limitation of the study, we only have subjects
who participated voluntarily, it is therefore not clear if these subjects are representative of the
entire population (or if this is a subset of generally more engaged visitors of the shop).
Considering these limitations, the present study can only be considered a starting point for
broader investigations with different CPA, and in a wider range of domains. Yet, we believe
our results provide some initial valuable insights that can help be better designed and possibly
personalize CPA.
7. Conclusion
The research presented in this paper sheds some light on psychological characteristics of the
users of CPA, a class of GUI-based conversational recommender systems frequently found in
e-commerce sites. We propose the concept of meta-intents which are high-level user preferences
with respect to the means that a CRS provides to support their decision process. In our study we
collected questionnaire responses related to the general decision-making dimensions of rational-
ity and intuitiveness and the proposed meta-intents as well as interaction data from 496 users of
a real CPA integrated in an online store. Our results provide evidence that these meta-intents are
linked to the general decision-making style of a user and can thus be instrumental in translating
decision styles into more concrete design guidance for CPA. While interesting correlations
between these psychological factors were found, they were not significantly correlated with
most interaction data, except for the overall duration of a conversation. Classifying users as
more rational or more intuitive decision makers was also not possible on the basis of the very
limited interaction data available. Clearly, several limitations of the study which was performed
with a single CPA need to be taken into account. Yet, our results provide initial evidence that it
appears worth exploring the influence of the proposed meta-intents with different types of CPA
and CRS in general, and in different domains and decision contexts.
References
[1] Y. Sun, Y. Zhang, Conversational recommender system, in: The 41st International
ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18,
Association for Computing Machinery, New York, NY, USA, 2018, p. 235–244. doi:10.
1145/3209978.3210002.
[2] Z. Fu, Y. Xian, Y. Zhang, Y. Zhang, WSDM 2021 Tutorial on conversational recommendation
systems, WSDM ’21, Association for Computing Machinery, New York, NY, USA, 2021, p.
1134–1136. doi:10.1145/3437963.3441661.
[3] C. Gao, W. Lei, X. He, M. de Rijke, T.-S. Chua, Advances and challenges in conversational
recommender systems: A survey, AI Open 2 (2021) 100–126. doi:https://doi.org/10.
1016/j.aiopen.2021.06.002.
[4] D. Jannach, A. Manzoor, W. Cai, L. Chen, A survey on conversational recommender
systems, ACM Comput. Surv. 54 (2021). doi:10.1145/3453154.
[5] B. Xiao, I. Benbasat, E-commerce product recommendation agents: Use, characteristics,
and impact, MIS Q. 31 (2007) 137–209.
[6] S. D. Gosling, P. J. Rentfrow, W. B. Swann, A very brief measure of the big-five per-
sonality domains, Journal of Research in Personality 37 (2003) 504–528. doi:10.1016/
S0092-6566(03)00046-1.
[7] K. Hamilton, S.-I. Shih, S. Mohammed, The development and validation of the rational
and intuitive decision styles scale, Journal of Personality Assessment 98 (2016) 523–535.
doi:10.1080/00223891.2015.1132426.
[8] B. Lika, K. Kolomvatsos, S. Hadjiefthymiades, Facing the cold start problem in recommender
systems, Expert Systems with Applications 41 (2014) 2065–2073. doi:https://doi.org/
10.1016/j.eswa.2013.09.005.
[9] Y. Zhang, X. Chen, Q. Ai, L. Yang, W. B. Croft, Towards conversational search and
recommendation: System ask, user respond, in: Proceedings of the 27th ACM International
Conference on Information and Knowledge Management, CIKM ’18, Association for
Computing Machinery, New York, NY, USA, 2018, p. 177–186. doi:10.1145/3269206.
3271776.
[10] Y. Ma, T. Kleemann, J. Ziegler, Mixed-modality interaction in conversational recommender
systems, in: Interfaces and Human Decision Making for Recommender Systems 2021:
Proceedings of the 8th Joint Workshop on Interfaces and Human Decision Making for
Recommender Systems, 2021, pp. 21–37.
[11] I. Kostric, K. Balog, F. Radlinski, Soliciting user preferences in conversational recommender
systems via usage-related questions, in: 15th ACM Conference on Recommender Systems,
RecSys ’21, 2021, pp. 724–729.
[12] D. Jannach, ADVISOR SUITE - A knowledge-based sales advisory system, in: Proceedings
of the 16th European Conference on Artificial Intelligence, ECAI ’04, IOS Press, NLD, 2004,
p. 720–724.
[13] A. Papenmeier, A. Frummet, D. Kern, “Mhm...” – conversational strategies for product
search assistants, in: ACM SIGIR Conference on Human Information Interaction and
Retrieval, CHIIR ’22, Association for Computing Machinery, New York, NY, USA, 2022, p.
36–46. doi:10.1145/3498366.3505809.
[14] T. Kleemann, M. Wagner, B. Loepp, J. Ziegler, Modeling user interaction at the convergence
of filtering mechanisms, recommender algorithms and advisory components, in: Mensch
Und Computer 2021, MuC ’21, Association for Computing Machinery, New York, NY, USA,
2021, p. 499–511. doi:10.1145/3473856.3473859.
[15] M. Atas, A. Felfernig, S. Polat-Erdeniz, A. Popescu, T. N. T. Tran, M. Uta, Towards
psychology-aware preference construction in recommender systems: Overview and re-
search issues, Journal of Intelligent Information Systems (2021) 1–23.
[16] E. Lex, D. Kowald, P. Seitlinger, T. N. T. Tran, A. Felfernig, M. Schedl, Psychology-informed
recommender systems, Foundations and Trends® in Information Retrieval 15 (2021)
134–242. doi:10.1561/1500000090.
[17] M. Tkalcic, L. Chen, Personality and Recommender Systems, Springer, Boston, MA, 2015,
pp. 715–739. doi:10.1007/978-1-4899-7637-6_21.
[18] T. Z. Gizaw, H. Dong Jun, A. Oad, Solving cold-start problem by combining personality
traits and demographic attributes in a user based recommender system, International
Journal of Advanced Research in Computer Science and Software Engineering 7 (2017).
[19] M. Millecamp, N. N. Htun, Y. Jin, K. Verbert, Controlling spotify recommendations:
Effects of personal characteristics on music recommender user interfaces, in: Proceedings
of the 26th Conference on User Modeling, Adaptation and Personalization, UMAP ’18,
Association for Computing Machinery, New York, NY, USA, 2018, p. 101–109. doi:10.
1145/3209219.3209223.
[20] J. Zhang, Anchoring effects of recommender systems, in: Proceedings of the 5th ACM
Conference on Recommender Systems, RecSys ’11, Association for Computing Machinery,
New York, NY, USA, 2011, p. 375–378. doi:10.1145/2043932.2044010.
[21] K. Hamilton, S.-I. Shih, S. Mohammed, The development and validation of the rational
and intuitive decision styles scale, Journal of Personality Assessment 98 (2016) 523–535.
doi:10.1080/00223891.2015.1132426.
[22] O. B. Büttner, A. Florack, A. S. Göritz, Shopping orientation as a stable consumer disposition
and its influence on consumers’ evaluations of retailer communication, European Journal
of Marketing (2014).
[23] O. B. Büttner, A. Florack, A. S. Göritz, How shopping orientation influences the effec-
tiveness of monetary and nonmonetary promotions, European Journal of Marketing
(2015).
[24] W. Cai, L. Chen, Predicting user intents and satisfaction with dialogue-based conversational
recommendations, in: Proceedings of the 28th ACM Conference on User Modeling,
Adaptation and Personalization, UMAP ’20, Association for Computing Machinery, New
York, NY, USA, 2020, p. 33–42. doi:10.1145/3340631.3394856.
[25] A. Felfernig, G. Friedrich, D. Jannach, M. Zanker, An integrated environment for the
development of knowledge-based recommender applications, International Journal of
Electronic Commerce 11 (2006) 11–34. doi:10.2753/JEC1086-4415110201.
[26] A. Jameson, M. C. Willemsen, A. Felfernig, M. d. Gemmis, P. Lops, G. Semeraro, L. Chen,
Human decision making and recommender systems, in: Recommender Systems Handbook,
Springer, Boston, MA, 2015, pp. 611–648. doi:10.1007/978-1-4899-7637-6_18.
[27] J. Shen, O. Brdiczka, J. Liu, Understanding email writers: Personality prediction from
email messages, in: User Modeling, Adaptation, and Personalization, Springer, Berlin,
Heidelberg, 2013, pp. 318–330.
[28] J. Golbeck, C. Robles, K. Turner, Predicting personality with social media, in: Extended
Abstracts of the 2011 Conference on Human Factors in Computing Systems, CHI EA ’11,
Association for Computing Machinery, New York, NY, USA, 2011, p. 253–262. doi:10.
1145/1979742.1979614.
[29] C. Stachl, Q. Au, R. Schoedel, S. D. Gosling, G. M. Harari, D. Buschek, S. T. Völkel, T. Schuw-
erk, M. Oldemeier, T. Ullmann, et al., Predicting personality from patterns of behavior
collected with smartphones, Proceedings of the National Academy of Sciences 117 (2020)
17680–17687.