Conversational recommender systems (CRS [1]) have been gaining increased attention in research and industry in recent years [2,3]. Generally, conversational techniques can provide users with strong guidance to achieve their goals combined with a high level of flexibility in expressing their needs. Jannach et al. [4] distinguish between natural language-based, formbased, and critiquing approaches. Due to the advances in NLP techniques in recent years, natural language-based CRS have become the subject of extensive research. However, despite various advantages, such as human-like dialog and a high level of flexibility, their capabilities are still limited and users may need to adapt to the underlying (invisible) vocabulary and domain coverage to interact successfully.
Despite the recent surge in NLP-based CRS, GUI-based forms of conversation, leading users IntRS'22: Joint Workshop on Interfaces and Human Decision Making for Recommender Systems, September 22, 2022, Seattle, US (hybrid event). yuan.ma@uni-due.de (Y. Ma); timm.kleemann@uni-due.de (T. Kleemann); juergen.ziegler@uni-due.de (J. Ziegler) through a pre-defined branching structure of form-based questions and response options, constitute an alternative that has a number of advantages from a user perspective. They provide a high level of transparency and user guidance, minimize errors, and incorporate expert domain knowledge relevant to decision-making. GUI-based conversational product advisors (CPA), have thus become frequently used tools in e-commerce websites. Designing CPA to support users in finding a suitable item efficiently and with a positive user experience, however, involves a number of critical challenges. Questions need to be formulated at an appropriate level of abstraction, for example, asking either about the intended use of the product or about some specific technical features. Dialog flow should follow the user's likely mental decision process, providing sufficient flexibility without becoming overly complex, and recommendations should be presented in appropriate numbers and with an appropriate level of detail.
To address these design challenges for CPA, a thorough understanding of user needs and their decision-making style is needed. Little research, however, has investigated the influence of psychological user characteristics and intents in the context of CPA thus far [5]. In this paper, we explore psychological characteristics of CPA users under two different objectives. First, we aim at obtaining a deeper understanding of psychological characteristics of CPA users, based on responses from questionnaire instruments. Here, we distinguish between stable individual traits including personality factors [6] and decision-making style [7], and, second, task-oriented characteristics that represent general user preferences when interacting with a CPA, such as obtaining detailed information about items or comparing products. We call the latter characteristics meta-intents since they describe user goals that are more general and high-level than the search goals typically extracted through intent detection methods in CRS.
As a second objective, we investigate whether psychological user characteristics can be predicted from their behavior in a CPA. Meta-intents such as interest in detail might influence, for example, the time spent in different steps of an advisor dialog. If such user characteristics could be derived from their interaction, this information would provide insights for optimally designing the dialogs, or for personalizing them.
In this paper, we describe a first study analyzing these questions and present the results of an online study conducted with users of a real CPA in productive, commercial use. Our contribution is twofold: We provide insights about CPA users' decision-making style (rationality vs. intuitiveness) and the relevance of different meta-intents we propose for CRS. Furthermore, we discuss the possibility of predicting psychological characteristics from users' interaction behavior in CPA in the light of our case study data.
Conversational Recommender Systems (CRS) have become a rapidly growing and popular research area because they provide a flexible, human-like multi-turn dialog for preference elicitation, which is essential for generating personalized recommendations [8]. Jannach et al. [4] distinguish three types of CRS differing in the style and structure of the interaction used: natural language-based, form-based, and critiquing-based. NLP-based CRS have received considerable interest recently due to the advancements in natural language processing. They typically use a question-answer format [9], sometimes mixing language and GUI elements in mixed-mode systems [10]. Form-based CRS present questions and answers in a GUI style, leading users through a predefined dialog structure. This type of CRS has many advantages as they provide guidance to the users, avoid errors, and can incorporate domain knowledge. Especially usage-related questions, asking users about the tasks they want to accomplish with a product, are important for users who have only limited knowledge about technical item properties [11]. CPA are thus often used as product advisors in e-commerce sites. An early example of such knowledge-based CPA is ADVISOR Suite [12]. Despite their relevance, very limited research has as yet studied users' interaction needs in CPA and related design questions. For example, Papenmeier et al. [13] investigated human advisory dialogs, identifying some recurring strategies such as funneling to successively narrow down the space of potential items. Kleemann et al. [14] investigated user behavior when using a CPA in combination with other decision aids.
To provide design guidance for CPA and to potentially adapt them to the individual user, a deeper understanding of the psychological factors influencing users' decision-making and interaction behavior in CRS and CPA is required. For recommender systems in general, the influence of psychological characteristics on users' preference construction and decision making has been shown repeatedly [15]. Lex et al. [16] distinguish between factors related to cognition, personality, and emotion. The influence of psychological characteristics such as the Big Five personality factors (e.g. [17,18]), Need for Cognition [19], or cognitive biases [20] has been studied in several works. However, these studies mostly aim at better understanding user preferences with respect to the recommended items and at improving their accuracy. In contrast, the relationship between psychological factors and the design of advisory dialogs in CPA remains an underexplored area. Especially theories related to human decision-making styles appear to be promising points of departure for studying this relation. The distinction between rational and intuitive decision-making styles [21] or cognitive styles such as the need for cognition may influence users' assessment of CPA dialogs. More domain-specific theories such as Shopping Orientation [22,23], distinguishing between task-focused and experiential shopping are also of interest. However, none of these approaches has yet been applied to CPA.
User goals and preferences when interacting with a CRS may be located on different levels of abstraction. Low-level preferences refer to concrete properties of the desired item (often called Intents in CRS, specifically Add Details [24]). On a more abstract level, meta-level preferences [25]. [26] may represent high-level product dimensions (such as economy and safety). Yet, they also refer to product-related aspects. Meta-level preferences that relate to the conversation style and type of questions in a CPA have, to our knowledge, not been studied yet.
Inferring psychological user characteristics and needs from different sources, such as email [27] social media posts [28], or smartphone use [29] has been subject of a considerable body of work, giving indications for the feasibility of such methods. However, it is an open question whether psychological characteristics can also be derived from users' interaction behavior in CPA where only a very limited number of signals can be extracted from the GUI dialog to be used as predictors. Investigating this research question is one of the goals of this paper.
To investigate differences in CPA users' psychological properties, we hypothesized that decisionmaking style might influence users' usage and interaction in CPA. Accordingly, we applied instruments to measure these properties, using the Decision Styles Scale (DSS) [21] for distinguishing rational and intuitive decision-making styles.
While general decision-making styles, e.g. rationality and intuitiveness, apply to arbitrary decision contexts, we also aimed at capturing users' preferences at a more specific, yet still abstract level. These meta-intents should bridge the gap between item-level intents and general decision-making style, and should also relate to the design and question-asking style in CRS and, specifically, CPA. They might also be relevant for more general recommendation scenarios. We postulated the following set of meta-intents (with sample questionnaire items in parentheses), partly related to general usage factors such as efficiency, effectiveness, and user guidance. We see this list as a first step towards defining factors relevant for users' decision-making process in CPA which is neither complete nor final.
• Efficiency orientation (For me, finding a suitable product quickly is more important than exploring all options.)
• Diversity orientation (When shopping online, I tend to explore a diverse range of products that might interest me.)
• Goal focus (I usually have a clear idea of what I want before visiting an online shop. I often only make up my mind once I see the available choices.)
• Openness for guidance (I appreciate it if a shop recommends products I might like.)
• Interest in detail (I usually gather as much information as possible about products that I want to buy.)
• Brand awareness (The brand of a product is an important factor for my decision.)
• Comparison orientation (Comparing the features of different candidate products is important for me.)
• Scope of choice (When the system recommends products, I rather like to see a longer list rather than a short one.)
To investigate CPA users' psychological characteristics, both at the level of decision-making style and CPA-specific meta-intents , as well as possible relations with their interaction behavior, we performed a long-term online experiment with users of a CPA embedded in an online shop. We hypothesized that there are relations between general traits (decision-making style) and individual preferences with respect to the meta-intents formulated. We also assumed that decision-making style and meta-intents would influence user interaction behavior, for example, the time spent on the dialog as a whole as well as for different categories of advisor questions.
We assessed the behavior of real users rather than relying on an artificial scenario in a laboratory experiment in order to obtain realistic data. For this purpose, we applied existing and self-developed questionnaires and linked them to a real-world CPA, designed for supporting customers when purchasing a new mattress. The domain was chosen both for reasons of opportunity (it is notoriously difficult for academics to get access to real e-commerce sites to perform experiments) and because mattress selection is a sufficiently complex decision process, involving a variety of decision criteria. Furthermore, the site offers a large variety of products, not limited to a single brand or product type and the embedded CPA also comprises a range of different question types.
Our study involves real users who enter and browse an online mattress shop and who freely choose to engage with a GUI-based CPA. In the CPA dialog, the user is first confronted with a series of six predetermined questions, after answering all questions (note that different question paths will appear according to different answers, but all the paths have the same depth of 6 questions), the recommended products will appear, at this point they can freely select, compare and view products. All visitors who had completed the advisory process were then invited to participate in our study with a pop-up on the results page. 1 We only asked users who had fully gone through the advisor if they wanted to participate in the study, firstly to ensure that we could conduct our analyses mainly with complete user interaction data. Second, we assumed that users might be more motivated to participate in a study and provide meaningful responses if they did not drop out of the advisory process early.
Once they clicked to participate in our study, a questionnaire opened in a separate browser tab, avoiding distracting them from interacting with the online store or the results of their advisory process. Thus, the participants could answer the questions at their discretion, either directly or at a later time. There was no compensation for participation. The questionnaire was presented using a self-hosted instance of the tool LimeSurvey. 2 All items had 5-point Likert response scales.
The questionnaire had two parts. First, we measured participants' decision style using Decision Styles Scale (DSS) [21]. Next, we assessed the participants' meta-intents , discussed in Section 3. Due to strict data protection requirements imposed by the online store and advisor provider, no demographic data was collected. Furthermore, participants were provided with the questionnaire in the language of the online store and the advisor (Dutch). Through an identification code that was automatically passed to the questionnaire, we were able to match the participants who took part in the study to their respective advisor's interaction data. Also, the interaction data of the advisor were completely anonymized.
The study ran over a period of five months. During this time, 18.914 visitors of the online store finished the CPA process. Of these, 3.782 visitors opened the questionnaire. Overall, a total of 506 visitors, subsequently referred as participants, responded to all questions. After cleaning the data by removing outliers (users with less then 10 advisor interactions) and participants with implausible response combinations (whose decision-making scale scores or meta-intents scores have all the same value), we were left with both questionnaire and advisor interaction data from 496 participants.
The advisor is composed out of 12 questions, which involves 4 question types (with a real CPA example in parentheses):
• Purpose-related. What is the intended use of this product? (What feeling do you want when you lie down? )
• Feature-related. Questions about product features. (What size do you want? )
• Context-related. What is the environment of using the product. (I'm sleeping . . . ? (prefer not to say; alone; with a partner))
• User-related. Questions about user preferences. (Select your weight for the perfect sleeping comfort)
Interactive features and descriptions. A total of 14 interaction features will be further used to train and test the classification model that attempts to predict users' decision-making styles. Total number of . . . Advisor actions advisor-interactions . . . events the user generates while interacting with the advisor.
Total number of . . . Product actions clickouts . . . clicks to view product details.
Total number of . . . Overall events . . . events the user interact with the online-shopping website. including active (i.e. selecting answers, switch between questions) and passive (i.e. webpage focus deactivation) events. webpage-focus-deactivation . . . deactivating this online-shopping website.
Note that different question paths will appear according to different answers, but all the paths have the same depth of 6 questions. This is due to the fixed design of the advisor which we did not alter. For each question step, besides choosing one answer, users have various interaction options: They may navigate back and forward, change their answers, or completely restart the advisory process. Once users have completed the CPA process, suitable products are displayed on a results page. Here, users have the option to browse through further results and to change the sort order of the results. For each product shown, a summary of the product features that match the given answers is displayed, by clicking, it opens a separate detailed product page.
We logged all interactions that occurred in the advisor and on the results page. This included data on the total time spent during the advisory process, as well as the time spans between each interaction. We also logged which products were clicked on by users in the results list (referred to as clickouts in Table 1). In addition, we recorded whether and for how long the browser window containing the advisor was inactive or merely opened in the background. All collected event types and the features subsequently generated from the data in a feature engineering process are summarized in Table 1. These features were used for classifying the decision-making styles of the participants. For this attempt, we used three popular machine learning models: support vector machine (SVM), multi-layer perceptron (MLP) and decision tree (DT). Limited by the number of datasets we did not use deep neural networks.
Figure 1: K -means clustering of rationality and intuitiveness factors. The X-axis is the intuitive score, Y-axis is the rational score. The blue-filled circle indicates the user is clustered to the rationality group (200 participants), red indicates the intuitiveness group (296 participants), and black crosses are centroids of groups. Due to data overlapping, we present each user with a filled circle with 10% transparency, light colors mean sparse, and dark colors mean dense.
With respect to decision making, we observed a higher degree of rationality (M = 4.11, SD = 0.32), whereas participants seemed to be less intuitive (M = 3.05, SD = 0.79). Note that most participants scored high on the rationality dimension, larger differences were seen for the intuitiveness dimensions. To assign participants to either the intuitiveness or rationality group, we applied k-means clustering. Thus, 200 participants could be assigned to the rationality group, while 296 participants were more likely to belong to the intuitiveness group as shown in Figure 1. To validate the clustering quality, besides visualization we also calculate the Silhouette score (0.40) and Calinski-Harabasz score (388.46) which indicates the clustering of the rationality group and the intuitiveness group are OK. Based on these results, we further analyzed the data for differences in meta-intents between the two groups and for potential relations with interactive behavior.
We first present descriptive statistics for meta-intents scores for the two groups in Figure 2 based on our self-constructed meta-intents items. In Figure 2, there were no overlaps between the whiskers of the rationality and intuitiveness group bar in three meta-intents factors: efficiency orientation, interest in detail and comparison orientation, which suggests there are significant differences between the two groups on these factors (at 0.05 level). While in Openness for guidance factor there is a small amount of overlap, in order to further rigorously verify its significance, we conducted a t-test.
Table 2 shows the t-test results which indicated that besides the aforementioned three factors in the last paragraph, another significant difference was observed in the openness for guidance factor between the rationality and the intuitiveness group (𝑝 = .017). In addition, we point out that since there are 8 comparisons in our experiment, there is a possibility of statistical coincidence, so we apply a Benjamini-Hochberg (BH) Procedure, under which the (𝑝 = .034) of this openness for guidance factor still meet the significance, as shown in the Table 2. We noticed these four factors are directly or indirectly related to the duration of the interaction behavior. For example, efficiency orientation means users tend to spend less time, and interest in detail would lead to spending more time, therefore we are motivated to investigate whether rationality and intuitiveness group people spend a significantly different time on the CPA process. To be concrete, the duration feature is the total time spent that the user interacts with CPA. We set 1000 seconds as a threshold to filter out outliers leaving us with 420 participants (A total of 496). This filtering left us with a still sufficiently large number of samples, and the shape of the distribution is also well preserved, as illustrated in Figure 3. We applied a Mann-Whitney U test (non-parametric test) since the distribution of the population does not follow a normal distribution. Results showed in Table 3 indicate that participants within the rationality group spent significantly more time in the advisor than participants within the intuitiveness group.
Based on these results, subsequently we address on the correlations between meta-intents, decision-making style, and interactive behaviors. Correlations between Decision style and Meta-Intents Since our questionnaire data are not continuous, but ordinal, we calculated the Spearman's rank correlation coefficient between decision-making style and meta-intents, and show the results in Table 4. We observed the rationality factor highly correlates with diversity orientation (.314), interest in details (.582) and comparison orientation (.524) with a positive value, and has a negative correlation with efficiency orientation (-.284), goal focus (-.156), and the intuitiveness factor (-.192). In contrast, the intuitiveness factor is negatively correlated with interest in details (-.222), comparison orientation (-.145) and the rationality factor (-.192), and has a positive correlation with efficiency orientation (.351), openness for guidance (.153), goal focus (.108). The factors rationality and intuitiveness are highly correlated with some of the meta-intents, indicating that decisionmaking style as a more stable personality factor can provide insights into PCA users' high-level intentions. Given the observation that the degrees of rationality and intuitiveness of a person have an effect on meta-intents, we further analyzed whether decision style and meta-intents also manifest themselves in certain features of the interaction with the CPA, that decision. To study potential effects, we defined a set of interaction features that were derived from interaction logs and performed further correlation analyses.
To identify correlations between interactive features, decision styles and meta-intents, we also performed a Spearman's Rank correlation test, and the results are shown in Table 5. Here we identified only weak correlations, thus leading to the assumption that there is no strong linear relationship between interactive behaviors and decision-making style and meta-intents at an individual level. The rationality factor has a significant (𝑝 < .001) but small correlation with overall dialog duration (𝜌 = .132) and webpage-focus-deactivation (𝜌 = .145), the latter feature is the number of times users switched focus to another webpage. A significant negative correlation between the intuitiveness factor and duration is observed (𝜌 = −.130). We noticed the largest difference in correlation values for the feature duration with the factors rationality and intuitiveness. This result gave us the motivation to investigate further the possibility of decision style classification based on interaction features in such short dialogs.
To train a classification model, group sizes were first aligned to obtain a balanced dataset. Therefore, 200 of the 296 participants previously assigned to the intuitive group were randomly selected, to avoid sampling errors, this procedure was repeated 10 times for both sampling and testing. Thereby, the average accuracy was calculated. To filter outliers, we checked all 14 interactive feature histograms and defined the threshold to keep a good shape of data distribution (keep most samples, remove outliers), e.g. as aforementioned duration smaller than 1000 seconds. Finally, we obtained a balanced dataset with 226 samples (75 % training set, 25 % testing set), and fitted them in three established classification models: support vector machine (SVM), multi-layer perceptron (MLP) and decision tree (DT). The results presented in Table 6 show that the performance of all classification models is very weak (𝑆𝑉 𝐶 = 0.46, 𝑀 𝐿𝑃 = 0.46, 𝐷𝑇 = 0.53).
Our results provide further evidence that indeed people differ in their general decision-making style as substantial previous research has shown. Most users in our study score themselves high on the rationality dimension while their scores vary more with respect to intuitiveness. We further find correlations between the decision-making style dimensions and the factors that we call meta-intents. More rational users seem to be more concerned with details of recommended products (and possibly more detailed questions in the CPA) while they seem less interested in efficient and goal-focused dialogs. They also appreciate diverse sets of products and the possibility to compare items. For the intuitiveness group, efficiency is more important and they seem to be more open for guidance (which is one of the strong features of a CPA). Those MI factors that are significantly correlated with rationality factors did not show a strong correlation with intuitiveness factors (no significant correlations observed). These findings provide some insights for the design of CPA conversations with respect to dialog structure, question design, and the presentation of recommendations. If data on the user's decision style were available, the findings can also provide a basis for personalizing the CPA. However, deriving users' decision style only from their interaction in a CPA seems difficult. We could only find small, yet significant and reverse correlations of rationality and intuitiveness with the overall time taken in the conversation (where rationality group takes longer). The attempt to classify users into rationality or intuitiveness group and to predict meta-intents from interaction features did not yield positive results. While this does not sound promising for the goal of developing user-adaptive conversations, we have to point out a number of limitations of this study. First, we only studied a single CPA in a single domain, which also had a relatively simple dialog structure. Furthermore, due to the specific use case where the CPA is operated as an external service, no data on users' interaction in the online shop itself were available, considerably limiting the number of signals by which user characteristics could be derived. Also, a single user mostly used the CPA only once which also prevents us from collecting a more comprehensive interaction history. As another limitation of the study, we only have subjects who participated voluntarily, it is therefore not clear if these subjects are representative of the entire population (or if this is a subset of generally more engaged visitors of the shop).
Considering these limitations, the present study can only be considered a starting point for broader investigations with different CPA, and in a wider range of domains. Yet, we believe our results provide some initial valuable insights that can help be better designed and possibly personalize CPA.
The research presented in this paper sheds some light on psychological characteristics of the users of CPA, a class of GUI-based conversational recommender systems frequently found in e-commerce sites. We propose the concept of meta-intents which are high-level user preferences with respect to the means that a CRS provides to support their decision process. In our study we collected questionnaire responses related to the general decision-making dimensions of rationality and intuitiveness and the proposed meta-intents as well as interaction data from 496 users of a real CPA integrated in an online store. Our results provide evidence that these meta-intents are linked to the general decision-making style of a user and can thus be instrumental in translating decision styles into more concrete design guidance for CPA. While interesting correlations between these psychological factors were found, they were not significantly correlated with most interaction data, except for the overall duration of a conversation. Classifying users as more rational or more intuitive decision makers was also not possible on the basis of the very limited interaction data available. Clearly, several limitations of the study which was performed with a single CPA need to be taken into account. Yet, our results provide initial evidence that it appears worth exploring the influence of the proposed meta-intents with different types of CPA and CRS in general, and in different domains and decision contexts.