In the field of recommender systems (RS), researchers are increasingly aware that optimizing accuracy is not enough to reach the full potential of recommender systems (RS) [1,2]. For example, users will not choose a recommended item unless they have trust in the system [3]. One possible way to increase this trust is providing explanations which reveal (a part of) the internal reasoning of the HUMANIZE: Joint Proceedings of the ACM IUI 2021 Workshops, April 13-17, 2021, College Station, USA martijn.millecamp@kuleuven.be (M. Millecamp); conati@cs.ubc.ca (C. Conati); katrien.verbert@kuleuven.be (K. Verbert) 0000-0002-5542-0067 (M. Millecamp); 0000-0002-8434-9335 (C. Conati); 0000-0001-6699-7710 (K. Verbert) RS to the user [4,5]. Especially the combination of these explanations with control can help users not only to understand the RS, but also to steer the RS with input and feedback [5]. Despite the increased interest in explanations for RS, it is still not clear how to implement explanations in practice as users have varying reactions to them which shows the need to personalize explanation to the user [6].
However, before the system could adapt explanations to personal characteristics (PCs), it needs to be aware of the PCs of the user. A possible way to obtain these characteristics is by explicitly asking the users to fill in questionnaires [7] or by implicitly inferring PCs through an analysis of the social media of the user [8]. Nonetheless, asking users to fill in questionnaires or to give access to their social media is often not desirable. Moreover, to personalize explanations it is not necessary to obtain a fine-grained result, but a classification into two categories suffices [9].
For this reason, we explore in this paper whether it is possible to classify users' personality traits during the interaction with a music RS with explanations by analyzing their gaze. We will focus on three different PCs: openness, need for cognition (NFC) and musical sophistication (MS) [9,10]. These PCs will be explained in detail in Section 2.
Openness is one of the Big Five personality traits which measures how open a person is to new experiences. Millecamp et al. [9] showed that there was a significant difference in the gaze pattern between low and high openness users. This is the reason we hypothesize that classifying openness based on gaze might be possible.
Similarly, we hypothesize that inferring MS, which is a measure of domain knowledge in the music domain, from gaze data might be possible as the study of Millecamp et al. [9] also found significant differences in gaze pattern between low and high MS.
NFC is a cognitive style which influences the way a person prefers to process information and thus looks at information. Previous studies already showed that NFC moderates the perception of explanations in a music recommender system, which was the motivation to explore whether inferring NFC from gaze would be possible.
Next to exploring the general accuracy, we also want to explore how much data we need to infer these PCs.
The contribution of this paper is twofold. First, to our knowledge, we are the first to explore whether it is possible to infer PCs during the interaction with a RS in the presence of explanations. Second, we make the gathered dataset publicly available to support the research in this area. This dataset is unique because it provides both gaze data and data about PCs.
With the increasing role of RS in our daily lives, the call for explainable, transparent RS also becomes louder so that users can make better informed decisions whether or not to follow the recommendations [11,6]. In combination with controls, this transparency also enables users to correct the RS whenever they feel it makes wrong assumptions [5]. However, research has shown that different users have different reactions to explanations [6,12,13]. In the field of music RS, recent research has shown that there are three PCs that could influence the way users perceive explanations: openness, NFC and MS [10,9].
Openness is one of the five factors of the Five Factor Model, also known as the Big 5 model [14]. This model describes personality in five different traits and it has been used in several studies which showed the positive impact of considering personality in RS [15]. The factor openness describes the breadth, depth and complexity of an individualś mental and experiental life [16]. It has been shown that openness is related to the preferred amount of diversity in RS and to the willingness to use a system with explanations [17,18,9].
Need for cognition has been shown to influence the success of a RS [13,12,19,20] and is defined as "a measure of the tendency for an individual to engage in, and enjoy, effortful cognitive activities" [21]. NFC has been shown to have an impact on the willingness of users to rely on a RS [12], on the confidence in a playlist created in a music RS with explanations [10], on preference matching [22], on the style of explanations they prefer [13] and on the reason why users need a transparent RS [23].
Musical sophistication is defined by Mullensiefen et al. [24] as a concept to describe the multi-faceted nature of musical expertise. In the music domain, Millecamp et al. [9] showed that users with high MS feel more supported to make a decision in a RS interface that provided explanations than an interface without such explanations, while this made no difference for users with low MS. Another study showed that users with high domain experience perceive a higher diversity in a scatter plot than in a simpler bubble chart [25].
To acquire the PCs of users, the most common way is to ask users to fill in validated questionnaire [7], but there exist also other approaches such as inferring PCs by analyzing the social media of the user [26,8], by analyzing a conversation with a chatbot [27] or by analyzing the physical signals such as brain activity [28] and gaze data [7].
The previously mentioned works rely on fine-grained personality scores. In contrast, in our work we focus on adapting interfaces to users for which we only need a classification in two groups. We aim to base this classification on the gaze pattern during the interaction with a music RS interface instead of asking users to watch carefully selected stimuli, to fill in questionnaires or to share their social media profile. Previous studies which classified users based on their gaze pattern during normal activities are almost all only focused on cognitive abilities and visualization experience [29,30,31,32]. One exception is the study of Hoppe et al. [33] which inferred the Big Five personality traits by studying the gaze of a walk through a campus. This study is different from our work, as we investigate if it is possible to infer PCs while interacting with a music RS and also focus on different PCs.
The gaze data that is used in this study was generated in a user study by Millecamp et al. [9]. We will provide a brief summary of this experiment, but a more elaborate description can be found in [9]. As mentioned in Section 2, we focus in this study on openness, NFC and MS as previous research has found that these PCs could affect the perception of explanations in a music RS [9,10] and the study of Millecamp et al. [9] already showed that openness and MS change the gaze pattern between an interface in the presence or absence of explanations. To measure these three characteristics, users were asked to fill out three questionnaires before the experiment started. To measure openness, we used the 44-item Big Five Inventory [34] and selected afterwards the questions related to openness. For NFC, we used the 18-items questionnaire of Cacioppo et al. [21] and for MS the Goldsmiths Musical Sophistication Index1 was used. The dataset we used in this study consists of the gaze data of 30 participants (21 male). For the three PCs, the participants were divided into a high and low group based on a median split. This resulted in equally distributed groups for MS and NFC and almost equally groups for openness (16 in the low and 14 in the high openness group). An short overview of the characteristics of the participants can be found in Table 1.
The gaze data was recorded with a Tobii 4C remote eye tracker at a sampling rate of 90Hz. Each sample contained information about the focus point on the screen denoted as an x and y coordinate, the distance between the participant and the screen, and the validity of these measures. To calibrate the eye tracker, the experiment started with a standard calibration procedure provided by Tobii Core Software. After the calibration, users were asked to explore the interface of a music RS in the presence of feature-based explanations until they understood all functionalities. A screenshot of the interface is shown in Figure 1.
As shown in Part A of this figure, users first can search for an artist they like through a search bar in the top left corner. When they add the artist, this artist is shown in Part B. Based on this artist, the system starts to generate recommendations which were listed in a two-column format as shown in Part F. When users hover over the cover of the picture of a recommended song, they can click a play button to listen to a 30s preview of the song. On the right side of each explanation, they can click on the thumb-up icon to add the song to their playlist. Through the sliders shown in Part D of Figure 1, users can modify several audio features 2 such as popularity, energy and danceability which are also taken into account in the recommendation process. To help users steer these sliders, the minimum and the maximum for each audio feature is shown for each artist.
After the user explored all the options of the interface, the recording of the gaze started. As shown in Part E of Figure 1, users were asked to create a playlist of five songs. To create this playlist, they could use all functionalities without any restriction. When they added the fifth song to their playlist, we stopped the recording of the gaze. On average, users took 4 minutes 26 seconds to complete their playlist. As part of this paper's contribution, this data is publicly available 3 .
2 https://developer.spotify.com/documentation/ web-api/reference/tracks/get-audio-features/ 3 augment.cs.kuleuven.be/datasets/classifeye
The Tobii 4C does not come with software to detect fixations and saccades so we identified fixations and saccades using an implementation of the ID-T algorithm [35] with a dispersion threshold of one degree and a duration threshold of 100ms [35]. This means that in this study a fixation is identified as a circle on the screen in which the user keeps focusing for at least 100ms without moving their eyes more than one degree. All other movements are then identified as saccades, i.e. quick movements of gaze from one fixation to another [30].
Based on these saccades and fixations, we generated a set of eye-tracking features as listed in Table 2. Most of these features are selected because they are widely used in previous eye tracking studies [7,30,36]. In addition to these features, we included Most frequent saccade direction and fixations in a 4x4 heatmap as the study of Hoppe et al. [33] indicated that these features are important in the extraction of personality. We did not include features that contain explicit information about the content of the interface, so called areas of interest (AOI) even as previous work has shown that these features could have more predictive power [30]. The reason for this is that this information is already partially captured in a more general way by Most frequent saccade direction and fixations in a 4x4 heatmap. Thus, at this stage we chose to investigate how far we can go with display-independent features, which also have the advantage of possibly being more generalizable to other interfaces.
To explore whether classification of the three PCs would be possible with only a partial amount of data, we generated three different data windows to simulate partial observations of gaze data during the task similar to Steichen et al. [30] and Conati et al. [31].
Each window consists of a partial observation of each participant based on relative duration: the first window consisted of the first 30% of data, the second window of the first 60% and the last window consisted of the first 90% of data. Despite the fact that this approach requires a task to be fully completed to determine what 100% of the data constitutes, it still allows to provide valuable insights into trends and patterns about inferring PCs from gaze data [30]. Each of these windows consist of three different measure-ments and for each of these measurements the data was divided in ten different segments of equal length. For each of these segments, we generated the mentioned set of eye-tracking features resulting in a feature vector of 260 features for each measurement.
The reasoning behind creating these different datasets is to verify whether we would be able to adapt the RS interface to the needs of the user during the task. As such we did not include a window with 100% of the data as the adaptation would be too late. Additionally, previous research [30] showed already that after a certain amount of data, the accuracy started to converge or even that the accuracy decreases after a certain amount of data. In this study, we want to explore whether we would notice similar trends for different PCs.
To classify users in a low and high category, we used scikit-learn to train five different classifiers and a baseline [37]. To evaluate the performance of the classifiers, we applied a leave-one-out methodology. Because of this evaluation methodology and the uniform groups, we could not use the most common majority class baseline which predicts the most likely class (this would lead to 0% accuracy) [30,31,33]. As a consequence, we choose a random uniform baseline which has a theoretical accuracy of 50%. To classify the characteristics, we trained Logistic Regression, Random Forest, Gaussian Naive Bayes, Linear Support Vector Machines and Gradient Boosting. The reasoning behind the implementation of all these classifiers is that in previous research there is no consensus about which classifiers work the best. Steichen et al. [30] found that Logistic Regression performed better than Decision Trees, Support Vector Machines and Neural Networks. Lallé et al. [38] and Hoppe et al. [33] found that Random forests worked the best. However, Berkovsky et al. [7] conclude that Naive Bayes and Support Vector Machines are the best. Additionally, Gradient Boosting performed well in the study of Barral et al. [39]. Because of the small sample size, we chose not to use deep learning methods. For each of these classifiers we tried to optimize the accuracy. The resulting parameters can be found in Table 3.
To strengthen the stability of the results, we ran this evaluation 10 times with different random seeds. We calculated the average accuracy over all participants, and all runs to measure performance of the classifier.
To examine whether it is possible to classify users in the correct personality group and whether this classification works better on specific windows, we ran for each PC a two-way repeated measures ANOVA with accuracy as the dependent variable and both classifier and window as independent variables. As we run multiple ANOVA's and pairwise comparisons, the reported p-values are adjusted using the Benjamini and Hoghberg procedure [40] to control for the family-wise false discovery rate. The main results of this analysis are shown in Figure 2 and we will report the results for each of the PCs in detail in the next paragraphs. Need for cognition. The results of the twoway repeated measures ANOVA revealed a significant main effect of classifier on accuracy (F(7.14) =18.8, p<.001). To investigate this main effect, we ran post-hoc pairwise comparisons which showed that the mean accuracy of the logistic regression classifier (0.59) performed statistically better than the baseline (p=.0491) which is shown in Figure 2a. This figure also shows the accuracy in the three different windows and that the peak accuracy (0.67) is reached in the last window.
Musical sophistication. The results of the two-way repeated measures ANOVA revealed that no classifier could outperform the baseline and that most of the classifiers performed even worse.
Openness. The results of the two-way repeated measures ANOVA revealed a significant interaction effect of classifier with window on accuracy (F(14,28)=4.88, p<.001). An analysis of the effect of classifier showed a significant effect for the first window (F(7,16)=4.512, p=.006) and a posthoc test revealed that in this window the Gradient Boost performed significantly better than the baseline (p=.020). The analysis of the effect of window showed a significant effect for the Gradient Boost classifier (F(2,6)=8.12, p=.020) and a post-hoc analysis showed that the gradient boost classifier performed significantly better in the first window than in the second (p=.028) and the third window (p=.029). Figure 2b shows that the highest accuracy of Gradient Boost is reached in the first window (0.66). This accuracy is significantly higher than the accuracy of the baseline and the accuracy of Gradient Boost in the other windows.
Our results show that we have a higher accuracy than the random baseline for NFC and for openness in the first window, but that we were not able to do beat the random baseline classifier for MS.
For the classification of openness, it is interesting that we are able to outperform the baseline while openness was one of the few traits of which Hoppe et al. [33] could not outperform the baseline. This might be due to a different classification technique as Hoppe et al. only used a Random Forest classifier while we outperformed the baseline with a Gradient Boost classifier. Another possible reason could be that this difference is due to the fact that we trained the classifiers on different data windows and that our results show that the performance to classify openness is only significantly better than the baseline in the first window. As far as we know, no other studies formally showed that classifying PCs on early stages of the task can outperform classifiers trained on more data. However, other studies such as the study of Steichen et al. [30] already discussed this trend for perceptual speed, verbal working memory and visual working memory. They argued that these characteristics most strongly affect the gaze pattern of the user during the initial phase of a task and that other factors dilute the gaze pattern as the task continues. This is probably also the reason why we are only able to classify openness in the beginning of the task. However, this is not necessary a problem as we want to adapt an interface to the openness of a user as early on as possible. Nevertheless, the obtained accuracy is still too low to be used to adapt the explanations. Also, more research is needed to verify that openness will always affect the gaze during the beginning of a task or only when they see a new interface.
To classify NFC, our results show a signif- icant main effect of Logistic Regression on accuracy. The reason that we do not see a significant difference between the windows could be that NFC is correlated with decisionmaking processes [12] and creating a playlist in a music RS constantly involves making decisions. Despite the significant main effect, the accuracy to classify NFC seems not high enough to adapt the interface, especially not in the first two windows. As a consequence, this means that further research needs to focus on reaching a higher accuracy in the beginning of the interaction to be able to adapt explanations early on in the process or on adapting the interface if the user re-visits the application. Additionally, further research should investigate why Logistic Regression performed the best to classify NFC as this is similar to previous studies in which Logistic Regression performed well to classify PCs, but we do not have an explanation why logistic regression outperforms other algorithms [30,41].
As a previous study in the field of music RS showed that MS influences the way users look to a music RS interface and previous studies in the field of information retrieval also showed the potential of predicting domain knowledge based on eye tracking [42,43,9], we expected to be able to classify MS based on gaze data. However, our results show that we could not outperform the baseline. A pos-sible reason for this could be that we did not include AOI related features which were included in the above-mentioned studies. An interesting further line of research is to verify whether including these AOI features can improve accuracy.
In this paper, we explored whether it would be possible to adapt the explanations in a music RS interface based on personal characteristics. To do so, we investigated whether a classification of personal characteristics could be inferred by studying the gaze pattern during the creation of a playlist in this system. More concretely, we classified musical sophistication, need for cognition and openness because these characteristics have shown to impact the user experience of explanation in a RS [9]. We trained the classifiers on different windows to detect whether the classification would already work with only a partial observation of the creation of a playlist.
Our results show that even as our accuracy is not yet high enough for practical use, we are able to outperform a baseline to classify need for cognition with Logistic Regression. If we only consider the first third of the data, our results show that the classification of openness with Gradient Boost beats the baseline. Despite the limitations in terms of accuracy, this finding is important because it shows the potential to adapt explanations during the interaction with a music RS interface. In a next step, we want to increase the accuracy of the classifiers particularly in the beginning of the interaction which we plan to do by gathering more training data and by using different features such as AOI related features. Additionally, more research is needed to verify whether the results of this study could be generalized to different tasks and interfaces which we also plan to address in future research.
Part of this research has been supported by the KU Leuven Research Council (grant agreement C24/16/017) and the Research Foundation Flanders (FWO).
users' domain knowledge from search behaviors, in: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011, pp. 1225-1226.