Due to the increasing importance of recommender systems in our life, the call to make these systems more transparent becomes louder. However, providing explanations is not as easy as it seems, as research has shown that diferent users have varying reactions to explanations. So not only the recommendations, but also the explanations should be personalised. As a first step towards these personalised explanations, we explore the possibility to classify users based on their gaze pattern during the interaction with a music recommender system. More specifically, we classify three personal characteristics that have been shown to play a role in the interaction with music recommendations: need for cognition, openness and musical sophistication. Our results show that classification based on eye tracking has potential for need for cognition and openness, as we are able to do better than random, but not for musical sophistication as no classifier did better than a uniform random baseline.
In the field of recommender systems (RS),
researchers are increasingly aware that
optimizing accuracy is not enough to reach the
full potential of recommender systems (RS)
[
However, before the system could adapt
explanations to personal characteristics (PCs),
it needs to be aware of the PCs of the user. A
possible way to obtain these characteristics is
by explicitly asking the users to fill in
questionnaires [
to obtain a fine-grained result, but a
classification into two categories sufices [
For this reason, we explore in this paper With the increasing role of RS in our daily
whether it is possible to classify users’ per- lives, the call for explainable, transparent RS
sonality traits during the interaction with a also becomes louder so that users can make
music RS with explanations by analyzing their better informed decisions whether or not to
gaze. We will focus on three diferent PCs: follow the recommendations [
Openness is one of the Big Five personal- ever, research has shown that diferent users
ity traits which measures how open a per- have diferent reactions to explanations [6,
son is to new experiences. Millecamp et al. 12, 13]. In the field of music RS, recent
re[
Similarly, we hypothesize that inferring MS, model [14]. This model describes personality
which is a measure of domain knowledge in in five diferent traits and it has been used
the music domain, from gaze data might be in several studies which showed the positive
possible as the study of Millecamp et al. [
NFC is a cognitive style which influences experiental life [16]. It has been shown that
the way a person prefers to process informa- openness is related to the preferred amount
tion and thus looks at information. Previ- of diversity in RS and to the willingness to
ous studies already showed that NFC moder- use a system with explanations [
Next to exploring the general accuracy, we ful cognitive activities” [21]. NFC has been also want to explore how much data we need shown to have an impact on the willingness to infer these PCs. of users to rely on a RS [12], on the confi
The contribution of this paper is twofold. dence in a playlist created in a music RS with
First, to our knowledge, we are the first to ex- explanations [
PC Age MS NFC
supported to make a decision in a RS interface that provided explanations than an interface without such explanations, while this made no diference for users with low MS.
Another study showed that users with high domain experience perceive a higher diversity in a scatter plot than in a simpler bubble chart [25].
To acquire the PCs of users, the most
common way is to ask users to fill in validated
questionnaire [
The previously mentioned works rely on ifne-grained personality scores. In contrast, in our work we focus on adapting interfaces to users for which we only need a classification in two groups. We aim to base this classification on the gaze pattern during the interaction with a music RS interface instead of asking users to watch carefully selected stimuli, to fill in questionnaires or to share their social media profile. Previous studies which classified users based on their gaze pattern during normal activities are almost all only focused on cognitive abilities and visualiza
consists of the gaze data of 30 participants
(21 male). For the three PCs, the participants
were divided into a high and low group based
on a median split. This resulted in equally
distributed groups for MS and NFC and
almost equally groups for openness (16 in the
low and 14 in the high openness group). An
short overview of the characteristics of the
3. Data participants can be found in Table 1.
The gaze data was recorded with a Tobii
The gaze data that is used in this study was 4C remote eye tracker at a sampling rate of
generated in a user study by Millecamp et al. 90Hz. Each sample contained information about
[
As shown in Part A of this figure, users circle on the screen in which the user keeps
ifrst can search for an artist they like through focusing for at least 100ms without moving
a search bar in the top left corner. When they their eyes more than one degree. All other
add the artist, this artist is shown in Part B. movements are then identified as saccades,
Based on this artist, the system starts to gen- i.e. quick movements of gaze from one
fixerate recommendations which were listed in ation to another [30].
a two-column format as shown in Part F. Based on these saccades and fixations, we
When users hover over the cover of the pic- generated a set of eye-tracking features as listed
ture of a recommended song, they can click in Table 2. Most of these features are selected
a play button to listen to a 30s preview of the because they are widely used in previous eye
song. On the right side of each explanation, tracking studies [
After the user explored all the options of tive power [30]. The reason for this is that the interface, the recording of the gaze started. this information is already partially captured As shown in Part E of Figure 1, users were in a more general way by Most frequent sacasked to create a playlist of five songs. To cre- cade direction and fixations in a 4x4 heatmap. ate this playlist, they could use all function- Thus, at this stage we chose to investigate alities without any restriction. When they how far we can go with display-independent added the fifth song to their playlist, we features, which also have the advantage of stopped the recording of the gaze. On aver- possibly being more generalizable to other inage, users took 4 minutes 26 seconds to com- terfaces. plete their playlist. As part of this paper’s contribution, this data is publicly available 3. 4.2. Data windows
web-api/reference/tracks/get-audio-features/ 3augment.cs.kuleuven.be/datasets/classifeye
A B D C G H I J
amount of data, we generated three difer- 60% and the last window consisted of the first ent data windows to simulate partial obser- 90% of data. Despite the fact that this apvations of gaze data during the task similar proach requires a task to be fully completed to Steichen et al. [30] and Conati et al. [31]. to determine what 100% of the data constiEach window consists of a partial observa- tutes, it still allows to provide valuable intion of each participant based on relative du- sights into trends and patterns about inferration: the first window consisted of the first ring PCs from gaze data [30]. Each of these 30% of data, the second window of the first windows consist of three diferent measurements and for each of these measurements Table 3 the data was divided in ten diferent segments Description of parameters of the diferent classiof equal length. For each of these segments, fiers we generated the mentioned set of eye-tracking Classifier Parameter features resulting in a feature vector of 260 features for each measurement. LBoagseisltiniceRegression sstorlavteerg:yli:bulinniefaorrm
The reasoning behind creating these dif- Random Forest estimators: 100 ferent datasets is to verify whether we would Gaussian Naive Bayes na boef tahbeleutsoeraddauprtinthgetRheS itnastekr.faAcse stoucthhewneeeddids Linear Support Vector Machines gparombmabai:listyca:lTerue not include a window with 100% of the data Gradient Boosting maximum depth: 4 as the adaptation would be too late. AddittcadPwihucaCoeacrtnsatuaw.a.cralayIolfnycutse,ytltprdhadrirasenetvcosecritdteoueircaudttesosayei,rsnscweiomsaeanefimtavwleareorarcruhngatntre[cett3eoon0rord]etfasxseidhpvnfaooleotawrnarme,detdiohwtfehuaarhetnelerntteaththoace-defry ttcebBwohaveaeauesyluttrsle.,seRsiBeAnaoeaddnfrntdedkhtdiehootepiemvSosussnltkepmfuaoyapdlarrloeyyenltlr,sitoaGntssalfVgr.mwaeB[mdc7paoti]rleroeercktnrahoestlMnoidBzedceaottls,huco.awhedsliteF.ebnionteec[hrgss3hate9po.ata]esrNH.cerehfaooBntiwhvrooemee--tfed these classifiers we tried to optimize the ac4.3. Classification methods curacy. The resulting parameters can be found To classify users in a low and high category, in Table 3. we used scikit-learn to train five diferent clas- To strengthen the stability of the results, sifiers and a baseline [37]. To evaluate the we ran this evaluation 10 times with diferperformance of the classifiers, we applied a ent random seeds. We calculated the average leave-one-out methodology. Because of this accuracy over all participants, and all runs to evaluation methodology and the measure performance of the classifier. uniform groups, we could not use the most common majority class baseline which pre- 5. Results dicts the most likely class (this would lead to 0% accuracy) [30, 31, 33]. As a consequence, To examine whether it is possible to classify we choose a random uniform baseline which users in the correct personality group and whether has a theoretical accuracy of 50%. To clas- this classification works better on specific winsify the characteristics, we trained Logistic dows, we ran for each PC a two-way repeated Regression, Random Forest, Gaussian Naive measures ANOVA with accuracy as the deBayes, Linear Support Vector Machines and pendent variable and both classifier and winGradient Boosting. The reasoning behind the dow as independent variables. As we run mulimplementation of all these classifiers is that tiple ANOVA’s and pairwise comparisons, the in previous research there is no consensus reported p-values are adjusted using the Benabout which classifiers work the best. Ste- jamini and Hoghberg procedure [40] to conichen et al. [30] found that Logistic Regres- trol for the family-wise false discovery rate. sion performed better than Decision Trees, The main results of this analysis are shown Support Vector Machines and Neural Networks.in Figure 2 and we will report the results for Lallé et al. [38] and Hoppe et al. [33] found
each of the PCs in detail in the next paragraphs.
Need for cognition. The results of the two- Our results show that we have a higher accuway repeated measures ANOVA revealed a racy than the random baseline for NFC and significant main efect of classifier on accu- for openness in the first window, but that we racy (F(7.14) =18.8, p<.001). To investigate this were not able to do beat the random baseline main efect, we ran post-hoc pairwise com- classifier for MS. parisons which showed that the mean accu- For the classification of openness, it is inracy of the logistic regression classifier ( 0.59) teresting that we are able to outperform the performed statistically better than the base- baseline while openness was one of the few line (p=.0491) which is shown in Figure 2a. traits of which Hoppe et al. [33] could not This figure also shows the accuracy in the outperform the baseline. This might be due three diferent windows and that the peak ac- to a diferent classification technique as Hoppe curacy (0.67 ) is reached in the last window. et al. only used a Random Forest classifier
Musical sophistication. The results of while we outperformed the baseline with a the two-way repeated measures ANOVA re- Gradient Boost classifier. Another possible vealed that no classifier could outperform the reason could be that this diference is due to baseline and that most of the classifiers per- the fact that we trained the classifiers on difformed even worse. ferent data windows and that our results show
Openness. The results of the two-way re- that the performance to classify openness is peated measures ANOVA revealed a signifi- only significantly better than the baseline in cant interaction efect of classifier with win- the first window. As far as we know, no other dow on accuracy (F(14,28)=4.88, p<.001). An studies formally showed that classifying PCs analysis of the efect of classifier showed a on early stages of the task can outperform significant efect for the classifiers trained on more data. However, ifrst window ( F(7,16)=4.512, p=.006) and a post- other studies such as the study of Steichen et hoc test revealed that in this window the al. [30] already discussed this trend for perGradient Boost performed significantly bet- ceptual speed, verbal working memory and ter than the baseline (p=.020). The analysis visual working memory. They argued that of the efect of window showed a significant these characteristics most strongly afect the efect for the Gradient Boost classifier gaze pattern of the user during the initial phase (F(2,6)=8.12, p=.020) and a post-hoc analysis of a task and that other factors dilute the gaze showed that the gradient boost classifier per- pattern as the task continues. This is probformed significantly better in the first win- ably also the reason why we are only able dow than in the second (p=.028) and the third to classify openness in the beginning of the window (p=.029). Figure 2b shows that the task. However, this is not necessary a probhighest accuracy of Gradient Boost lem as we want to adapt an interface to the is reached in the first window ( 0.66). This ac- openness of a user as early on as possible. curacy is significantly higher than the accu- Nevertheless, the obtained accuracy is still too racy of the baseline and the accuracy of Gra- low to be used to adapt the explanations. Also, dient Boost in the other windows. more research is needed to verify that openness will always afect the gaze during the beginning of a task or only when they see a new interface.
To classify NFC, our results show a signifiFrst pOens rGdienta (a) Accuracy of Logistic Regression for NFC.
(b) Accuracy of Gradient Boost for openness. could be that NFC is correlated with decision- interesting further line of research is to vermaking processes [12] and creating a playlist ify whether including these AOI features can in a music RS constantly involves making de- improve accuracy. cisions. Despite the significant main efect, the accuracy to classify NFC seems not high enough to adapt the interface, especially not in the first two windows. As a consequence, this means that further research needs to focus on reaching a higher accuracy in the beginning of the interaction to be able to adapt explanations early on in the process or on adapting the interface if the user re-visits the application. Additionally, further research should investigate why Logistic Regression performed the best to classify NFC as this is similar to previous studies in which Logistic Regression performed well to classify PCs, but we do not have an explanation why logistic regression outperforms other algorithms we could not outperform the baseline. A pos
In this paper, we explored whether it would be possible to adapt the explanations in a music RS interface based on personal characteristics. To do so, we investigated whether a classification of personal characteristics could be inferred by studying the gaze pattern during the creation of a playlist in this system.
tication, need for cognition and openness because these characteristics have shown to impact the user experience of explanation in a
windows to detect whether the classification would already work with only a partial observation of the creation of a playlist.
Our results show that even as our accuracy is not yet high enough for practical use, we are able to outperform a baseline to classify need for cognition with Logistic Regression. If we only consider the first third of the data, our results show that the classification of openness with Gradient Boost beats the baseline. Despite the limitations in terms of accuracy, this finding is important because it shows the potential to adapt explanations during the interaction with a music RS interface. In a next step, we want to increase the accuracy of the classifiers particularly in the beginning of the interaction which we plan to do by gathering more training data and by using diferent features such as AOI related features. Additionally, more research is needed to verify whether the results of this study could be generalized to diferent tasks and interfaces which we also plan to address in future research.
mation Science, Springer, 2020, pp. 212– based recommender systems: technolo228. gies and research issues, in: Proceed[12] S. T. Tong, E. F. Corriero, R. G. Math- ings of the 10th international confereny, J. T. Hancock, Online daters’ will- ence on Electronic commerce, 2008, pp. ingness to use recommender technol- 1–10. ogy for mate selection decisions., in: In- [21] J. T. Cacioppo, R. E. Petty, C. Feng Kao, tRS@ RecSys, 2018, pp. 45–52. The eficient assessment of need for [13] S. Naveed, T. Donkers, J. Ziegler, cognition, Journal of personality asArgumentation-based explanations in sessment 48 (1984) 306–307. recommender systems: Conceptual [22] K. Y. Tam, S. Y. Ho, Web personalizaframework and empirical results, in: tion as a persuasion strategy: An elaboAdjunct Publication of the 26th Con- ration likelihood model perspective, Inference on User Modeling, Adaptation formation systems research 16 (2005) and Personalization, 2018, pp. 293–298. 271–291. [14] L. R. Goldberg, The structure of pheno- [23] M. Millecamp, R. Haveneers, K. Verbert, typic personality traits., American psy- Cogito ergo quid? the efect of cognitive chologist 48 (1993) 26. style in a transparent mobile music rec[15] R. Hu, P. Pu, Enhancing collaborative ommender system, in: Proceedings of ifltering systems with personality infor- the 28th ACM Conference on User Modmation, in: Proceedings of the fifth eling, Adaptation and Personalization, ACM conference on Recommender sys- 2020, pp. 323–327.
tems, 2011, pp. 197–204. [24] D. Müllensiefen, B. Gingras, J. Musil, [16] V. Benet-Martinez, O. P. John, Los cinco L. Stewart, The musicality of nongrandes across cultures and ethnic musicians: an index for assessing mugroups: Multitrait-multimethod analy- sical sophistication in the general popses of the big five in spanish and en- ulation, PloS one 9 (2014) e89642. glish., Journal of personality and social [25] Y. Jin, N. Tintarev, K. Verbert, Efects psychology 75 (1998) 729. of individual traits on diversity-aware [17] N. Tintarev, M. Dennis, J. Masthof, music recommender user interfaces, in: Adapting recommendation diversity to Proceedings of the 26th Conference on openness to experience: A study of hu- User Modeling, Adaptation and Personman behaviour, in: International Con- alization, 2018, pp. 291–299. ference on User Modeling, Adaptation, [26] J. Golbeck, C. Robles, M. Edmondson, and Personalization, Springer, 2013, pp. K. Turner, Predicting personality from 190–202. twitter, in: 2011 IEEE third interna[18] L. Chen, W. Wu, L. He, How personality tional conference on privacy, security, influences users’ needs for recommen- risk and trust and 2011 IEEE third interdation diversity?, in: CHI’13 extended national conference on social computabstracts on human factors in comput- ing, IEEE, 2011, pp. 149–156. ing systems, 2013, pp. 829–834. [27] M. X. Zhou, G. Mark, J. Li, H. Yang, [19] U. Gretzel, D. R. Fesenmaier, Persua- Trusting virtual agents: the efect of sion in recommender systems, Interna- personality, ACM Transactions on Intional Journal of Electronic Commerce teractive Intelligent Systems (TiiS) 9 11 (2006) 81–100. (2019) 1–36. [20] A. Felfernig, R. Burke, Constraint- [28] J. Wache, R. Subramanian, M. K. Abadi, R.-L. Vieriu, N. Sebe, S. Winkler, Im- search & applications, ACM, 2000, pp. plicit user-centric personality recogni- 71–78. tion based on physiological responses [36] J. H. Goldberg, X. P. Kotval, Comto emotional videos, in: Proceedings of puter interface evaluation using eye the 2015 ACM on International Confer- movements: methods and constructs, ence on Multimodal Interaction, 2015, International journal of industrial erpp. 239–246. gonomics 24 (1999) 631–645. [29] D. Toker, C. Conati, B. Steichen, [37] F. Pedregosa, G. Varoquaux, A. GramG. Carenini, Individual user charac- fort, V. Michel, B. Thirion, O. Grisel, teristics and information visualization: M. Blondel, P. Prettenhofer, R. Weiss, connecting the dots through eye track- V. Dubourg, J. Vanderplas, A. Passos, ing, in: proceedings of the SIGCHI D. Cournapeau, M. Brucher, M. Perrot, Conference on Human Factors in E. Duchesnay, Scikit-learn: Machine Computing Systems, 2013, pp. 295–304. learning in Python, Journal of Machine [30] B. Steichen, C. Conati, G. Carenini, Learning Research 12 (2011) 2825–2830.
Inferring visualization task properties, [38] S. Lallé, C. Conati, G. Carenini, Predicuser performance, and user cognitive tion of individual learning curves across abilities from eye gaze data, ACM information visualizations, User ModTransactions on Interactive Intelligent eling and User-Adapted Interaction 26 Systems (TiiS) 4 (2014) 1–29. (2016) 307–345. [31] C. Conati, S. Lallé, A. Rahman, D. Toker, [39] O. Barral, S. Lallé, G. Guz, A. Iranpour, Further results on predicting cognitive C. Conati, Eye-tracking to predict user abilities for adaptive visualizations, IJ- cognitive abilities and performance for CAI International Joint Conference on user-adaptive narrative visualizations, Artificial Intelligence (2017) 1568–1574. in: Proceedings of the 2020 Internadoi:10.24963/ijcai.2017/217. tional Conference on Multimodal Inter[32] M. Gingerich, C. Conati, Constructing action, 2020, pp. 163–173. models of user and task characteristics [40] Y. Benjamini, Y. Hochberg, Controlling from eye gaze data for user-adaptive in- the false discovery rate: a practical and formation highlighting, in: Proceed- powerful approach to multiple testing, ings of the AAAI Conference on Arti- Journal of the Royal statistical society: ifcial Intelligence, 1, 2015. series B (Methodological) 57 (1995) 289– [33] S. Hoppe, T. Loetscher, S. A. Morey, 300.
A. Bulling, Eye movements during [41] S. Kardan, C. Conati, Exploring gaze everyday behavior predict personality data for determining user learning with traits, Frontiers in Human Neuro- an interactive simulation, in: Interscience 12 (2018) 1–8. doi:10.3389/ national Conference on User Modelfnhum.2018.00105. ing, Adaptation, and Personalization, [34] O. P. John, E. M. Donahue, R. L. Kentle, Springer, 2012, pp. 126–138.
The big five inventory—versions 4a and [42] M. J. Cole, J. Gwizdka, C. Liu, N. J. 54, 1991. Belkin, X. Zhang, Inferring user knowl[35] D. D. Salvucci, J. H. Goldberg, Iden- edge level from eye movement patterns, tifying fixations and saccades in eye- Information Processing & Management tracking protocols, in: Proceedings of 49 (2013) 1075–1091. the 2000 symposium on Eye tracking re- [43] X. Zhang, M. Cole, N. Belkin, Predicting users’ domain knowledge from search behaviors, in: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011, pp. 1225–1226.