Context-aware recommender systems (CARS) have been developed to adapt to users' preferences in different contextual situations. Users' emotions have been demonstrated as one of effective context information in recommender systems. However, there are no work exploring the effect of emotional reactions (or expressions) in the recommendation process. In this paper, we assume that users may give similar ratings even if they present different emotional reactions or expressions on the movies. We further model the traits of emotional reactions and incorporate them into context-aware matrix factorization as regularization terms. Our experimental results based on the LDOS-CoMoDa movie data set validate our assumptions and prove that it is useful to take emotional reactions into consideration in context-aware recommendations.
H.3.3 [Information Search and Retrieval]: Information filtering, Retrieval models; H.1.2 [Models and Principles]: User/Machine Systems - human information processing
Recommender systems (RS) are an effective way in alleviating information overload by tailoring recommendations to users’ personal preferences. Context-aware recommender systems (CARS) take contextual factors (such as time, location, companion, occasion, etc) into account in modeling user profiles and in generating recommendations. For example, users’ choice on movies may be very different if the user is going to watch the movie with children rather than with his or her partner.
Context, is usually defined as, "any information that can be
used to characterize the situation of an entity. An entity is
a person, place, or object that is considered relevant to the
interaction between a user and an application, including the user
and applications themselves [
Emotional reactions or expressions are highly correlated with
the traits of user personalities. Personality accounts for the most
important ways in which individuals differ in their enduring
emotional, interpersonal, experimental, attitudinal and motivational
styles [
Our previous research [
Therefore, the users’ rating profiles associated with different (or even opposed) emotional reactions therefore could be useful to assist recommendations. In this paper, we propose to incorporate emotional reactions (or expressions) as regularization terms in the context-aware matrix factorization approach, and further explore its effect on the performance of context-aware recommendations.
The following sections are organized as follows: Section 2 introduces related work, including the background of contextaware recommendation, and the role of emotions and personality in recommender systems. Section 3 gives the preliminary description of essential information, such as the LDOS-CoMoDa movie data which contains emotional variables, and the introduction about the CAMF technique. Section 4 discusses our methodology that incorporates the emotional reactions as regularization terms into the CAMF approach. Section 5 describes our experimental results and discussions, followed by Section 6 which concludes our findings and discusses our future work.
One of the goals in the recommender systems (RS) is to assist
users’ decision making by providing a list of recommendations.
Due to the fact that users’ choice usually varies from time to time
and from context to context, context-aware recommender systems
(CARS) [
In rating-based RS applications, such as movie or book ratings,
the standard formulation of the recommendation problem begins
with a two dimensional matrix of ratings, organized by user and
item: Users Items ! Ratings. The key insight of CARS is that
users’ preferences on items may be also a function of the context in
which those items are encountered. Incorporating contexts requires
that we estimate user preferences using a multidimensional rating
function, Users Items Contexts ! Ratings [
In the past decade, several context-aware recommendation
algorithms have been developed. By additionally incorporating context
information, these algorithms have been demonstrated to be useful
to improve recommendation performance in numerous domains,
such as e-commerce [
It is well known that human decision making is subject to both
rational and emotional influences [
Emotional states, accordingly, are also viewed and used as
contexts in recommender systems. Shi et al. [
Emotional states are usually dynamic and may change from time
to time. Based on the introduction about the affective recommender
systems [
To further discuss the topics in the context-aware recommendation, it is necessary to introduce some terminologies:
Table 1 shows an example of context-aware movie data which contains five rating profiles given by three users on four movies in different contextual situations. In our discussions, we will use the term contextual dimension to denote the contextual variable, such as "Location", "Time" and "Companion". The term contextual condition refers to a specific value in a contextual dimension, e.g. "Home" and "Cinema" are two contextual conditions for the dimension "Location". Context or contextual situation therefore refers to a combination of contextual conditions, e.g., {Weekday, Home, Kids}.
Next, we introduce the LDOS-CoMoDa movie data 1 which is a data set with multiple contextual dimensions including several emotional variables. We also introduce context-aware matrix factorization which is a popular algorithm in CARS and we use it as a base algorithm in this paper. 3.1
In the domain of context-aware recommendation, there are very
limited number of data sets available for public research, not to
mention the data that contains emotional variables. The
LDOSCoMoDa data set [
Among these 12 contextual dimensions, there are three ones that can be considered emotional dimensions: endEmo, domEmo and mood. "endEmo" is the emotional state experienced at the end of the movie (i.e., emotion in the exit stage). "domEmo" is 1LDOS-CoMoDa data set, http://www.ldos.si/comoda.html the emotional state experienced the most during watching (i.e., emotion in the consumption stage). "mood" is the emotion of the user during that part of the day when the user watched the movie (i.e., emotion in the entry stage). "EndEmo" and "domEmo" contain the same seven conditions: Sad, Happy, Scared, Surprised, Angry, Disgusted, Neutral, while "mood" only has three simple conditions: Positive, Neutral, Negative.
Context selection is usually performed before we apply any
context-aware recommendation algorithms. We’d like to retain
the most influential context dimensions, since irrelevant ones may
introduce noises in the data and further hamper the
recommendation accuracy. Based on the statistical selection method introduced
in [
The three emotional variables (i.e., mood, domEmo and
endEmo) describe users’ affective states during the user interactions
with the movies in terms of three stages respectively: entry stage,
consumption stage and exit stage as introduced in [
One of the most popular context-aware recommendation
algorithms is the one built upon matrix factorization, namely, the
context-aware matrix factorization (CAMF) approach [
N + X Bu;cj + bi + pTu qi j=1 (1)
Assume there are totally N contextual dimensions. c1c2!::cN is used to denote a contextual situation, where c1 indicates the value of contextual condition in the 1st context dimension. r^uic1c2:::cN " # # therefore represents the predicted rating for user u on item i in the situation c1c2::cN . The prediction function is composed of four components: the global me$anratin%g, item rating bias bi, N the aggregated contextual &rating’b’ias P Bu;cj , and user-item j=1 interaction represented by th(e$dot p’roduct of a user vector and item ) vector, pTu qi. pu is the user vector represented by a set of latent factors, and qi is the item vector represented by the same set of () factors. pu can tell how much the user u likes those latent factors, while qi indicates how the item i ob$tains the)sefactors. Therefore, * the dot product function is used to estimate how much the user will like this item. (+ )
The term Bu;cj is the estimated contextual rating bias for user u in context condition cj . It is used to denote how user u’s rating is , deviated in each contextual condition.
- . / err = ruic1c2:::cN r^uic1c2:::cN (2) min X 4 21 err2 + B ;b ;p ;q r2R
N (X Bu2;cj + bi2 + jjpujj2 + jjqijj2)5 2 j=1 (3)
Afterwards, the algorithm is able to learn the corresponding parameters by minimizing the squared errors in prediction. The loss function as shown in Equation 3 is a composition of squared error and regularization terms which are used to alleviate the overfitting problems, where ruic1c2:::cN is the real and known rating given by user u on item i in context c1c2::cN , and is the regularization rate used in the optimization process. By stochastic gradient descent, we are able to learn the parameters iteratively and finally achieve the best performing CAMF_CU model.
CAMF is an effective algorithm and it is able to alleviate the data
sparsity to some extent. We choose CAMF_CU because we are
going to explore the correlation between users and their emotional
reactions, which requires a user-specific context-dependent model.
The same thing can also happen to other algorithms which explore
intersections or the dependency between users and contexts, such
as the CSLIM_CU approach [
In the next section, we will introduce how to incorporate the emotional reactions as regularization terms to CAMF_CU. 4.
In this section, we introduce our methodology of how to incorporate emotional reactions into context-aware recommender systems. 4.1
Recall that we assume that the different emotional reactions or expressions can be used to model users’ rating behaviors. For example, assume two users gave a high rating on a same tragedy drama movie. One of them indicated his or her emotional state as "happy" when finishing the movie, because this user thought it was a really good movie. But another user may express his or her feeling as "sad" since it is a tragedy movie. The same thing may also happen to the domEmo in addition to the endEmo. The emotional reactions or expressions in this paper, refer to the different values in the dimension domEmo and/or endEmo in the LDOS-CoMoDa data.
Figure 1 presents the distribution of rating counts in each emotional state. Note that "Unknown" indicates the missing value in the LDOS-CoMoDa. We can observe that Neutral and Happy are the most two common emotional expressions in both domEmo and endEmo.
1000 800 700 >600
500 =400 <# 00 . / !!"#$%&’ 0 !!"#$%&’ ) ) 2 34’$#$56"#$%&’
1 -./.
-/.
-./.
-/.
Furthermore, we’d like to learn the unusual case to see whether users present different emotional reactions in this data. An unusual case could be two situations: 1). a user leaves a negative rating, but expresses positive emotional states in either domEmo or endEmo; 2). a user gives a positive rating while he finally indicates a negative emotion in either domEmo or endEmo.
To explore these unusual cases, we need to define which ratings and which emotions are positive or negative. In our experiments, we simply view a rating as a positive one if the rating is no less than 4; otherwise, the rating is negative. In terms of the emotional states, we only consider "Happy" and "Surprised" as positive ones, while other emotional states are negative.
A simple statistics about unusual cases in this data can be depicted by Figure 2. First of all, 66.5% of the ratings are positive ones as shown in subfigure a). Based on the subfigure b), we can observe that 36.9% of all the rating records are unusual cases (i.e., the two situations mentioned above) based on the domEmo variable, while it is 33.4% in the endEmo variable. This may tell that domEmo could be more effective and useful than endEmo in modeling users’ emotional reactions.
The subfigure c) and d) further describe the two unusual situations among the positive and negative ratings respectively. In the piece of profiles with positive ratings, 43.4% of them are associated with negative emotions in domEmo – many more than the cases in endEmo. It is not surprising, since the theme or the genre of the movie will affect user’s dominating emotions during the process of movie watching. For example, users may feel horrible or scared when watching a horrible movie, but finally leave a positive rating since it is a good movie. On the other hand, in terms of the records with negative ratings, there are no significant differences for the unusual cases between domEmo (24%) and endEmo (25%) based on the observations subfigure d). Recall that, there are many more positive ratings than the negative ones in this data. Therefore, it seems that users may express more unusual emotional reactions in domEmo rather than in endEmo. We suspect that the emotional reactions in domEmo may leave more influential impact on our proposed recommendation models.
The underlying assumption in our proposed approach is that user’s emotional reactions or expressions on the future emotional states (e.g., domEmo and/or endEmo) can be used to improve recommendations, since they may indicate similar user tastes even if the emotional reactions are different or even opposed. The research problem can be summarized as how to incorporate these emotional reactions into existing recommendation algorithms. More specifically, we want to explore the approach to incorporate them into the CAMF approach. There are three questions we are particularly interested in:
Which emotional reaction is more effective? The reactions based on domEmo or endEmo? 4.2
First of all, how the user reacts on the movies in terms of emotional status is dependent with what type of movies the user is watching. In this case, we additionally use movie genre information in the LDOS-CoMoDa data and aggregated users’ ratings for each movie type. A sample of the aggregated data can be shown in Table 3.
In Table 3, we replace the column of item by movie genre to construct a new rating matrix. We will use the same 7 contextual dimensions introduced previously. Note that in the LDOS-CoMoDa data we do not know what the movie genre is, since the genre was encoded as numbers in this data.
Afterwards, we can fuse an emotional dimension (either endEmo or domEmo) into the user dimension to create a two-dimensional rating matrix. Let’s take the domEmo for example, the converted rating matrix can be described by Table 4.
Specifically, we fuse the values in domEmo into the user column to create new users. The new user is represented by a combination of original user ID and value in the domEmo, and we name those new users as emotional users. Meanwhile, we eliminate the other contextual dimensions from the rating matrix. In this case, we can build a matrix factorization model based on this converted twodimensional rating matrix. And then we are able to calculate the similarity between emotional users based on the cosine similarity of each two vectors which represent emotional users. For example, we can measure how similar the "U1, Sad" to "U2, Angry" based on their co-ratings on the movies with the same genre information.
Theoretically, we can use the item information (e.g., item ID) instead of the movie genre in the rating matrix, but it will increase data sparsity. We use movie genre information only for two reasons: On one hand, using movie genre is based on our assumptions that users’ different emotional reactions depend on the movie genre and user’s emotional reactions, for example, user may express as happy or sad on a tragedy movie. On the other hand, it is able to alleviate the rating sparsity in the converted twodimensional rating matrix so that we can obtain more reliable user similarities. We have tried to use item ID, but emotional users have very few co-ratings on the items, which results in worse recommendation performance compared with that when we use genre information only. Note that we use domEmo as an example in Table 4, while we can also have the same process based on the variable endEmo.
In short, the emotional users should be similar if they have similar ratings on the movies with same genre information, even if the original users have different emotional reactions on domEmo or endEmo. For example, the ratings given by "U1, Sad" and "U2, Angry" are all 3-star on the drama movies shown in the Table 4. Therefore, U1 with dominating emotion as "Sad" may share similar user tastes with U2 with dominating emotion as "Angry" to some extent.
Accordingly, we are able to create a regularization term based on the similarity of contextual users. The new loss function can be shown as Equation 4, where is the regularization rate for the new regularization terms.
We will use the same function shown in Equation 1. In addition, we incorporate a new regularization term in Equation 4 compared with the loss function described by Equation 3.
More specifically, we use m to denote the index of an emotional variable (i.e., either domEmo or endEmo). Take domEmo for example, m indicates the position of domEmo in the list of contextual dimensions, thus cm is used to express user’s emotional state in domEmo. According, "u; cm" is the emotional user (introduced as Table 4), and we use K to denote the top-K nearest neighbor of emotional user "u; cm" based on the user similarity calculated based on the matrix factorization model built upon the converted two-dimensional rating matrix. Namely, "v; cm+" is one of the identified top-K nearest neighbors. We use cm+ to denote the emotional state in domEmo, since it is not necessary to be the same value as cm. But they should be the contextual condition in the same dimension (i.e., the mth dimension).
As mentioned previously, more similar two emotion users are, their ratings on the items (with same genre) should be similar. In our CAMF_CU model, it can be derived that user’s contextual rating deviations in this emotional variable (i.e., the mth contextual variable) should be similar. Namely, Bu;cm and Bv;cm+ should be very close. We add the squared difference of these two deviations (e.g., Equation 5) as the regularization term in Equation 4.
Additionally, how close the two contextual rating deviations are should be dependent with the similarity of two emotional users. In this case, the regularization term is weighted by the similarity between two emotional users. We name this term as "emotional regularization term" in this paper.
Recall that our assumption is that the emotional users should be similar because two difference users have similar ratings even if their emotional reactions are different. It can also tell that the two users actually share something in common, so we assume there should also be a similarity between two users to some extent. Therefore, we are able to additionally incorporate a "user regularization term" to build a finer-grained recommendation model, where the loss function can be shown in Equation 6. Again, the user regularization is also weighted by the similarity between two emotional users.
Based on those two different loss functions, we are able to build two new CAMF approaches by incorporating the emotional reactions as the regularization terms. We can learn the corresponding parameters based on the gradient decent accordingly. Note that the performance of the models may also depend on the number of Knearest neighbors used in the algorithm. In our experiments, we set different values to explore the best options in these parameters. 5.
In this section, we introduce our evaluation settings and experimental results, as well as our findings. 5.1
We employ a 5-folds cross-validation on the LDOS-CoMoDa data set. Namely, we split the rating profiles into 5 folds and perform 5 rounds evaluations. For each round, one of the fold will be used as testing set, and the other 4 folds of data will be used as training data. We build our recommendation models based on the training set and evaluate the results according to the ground truth inferred from the testing set.
We use CAMF_CU approach as baseline, and compete its
recommendation performance with the CAMF_CU models with
different regularization terms. We use the CAMF_CU approach
implemented in the open-source toolkit, CARSKit [
More specifically, we evaluate the recommendation performance
based on the rating prediction and top-10 recommendation tasks.
In the rating prediction task, we use mean absolute error (MAE)
as evaluation metric. We also further examine the statistical
difference of MAE among different algorithms based on paired
ttest at a 95% confidence level. In the top-10 recommendation, We
adopt precision as the relevance metric and Normalized Discounted
Cumulative Gain (NDCG) [
First of all, we present our results based on the rating prediction task in Figure 3(a). We use CAMF_CU to denote the original approach without emotional or user regularization terms. Our approaches introduced in this paper are built upon CAMF_CU approach and they can be generated based on either domEmo or endEmo. We evaluate the performances by them individually. We use "domEmo_B" to represent the model using domEmo for emotional regularization, i.e., cm denotes the emotional state in domEmo in Equations 4. By contrast, "domEmo_B, u" is used to denote the finer-grained model described in Equation 6 which contains both emotional and user regularization terms. Accordingly, "endEmo_B" and "endEmo_B, u" are the two recommendation models by using endEmo to generate the regularization terms.
Based on the results shown in Figure 3(a), our proposed approaches only using the emotional regularization term can help obtain lower MAE. All of these improvements are statistically significant based on the paired t-test. When we try to use both emotional and user regularization terms, it is able to further improve prediction accuracies. However, the improvement by endEmo_B,u fails the paired t-test compared with the endEmo_B approach. The best performing model in the rating prediction task is domEmo_B,u, where we apply emotional and user regularization terms at the same time, and these regularization terms are generated based on the emotional reactions by domEmo.
We show the top-10 recommendation results based on precision and NDCG in Figure 3(b). The bars present results based on precision at top-10 recommendation, the curve tells the results in NDCG. We can observe similar patterns shown in the rating prediction task: first, we see that the CAMF_CU models with our regularization terms are able to outperform the original CAMF_CU approach in both precision and NDCG. This finding confirms that incorporating emotional regularization terms inferred from users’ emotional reactions is helpful to improve performance of contextaware recommendation.
Furthermore, we can observe the finer-grained model with additional user regularization term contributes to obtain more improvements. For example, domEmo_B,u works better than domEmo_B (19.6% improvement on precision, and 18.2% on NDCG), and endEmo_B, u outperforms endEmo_B (30.1% improvement on precision, and 18.5% on NDCG).
As mentioned before, the number of selected neighbors in our models may impact the recommendation performance. We present the impact by the number of neighbors in the finer-grained CAMF_CU approaches with two regularization terms, as shown by the Figure 4. Simply, we vary the number of neighbors from 10 to 80 with an increment of 10 on each step. The best number of neighbors should be around 40 to 50 in this data set. It is essential to examine different number of neighbors to find out the optimal selection for each recommendation model.
Finally, the experimental results help us identify that the domEmo is more useful and effective to be adopted than using endEmo. This finding is consistent with our previous analysis on the unusual cases shown in Figure 2. It makes sense since the emotional status during the process of movie watching may be very different than their emotions at the end. For example, a user may feel horrible if he or she is watching an adventure movie, but finally he or she might feel happy since it is a good movie. 5.3
Why emotional reactions or expressions can be reused to improve
the recommendation performance? As we mentioned before, one
of the potential reasons is that the different emotional reactions
are caused by the traits in different user personalities – users may
express their emotional states or reactions in different ways. It
has been well studied that the emotional expression has strong
correlations with user personality, especially in the areas of
psychology and social science. For example, the correlation between
emotional expression and personality can be used to assist health
care [
In this paper, we believe that users may place similar ratings even if they may have different emotional reactions or expressions. We propose to incorporate the corresponding regularization terms in the CAMF_CU approach to assist context-aware recommendation. Our findings based on the experimental results over the LDOSCoMoDa movie data demonstrate that modeling user’s emotional reactions is helpful to improve recommendation performance. The results also reveal that domEmo is better than endEmo to generate the regularization terms in this data set. And the finer-grained model by additionally incorporating user regularization is able to offer further improvements.
One of our future work is to incorporate these regularization
terms based on different emotional reactions to more context-aware
recommendation models. It is interesting to examine the similar
approach in the similarity-based context-aware recommendation
algorithms [