Online news reading has become very popular as the web provides access to news articles from millions of sources around the world. As a specific application domain, news recommender systems aim to give the most relevant news article recommendations to users according to their personal interests and preferences. Recently, a family of models has emerged that aims to improve recommendations by adapting to the contextual situation of users. These models provide the premise of being more accurate as they are tailored to satisfy the continuously changing needs of users. However, little attention has been paid to the emotional context and its potential on improving the accuracy of news recommendations. The main objective of this paper is to investigate whether, how and to what extent emotion features can improve recommendations. Towards that end, we derive a large number of emotion features that can be attributed to both items and users in the domain of news. Then, we devise state-of-the-art emotion-aware recommendation models by systematically leveraging these features. We conducted a thorough experimental evaluation on a real dataset coming from news domain. Our results demonstrate that the proposed models outperform state-of-the-art non-emotion-based recommendation models. Our study provides evidence of the usefulness of the emotion features at large, as well as the feasibility of our approach on incorporating them to existing models to improve recommendations.
• Information systems → Recommender systems; Sentiment analysis; • Computing methodologies → Neural networks. news recommender systems, contextual information, emotion features
Recommender systems (RS) have widely and successfully been
employed in domains as diverse as news and media, entertainment,
e-commerce and financial services, to name a few. The main
utility of such systems is their ability to suggest items to users that
they might like or find useful. Traditionally, research on
recommendation algorithms has focused on improving the accuracy of
predictive models based on a combination of descriptive features
of the items and users themselves (e.g., user behavior, interests and
preferences) and the history of a user’s interactions with the items
through ratings, reviews, clicks and more [
While emotions can be manifested in various ways, we focus on
emotions expressed in textual information that is associated with
items or users in the system. For example, the content of a news
article, the content of an online review or the lyrics of a song are
good examples of textual information directly associated with an
item’s emotional context. On the other hand, the emotional profile
of a user can be determined through explicit or implicit feedback of
users to items. Explicit feedback, such as providing ratings and/or
submitting reviews to items, can represent an accurate reflection
of a user’s opinion about the item, but it is considered an intrusive
process that disrupts the user-system interaction and negatively
impacts user experience [
To further motivate this, consider Figure 1 that illustrates the emotional profiles of two users, U 1 and U2, based on eight basic emotions, expressed in articles read by them over a period of three months. One can notice that emotions of sadness and fear are mostly expressed in the articles read by U1 while other emotions, such as joy are less expressed. In addition, one can observe trends such as the expression of anger increasing over time. On the other hand, for U2, the emotions of joy and trust are mostly expressed and other emotions, such as disgust are less expressed. Moreover, emotions of fear and anticipation are increasingly expressed in the articles read by this user. Although, the emotional tone derived from news articles read by a user cannot justify the personality and state of mind of the user, it can be considered as the taste or preference of the user, where it shows the type of articles they are more interested in. Inspired by these observations, recent advancement in methods for emotion detection and the success of emotion-aware recommendation algorithms, the main motivation of our research is to investigate whether, how and to what extent emotion features can improve the accuracy of recommendations.
The Problem. More formally, the recommendation task can be described as follows. Let a set of users U = {1, 2, ..., } and a set of items I = {1, 2, ..., }. Let us also assume that each user has already interacted with a set of items I ⊆ I (e.g., consumed news articles). Then, the problem is to accurately predict the probability , with which a user ∈ U will like item ∈ I \ I . The task can also take the form of recommending a set I ⊆ I \ I of items that the user will find most interesting (top- recommendations). For example, in the news domain, the task is that of recommending an unread article.
Challenges & Approach. In order to evaluate the importance of
the emotional context to recommendations, we had to incorporate
emotional features [
Non-Emotion-based Features item-related user-related
Emotion-based Features item-related user-related user-specific properties item-specific properties user-item interactions Feature Generation Feature Extraction Feature Selection
Model Training
Blending
&
Ensemble
Item
Predictions Contributions. The major contributions of this paper are as follows: • We systematically identify, extract and select the most relevant emotion-based features for use in news recommendation models. These features are associated with both items (e.g., news articles) and users (e.g., readers). • We devise a number of state-of-the-art models for generating recommendations that incorporate the additional emotion features. These models include variations of gradient boosting decision trees, deep matrix factorization methods and deep neural network architectures. In addition, we use ensembling methods to increase the predictive performance by blending or combining the predictions of multiple constituent models. • We propose EmoRec, an emotion-aware recommendation model, which demonstrates the best accuracy performance in news recommendation task. EmoRec itself is an ensemble model. • We conduct a thorough experimental evaluation on a real dataset coming from news domain. Our results demonstrate that the emotion-aware recommendation models consistently outperform state-of-the-art non-emotion-based recommendation models. Our study provides evidence of the usefulness of the emotion features at large, as well as the feasibility of our approach on incorporating them to existing models to improve recommendations.
Prior research has found a range of features to be useful in the
context of news recommender systems, such as user location [
In the context of recommender systems, one of the earliest
works [
While emotions as features have been studied in movie
recommendations [
Emotion in news articles has been studied for categorizing news
stories into eight emotion categories [
This section describes the feature extraction procedure which is utilized in our proposed framework. The features are grouped into two main categories: (i) emotion-based features for items and users, and (ii) non-emotion-based features for items and users. 3.1
The main objective of this paper is to improve the performance of recommender system by leveraging the user/item emotion features.
Figure 3 shows an example of textual content of items (i.e., an
article) in news domains. As it can be observed, there are several
words such as win and gratifying, expressing the emotion of
happiness. Moreover, interjections such as yay and oh can be indicators
of diferent emotions [
Number of Emotion Words: This feature represents the number of words in an emotion lexicon (i.e., WordNet-Afect, see Table 1) that occur in the item (i.e., news article) more than once.
Ekman’s Emotion Label: We count the number of emotion
words occurring in the text document for each emotion type
(Ekman’s six emotion categories [
Mixed Emotions: This feature indicates whether an item has more than one document-level emotion labels based on Ekman’s emotion model (i.e., if two or more emotions have the highest score, this feature is valued at 1, otherwise 0). Since the initial annotation efort (previous feature) illustrated that in many cases, a sentence can exhibit more than one emotion, we have an additional category called mixed emotion to account for all such instances.
Sentiment Feature: The text is classified into three categories:
positive, negative and neutral. We utilize the approach introduced
in [
Interjections: This feature counts the number of interjections in a document. A short sound, word or phrase spoken suddenly to express an emotion, e.g., oh, look out!, ah, are called interjections1. Our preliminary analysis found that interjections were common in quotes in news articles, which can be detected for potential emotions.
Capitalized Words: This feature counts the number of words
in a document with all uppercase characters. People use capital
1List of interjections derived from: i) https://surveyanyplace.com/
the-ultimate-interjection-list, ii) https://7esl.com/interjections-exclamations,
and iii) https://www.thoughtco.com/interjections-in-english-1692798
words to express an emotion [
Punctuation: Two features are included to model the
occurrence of question marks and exclamation marks repeated more
than two times in a document. Using punctuation can clarify the
emotional content of the texts that are sometimes easy to miss [
Grammatical Markers and Extended Words: This feature
counts the number of times words with a character repeated more
than two times (e.g., haaappy or oh yeah!!????) [
Plutchik Emotion Scores: First, we measure the semantic
relatedness score between a word in the text and an emotion
category in the NRC lexicon (see Table 1) as follows [
As we do not have access to users’ explicit emotion towards items, we develop users’ implicit emotional profile based on their historical interactions with items. By computing the emotion profile of the items with which a user is interacting, we derive the emotional taste of the user over that period of time over the set of items.
User Emotions Across Items: We determine the emotion score (i.e., Plutchik’s emotion scores) for the last accessed item before subscription as well as for the last 20 items accessed by the user. Then, we pick the top 3 frequent emotions.
User Emotions Across Categories: We determine the emotion of categories of items (e.g., sports in news domain) accessed by a user by counting the number of items assigned to an emotion in a specific category, with the most frequent emotion considered as the emotion of the category. The feature is calculated for the whole history of the user. 3.2
Non-emotion-based features can also be classified into item-based and user-based features. 3.2.1 Item Non-Emotion-based Features.
Item Topic: We extract topics in the article using Latent
Dirichlet Allocation (LDA) [
Topic Label: We use lda2vec [
TF-IDF: This feature represents items as n-grams (unigram,
bigram, trigram) with the TF-IDF weighting approach [
Coherence: We first calculate the cosine similarity scores between all pair of words in an item using word2vec pre-trained word vectors2, and then record average of similarity scores, standard deviation of similarity scores, the lowest score that is higher than the standard deviation, and the highest score that is lower than the standard deviation as four features.
Potential to Trigger Subscription: This feature represents the
total number of times the item was requested right before a paywall
was presented to a user who subsequently made a subscription [
Visit Count: We calculate the average number of items (articles) accessed by a user per visit. A visit is terminated if a user is inactive for more than 30 minutes.
User Spent Time: Two features are represented. One is the average time the user spent per item, and the other is the average time the user spent per visit.
User Interest in Subcategory: This feature represents the empirical probability of subcategory given a user and a category denoted as ( |, ).
For example, (election|, politics) can be determined by the total number of articles the user read on election over the total number of articles that the user read on politics. In our experiments, the categories and subcategories were provided with the dataset and we consider only the top 50 most frequently visited subcategories for this feature.
User Latent Vector: We calculate the latent vector for each user
based on matrix factorization introduced in [
One of the critical steps after feature extraction is to select important
features for recommendation. Table 2 reports the most important
features according to gain importance score for the news data set.
We evaluate feature importance by averaging over 10 training runs
of a gradient boosting machine learning model XGBoost [
In this section, we introduce a tailored structure of an Emotionaware Recommender System Model (EmoRec) for personalized recommendation. Our final model is an ensemble model of three models leveraging both emotion/non-emotion-based features. We describe the structure of the proposed model and the training methods next. 4.1
Model 1 (Boost Model): Gradient Boosting Decision Tree (GBDT)
methods are among the most powerful machine learning approaches
which have been efectively used in many domains [
XGBoost uses pre-sorted/histogram-based algorithm to compute the best split while CatBoost uses ordered boosting, a permeation based algorithm, to learn the weak learners efectively. Moreover, XGBoost uses one-hot encoding before supplying categorical data, but CatBoost handles categorical features directly. We train both models individually (three base models for each). The final model output (i.e., probability that a user is interested in an item) is the 3Variance refers to the sensitivity of the learning algorithm to the specifics of the training data (e.g., the noise and specific observations). combination of all base models outcomes: 6 Õ (3) where is the probability that the user is interested in the item according to base model and is the weight of base model learned by XGboost/Catboost.
Model 2 (Deep Neural Network (DNN)): Figure 4 shows our
proposed Deep Neural Network architecture for leveraging the
emotion features (and other commonly available features) for the
recommendation purpose. The input is divided into four groups [
Model 3 (Deep Matrix Factorization (Deep MF)): Inspired by
the models proposed in [
Ensemble/Blending Model: The final model EmoRec was the
weighted average of the three models’ predictions. We use
NelderMead Method [
In this section, we introduce the data, evaluation protocols and the specific configurations used in our experiments. Our experiments are conducted on a real-world news dataset. The Globe and Mail is one of the major newspapers4 in Canada. We use the data spanning from January to July 2014 (a 6-month period) in our experiments where the data in the first four months were used for training, and the last two months for testing. The dataset contains information for 359,145 articles in total and 88,648 users in total, out of which 17,009 became subscribers during this period, and 71,639 were non-subscribers. Every time a user reads an article, watches a video or generally takes an action in the news portal, the interaction is recorded as a hit. Typically, a hit contains information like date, time, user id, visited article, special events of interest like subscription, sign in, and so on. 5.2
We use F-score to measure the predictive performance of a recommender system. For each user in the test data set, we use the original set of read articles in the test period as the ground truth, denoted as . Assuming the set of recommended news articles for the user is , precision, recall, and F-measure are defined as follows:
Precision = | ∩ | , Recall = | ∩ |
| | | |
• Single Boost Model: We run XGBoost and Catboost separately to make predictions and collect the average of their F-scores. • Boost Blend: This is the 6-model ensemble described in Model 1 in Section 4.1. • Deep MF: This is the deep matrix factorization model described in Section 4.1. • Single DNN model: We run the DNN model for 5 times with the same hyperparameters but diferent random seeds and collect the average result over 5 runs. • DNN Ensemble: An ensemble of 5 DNN models with diferent hyperparameters (e.g., diferent learning rates, etc.) is run 5 times each with a diferent random seed. The average result over the 5 runs is collected. • Boost Blend + Deep MF: This is an ensemble consisting of
Boost Blend and Deep MF. • Boost Blend + DNN Ensemble: This an ensemble consisting of Boost Blend and DNN Ensemble. • Deep MF + DNN Ensemble: This is an ensemble consisting of
Deep MF and DNN Ensemble. • Boost Blend + Deep MF + DNN Ensemble: an ensemble consisting of Boost Blend, Deep MF and DNN Ensemble.
We train each of the above models using the training data of our data set and use the trained model to make recommendations by predicting a user’s interest in an item in the test data. Table 3 shows the results (in F-score) of using these recommendation methods with and without emotion features on the news data set, where the whole set of emotion features described in Section 3.3 is used in the results for "All", while none of the emotion features is used in the results for "Non-Emo". As can be seen, adding emotion features improves the predictive performance for all the recommendation methods. Among the single recommendation models (i.e., Single Boost Model, Deep MF and Single DNN Model), Deep MF performs the best. The results also show that ensemble methods perform better than single/component models. The best performance is produced by the largest ensemble (i.e., Boost Blend + Deep MF + DNN Ensemble). We refer to this best-performing model as our EmoRec model. 5.4
We also compare our EmoRec model with the following three stateof-the-art recommendation methods with well-tuned parameters (that is, the parameters are optimally tuned to ensure the fair comparison). The objective is to investigate whether emotion features can smarten up these recommender systems. A brief description of these three models is as follows:
Basic MF : This is the simple matrix factorization model where
used for discovering latent features between two entities (i.e., user
and articles) [
FDEN and GBDT : an ensemble of diferent models, including
Field-aware Deep Embedding Networks and Gradient Boosting
Decision Trees [
Truncated SVD-based Feature Engineering: a gradient boosted
decision trees model with truncated SVD-based embedding
features [
The results are illustrated in Table 4, which shows that emotion features can also improve the recommendation performance of these three state-of-the-art baselines. In addition, our EmoRec model performs significantly better than these three baselines in both cases of using emotion features and not using emotion features. 5.5
Table 5 presents the results of a feature ablation study in order to further understand the efect of individual emotion features used in EmoRec. In each run of this study, we keep all the features except one type of emotion features. The results indicate that removing Plutchik emotion scores (item feature), User emotions across categories and User emotions across items (user features) lead to considerable decline in the performance. It also shows that our model is able to capture useful implicit user emotion efectively.
To further validate the efectiveness of the top emotion features as learned from our experiments, we run a further experiment incorporating only the top three emotion features (i.e., Plutchik emotions, User emotions across categories, and User emotions across items) on six state-of-the-art recommendation models. As the results in Table 6 show, only using these three emotion features can also improve the recommender systems, with Basic MF showing the most gain. 6
Motivated by the recent development in emotion detection methods (in textual information), we considered the problem of leveraging emotion features to improve recommendations. Towards that end, we derived a large number of emotion features that can be attributed to both items and users in news domain and can provide an emotional context. Then, we devised state-of-the-art non-emotion and emotion-aware recommendation models to investigate whether, how and to what extent emotion features can improve recommendations. To the best of our knowledge, this is the first attempt to systematically and broadly evaluate the utility of a number of emotion features for the recommendation task. Our results indicate that emotion-aware recommendation models consistently outperform state-of-the-art non-emotion-based recommendation models.
Model Basic MF Boost Blend FDEN and GBDT Deep MF Truncated SVD-based DNN Ensemble Furthermore, our study provided evidence of the usefulness of the emotion features at large, as well as the feasibility of our approach on incorporating them to existing models to improve recommendations.
As a more tangible outcome of the study, we proposed EmoRec, an emotion-aware recommendation model, which demonstrates the best predictive performance in news recommendation task. EmoRec itself is an ensemble model combining three models (Boost Blend + Deep MF + DNN Ensemble). It significantly outperforms other state-of-the-art recommendation methods evaluated in our experiments. We also evaluated the proposed emotion features individually. Among the emotion features examined, the Plutchik emotion scores of items (obtained by computing PMI scores between words) and user emotion profiles (based on the emotion scores of the items that the user accessed) are the most important.
Employing emotional context in recommendations appears to be
a promising direction of research. While the scope of our current
study is limited to emotions extracted by textual information, there
is evidence that emotions can be extracted through other means of
communication, such as audio and video, or other cues [
This work is funded by Natural Sciences and Engineering Research Council of Canada (NSERC), The Globe and Mail, and the Big Data Research, Analytics, and Information Network (BRAIN) Alliance established by the Ontario Research Fund Research Excellence Program (ORF-RE). We would like to thank The Globe and Mail for providing the dataset used in this research. In particular, we thank Gordon Edall and the Data Science team of The Globe and Mail for their insights and collaboration in our joint project.