Abstract

Personalized news recommender systems are widely deployed to filter the information overload caused by the sheer amount of news produced daily. Recommended news articles usually have a sentiment similar to the sentiment orientation of the previously consumed news, creating a self-reinforcing cycle of sentiment chambers around people. Wu et al. introduced SentiRec - a sentiment diversity-aware neural news recommendation model to counter this lack of diversity.

1. Introduction

Content-based recommenders usually recommend items to users similar to items they have liked in the past [ 1 ]. Also, recent well-performing neural news recommendation methods follow this principle. They model users based on their previously browsed news articles and, in turn, rank candidate news articles based on a relevance score considering the user model [ 2 ]. However, such approaches are prone to a lack of diversity. Especially since news with negative sentiment is more often clicked than positive ones, diversifying the sentiment is essential in news recommendations [ 3 ].

Taking all this into account, Wu et al. [ 3 ] introduced SentiRec, a sentiment diversity-aware neural news recommendation method. They learn sentiment-aware news representations by considering the content of the news and jointly training the recommendation model together with an auxiliary sentiment prediction task. Users are modeled by their previous clicked and non-clicked (i.e., seen but not clicked) news articles. The SentiRec approach regularizes and thus increases sentiment diversity by penalizing candidate news with similar sentiment compared to the users’ overall sentiment orientation. In both sentiment regularization and sentiment Perspectives on the Evaluation of Recommender Systems Workshop (PERSPECTIVES 2022), September 22nd, 2022, prediction tasks, VADER [ 4 ], a rule-based sentiment-analyzer, is utilized to determine the sentiment polarity score as the label.

In this work, we reproduce SentiRec without having access to the original source code or dataset. Our request for access to the original source code and data set has not been answered yet. Thus, we re-implement SentiRec from scratch and use the Microsoft MIND [ 2 ] dataset (same data source but diferent subset as in the original work) for our experiments. We evaluate our reproduction from diferent perspectives, namely i) efectiveness, ii) user-centric sentiment diversity, iii) intra-list sentiment diversity, and iiii) topical diversity. In our first evaluation perspective we aim to compare efectiveness trends from the original paper with our implementation and study:

RQ1 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning efectiveness? In contrast to the original work, our reproduction does not significantly outperform the baselines, which might be due to the dataset diferences, highlighting the shortcomings of SentiRec regarding generalizability. We also employed a pre-trained neural sentiment analyzer (BERTSA1) in addition to the rule-based one (VADER-SA [ 4 ]). When using BERT-SA, we observe no gains in recommendation performance and sentiment diversity compared to the VADER-SA setting. Our next evaluation perspective is user-centric sentiment diversity, as defined in the original paper; thus, we investigate:

RQ2 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning user-centric sentiment diversity? Opposing the original paper, we could not achieve the best user-centric sentiment diversity results by outperforming the random model while maintaining the best efectiveness. Moreover, we demonstrate that some baselines already reach suficient user-centric sentiment diversity and significantly outperform SentiRec, (again) highlighting the lack of generalizability. While the original paper focuses on user-centric sentiment diversity by comparing the recommended list of news to the user’s interaction history, our third perspective focuses on sentiment diversity between news articles within a recommendation list, i.e., intra-list sentiment diversity. Thus we investigate:

RQ3 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning intra-list sentiment diversity? In contrast to the user-centric evaluation and although been penalized for user-centric sentiment similarity, our reproduction significantly outperforms most baselines if intra-list sentiment diversity is considered. This calls for a discussion on whether to employ a user-centric or an intra-list diversification and further investigations. While the original paper only considers sentiment diversity, we also analyze topical diversity, and thus in our final evaluation perspective, we study:

RQ4 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning user-centric and intra-list topical diversity? The user-centric topical diversity compares the user’s interaction history to the recommendation list. We demonstrate that the baselines already reach significantly better user-centric topical diversity than our Sentirec reproduction - highlighting the tradeof between diferent objectives. 1https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english In intra-list topical diversity, our reproduction reaches comparable results to the baselines (taking aside the random model).

The contributions of this work are as follow: • We reproduce SentiRec [ 3 ] without having access to the original source code and dataset.

Instead, we re-implement SentiRec from scratch and use the MIND [ 2 ] dataset. Although our implementation shows similar trends, we fail to reproduce the original findings, which might be caused due to dataset diferences. In particular, the baselines in our experiments already show decent recommendation and sentiment diversity performance. • We propose extending the experiment by using a pre-trained neural sentiment analyzer instead of a rule-based sentiment analyzer. However, we observe no gains in efectiveness nor sentiment diversity. • We propose extending the experiment by considering user-centric topical diversity and intra-list topical and sentiment diversity. While the baselines outperform our reproduction if user-centric and intra-list topical diversity is considered, it significantly outperforms the baselines in intra-list sentiment diversity. • We publish the first open implementation of SentiRec for the community at: https://github.com/MeteSertkan/newsrec

2. Background

The way how items are presented often influences the decision behavior of users [ 5 ]. Thus, when interacting with news articles, also their textual style plays an essential role [ 3, 6, 7 ] besides semantic or syntactic properties. However, these features are hard to engineer by hand. Recently, deep learning architectures have been increasingly used in recommendation scenarios [ 8 ]. These architectures have proven highly beneficial when capturing various patterns (e.g., user sessions, structure in pictures or language) or dealing with high complexity (e.g., multi-modal data, very dynamic settings, etc.). They usually follow an end-to-end feature extraction paradigm, where the recommendation model and the representation model (i.e., item and user encoder) are trained simultaneously. Thus handcrafted heuristics are avoided [ 9 ]. The trend has also reached the new recommendation domain. For example, NAML [ 10 ] uses attention networks to incorporate diferent views of a news article, e.g., title, abstract, category, etc., into the news; LSTUR [ 11 ] captures the short-term interest of users by applying GRU on recently clicked items and long-term interest by considering a user’s whole history track; and NRMS [ 12 ] uses multi-head self-attention in combination with additive-attention to model news articles, and in turn, users. However, by only considering the content of the users’ previous interactions, they are prone to recommend in a ”more of the same” way, and consequently, they might lack diversity. Therefore, we study news diversification and, in particular, sentiment diversification. In this work, we re-implement, extend, and analyze SentiRec [ 3 ]. SentiRec learns sentiment-aware news representations using an auxiliary sentiment prediction task and introduces a sentiment regularization method to obtain sentiment-diverse recommendations. While sentiment-aware recommendations have been studied in the tourism domain [13, 14], movie domain [15, 16], and e-commerce [17, 18], to name a few, less attention has been paid to sentiment-aware recommendations in the news domain and nor to sentiment diversification.

News Representation

Sentiment

Score Sentiment

Predictor Transformer ...

...

Word Embedding t1

tM News Encoder Sentiment Monitor

Click Score

5

Click Predictor

User Encoder ...

3

DN News Encoder

Sentiment Analyzer

News Encoder

SAennatilmyzeenrt ... Encoder

News

Sentiment

Analyzer DC 2

D1 1

Candidate News

...

Browsed News

3. Methods

3.1. SentiRec

SentiRec aims to optimize recommendation accuracy and sentiment diversity, which naturally leads to a trade-of between accuracy and diversity. The overall task is to rank candidate items based on a user’s history of previous items. Given for a user a history set of previously browsed news articles [ 1, …, ] with sentiment polarity scores [ 1, …, ], the aim is to rank a set of candidate news articles [ 1, ..., ] (with sentiment polarity scores [ 1, …, ]) by assigning each article a score i.e., [ 1̂, …, ̂ ]. In particular, SentiRec seeks for sentiment diversity in the recommendation list. Higher diversity is achieved if top-ranked news articles have diferent sentiment polarity scores than the overall sentiment orientation =̄ ([ of the user’s previously browsed news. In the following we describe the diferent SentiRec 1, …, ]) components as shown in Figure 1.

1 News Encoder. The task of the news encoder is to find a representation of candidate news as well as representations [ 1, …, ] of browsed news [ 1, …, ] by taking their title as input. It consists of an embedding layer followed by a transformer layer to obtain a representation out of a sequence of terms. Since no details about the transformer layer were given, we follow the architecture of the closely related NRMS [ 12 ] model. Thus, we use multi-head self-attention for contextualization and additive-attention to obtain a unified embedding out of the contextualized word embeddings. The news encoder is jointly trained with an auxiliary sentiment prediction task in order to infuse sentiment awareness to the news representation. The sentiment score ̂ is predicted using a linear layer, i.e, =̂ × + , where and are learnable parameters and is the news representation. As loss function the mean absolute error between predicted sentiment scores ̂ and the sentiment scores determined by the sentiment analyzer is used as follows : ) ( ̂ +) + ∑=1 ( ̂ − , ) ) 2 Sentiment-Analyzer. Given the title of a news article, the sentiment analyzer determines the sentiment polarity score ranging in [ -1, 1 ], which is considered as the sentiment label of the respective news article. The original paper uses VADER [ 4 ] (a rule-based method) as sentiment analyzer (VADER-SA). In addition, we also study a pre-trained neural sentiment analyzer2 (BERT-SA).

3 User Encoder. The user encoder gets the sentiment-aware representations of the previously browsed news, i.e., [ 1, …, ], as input and uses a transformer layer (i.e., multi-head self-attention followed by additive attention according to NRMS [ 12 ]) to obtain a representation of the user.

4 Click Predictor. The click predictor uses the dot-product between user and candidate embedding, i.e, , to determine a click score .̂

5 Sentiment Monitor. The sentiment monitor observes to what extent the sentiment polarity score (obtained by the sentiment analyzer) of a candidate news article diverges from the users’ overall sentiment orientation =̄ ([

1, ..., ]) (i.e., the mean sentiment polarity score of the users browsing history). This diversity in sentiment is measured by = (0, ̄ larger values of indicate less sentiment diversity. The sentiment diversity score is further used to regularize and steer the model into a more sentiment diverse direction. Following loss )̂ , where function is used for this purpose: ℒ = 1 || ∈

∑ where is the training set and the sentiment diversity score of the -th sample.

Negative sampling is used in order to create a labeled dataset for the recommendation task. For each clicked news in a user impression, non-clicked samples from the same impression are randomly selected. The recommendation loss is the negative log-likelihood of the clicked samples and is defined as follows: (1) (2) (3) (4) where ̂ + is the click score of -th clicked news and ,̂− the click score of the -th sample of the corresponding negative samples, and is the training set. The final loss function brings all three losses, i.e., recommendation loss, sentiment prediction loss, and sentiment diversity loss, together as follows: ℒ = ℒ + ℒ + ℒ where and are hyperparameters controlling the influence of the sentimen prediction loss and sentiment diversity loss respectively. 2https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english

3.2. Evaluation Perspectives

We evaluate our reproduction from five diferent perspectives: efectiveness, user-centric sentiment diversity, intra-list sentiment diversity, user-centric topical diversity, and intra-list topical diversity. Note, in contrast to the intra-list diversity measures, the user-centric measures assess diversity in relation to the user’s previous news consumption. We compare the results of our reproduction against all baselines and our extensions, using paired t-test with Bonferroni

We evaluate efectiveness using

, .

User-Centric Sentiment Diversity. We evaluate user-centric sentiment diversity using the correction [19, 20].

Efectiveness. sentiment alignment metrics follows:

and @ , introduced by WU et al. [ 3 ], which is defined as is the sentiment polarity score of the news article ranked at position in this list; and ̄ is the overall sentiment orientation of the corresponding user. Hence, the closer top-ranked candidates’ sentiment to the users’ overall sentiment orientation, the higher the sentiment alignment metrics. Ergo, lower sentiment alignment scores indicate more sentiment-diverse recommendations.

Intra-List Sentiment Diversity (not included in the original paper). As the sentiment polarity score of a news article is only one scalar, we compute the intra-list sentiment diversity by averaging the absolute diference of sentiment polarity scores and between each news pair in the Top-K list of recommended candidate articles: = ( , ), @ = ( @ , ) (5) (6) (7)

2 | − | representation news articles.

The intra-list sentiment diversity score lies between 0 and 1, with 0 being maximal divers.

User-Centric Topical Diversity (not included in the original paper). We consider the news articles’ categories (e.g., sports) and subcategories (e.g., soccer) to compute topical diversity. We represent a (sub)category of a news article with a 1-hot-encoding. We compute the user’s category representation by summing up all browsed news category representations. Similarly, we compute the recommendations list’s category representation @ by summing up the category representations of the recommended top-K candidate news articles. We then measure

by taking cosine similarity between and @ . This leads to a measure between 0 and 1, with 0 being maximal divers. Similarly, we measure with the diference being computing a weighted average of all candidates’ category representations to obtain a of the recommendation list, where the weight is the rank of corresponding

Intra-List Topical Diversity (not included in the original paper). We again represent a (sub)category of a news article with a 1-hot-encoding. We measure the intra-list topical diversity of the recommendation list by computing the average pairwise cosine similarity between the 1-hot-encoded category representations of the recommended top-k news articles. This leads to a measure between 0 and 1, with 0 being maximal divers.

2

4. Experimental Setting

Dataset. The dataset of the original paper is constructed from MSN News3 logs collected from 0ctober 31, 2018, to January 29, 2019, but has not been open-sourced, and our access request has not been answered yet. Thus, we use the MIND [ 2 ] dataset - specifically the MIND-small 4 version - in our experiments, as it stems from the same source. It was randomly sampled from 50K users (with at least five clicks) during six weeks, from October 12 to November 22, 2019, where the first five weeks are for training and the last week for testing. One sample is composed of a timestamp, the user-id, a list of chronologically ordered news-ids representing the user’s click history, and a list of shufled candidate news-ids with corresponding labels (i.e., 1 for clicked and 0 for seen but not clicked). Detailed statistics of the datasets are summarized in average about seven times fewer positive interactions per user (seven clicks vs. 49) than the

SentiRec dataset.

0.0001 AUC and patience of 5. Note that we use 300-dimensional Glove embeddings [21] in all models to initialize the word embedding layer and NLTK [22] word tokenizer for tokenization. Further, we limit the number of browsed news in each impression to 50 and the title length to 20 terms (smaller sequences are zero-padded).

Parameter Settings. We set the negative sampling ratio K to 4. We apply 20% dropout to the word embeddings. We use multi-head self-attention with 15 attention heads followed by an additive-attention layer with a 200-dimensional query vector. We use the ADAM [23] optimizer with a learning rate of 0.0001 and a batch size of 128. For the VADER-SA based model ( ) we set = 0.4 and = 10 and for the BERT-SA based model ( ) we set = 0.4 and = 1 .

Baselines. We compare the reproduced and adapted models against following baselines suggested by the dataset providers [ 2 ]:

LSTUR [ 11 ] (not included in the original paper) - Neural news recommender capturing users’ long- and short-term interests. We initialize the GRU network with the user embedding. We set the masking probability of the users’ long-term interests to 50%. We apply 20% dropout to the word embeddings. The negative sampling ratio K is set to 4. For the CNN, we set the number of filters to 300 and the window size to 3. We use a 200-dimensional query vector for the additive-attention layer. We use the ADAM [23] optimizer with a learning rate of 0.0001 and a batch size of 256.

NAML [ 10 ] (not included in the original paper) & (adaptation of NAML as in the original paper) - Neural news recommender incorporating multiple views (i.e., title, category, and abstract) into the news representation. We limit the abstract length to 50 terms. We apply 20% dropout to the word embeddings. We set the category embeddings dimension to 100. The number of CNN filters is set to 400 and the window size to 3. We use 200-dimensional query vectors in the additive-attention layers. The negative sampling ratio K is set to 4. We use the ADAM [23] optimizer with a learning rate of 0.0001 and a batch size of 256. We also trained - a ”title only” version - as used in the original paper [ 3 ]. We obtained the same parameters as NAML without the need for category dimensions.

NRMS [ 12 ] - Neural news recommender which utilizes multi-head self-attention within both the news encoder and the user encoder. We use multi-head self-attention with 15 attention heads followed by an additive-attention layer with a 200-dimensional query vector. We apply 20% dropout to the word embeddings. We set the negative sampling ratio K to 4. We use the ADAM [23] optimizer with a learning rate of 0.0001 and a batch size of 128.

5. Results and Analysis

In this section, we present and analyze our results and answer our previously stated research questions. We investigate whether the reproduced models perform as described in the original paper and study:

RQ1 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning efectiveness? We compare the recommendation performance (i.e., , , @5 , and @10 ) of the reproduced model (i.e., ) against the baselines (i.e., [ 11 ], & [ 10 ], [ 12 ], and ), which is summarized in rows 1-6 of Table 2. Opposing the original work, our sentiment reproduction does not significantly outperform all baselines concerning recommendation efectiveness. Moreover, it performs similarly to the closely related baseline. Furthermore, utilizing a pre-trained neural sentiment analyzer instead of the rule-based one does not yield performance gains (compare rows 6 to 7 in Table 2).

RQ2 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning user-centric sentiment diversity? We investigate sentiment diversity by comparing the sentiment alignment scores (i.e., , Model

AUC

MRR Comparing user-centric sentiment and topic alignment (i.e., alignment scores indicate better diversity. Subscripts V (VADER-SA) and B (BERT-SA) indicate the used sentiment analyzer. Note, † indicates a statistically significant diference to

Model

VADER-SA Labels

BERT-SA Labels Comparing efectiveness (i.e., AUC, MRR, nDCG@5, and nDCG@10). Higher efectiveness scores indicate better performance. Subscripts V (VADER-SA) and B (BERT-SA) indicate the used sentiment analyzer. Note, † indicates a statistically significant diference to at alpha 0.05. .4570† .3744† .4220† .4353 .4328 .4414† .5047† .4270† .4867† .4926† .4891 .4942† @5 , and @10 – lower scores indicate higher sentiment diversity) of our reproduced model, i.e.,

, and the baselines (see rows 1-6 in Table 3). In the original work [ 3 ], SentiRec outperforms all baselines in sentiment diversity - even the Random model - while maintaining the highest recommendation performance scores. We can not confirm these findings. Moreover, our results suggest that the baselines already perform well in all aspects, i.e., recommendation performance and sentiment diversity. In particular, we do not observe large margins in sentiment diversity as in the original paper While the original paper studies sentiment diversity with a user-centric focus, it is also essential to investigate sentiment diversity within a recommended list of news articles; thus, we ask: concerning intra-list sentiment diversity?

RQ3 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines We compute the intra-list sentiment similarity at cutof K, i.e., @ , by considering the pairwise diferences of news articles within a top K recommendation list. Table 4 (rows 1-7) @5,

Lower intra-list similarity scores indicate better diversity. Subscripts V (VADER-SA) and B (BERT-SA) indicate the used sentiment analyzer. Note, † indicates a statistically significant diference to Model

VADER-SA Labels

BERT-SA Labels @10 contrast to our user-centric diversity findings, where the baselines already exhibit decent performance, we observe that our reproduced model, i.e., , significantly outperforms most baselines concerning intra-list sentiment diversity. In comparison, the baseline shows the worst performance. Suggesting that additional modalities might foster user-centric sentiment diversity (see Table 3) but hurt intra-list sentiment diversity by recommending top K news articles with a rather higher sentiment similarity. Efectiveness and sentiment diversity are the emergent perspectives to evaluate SentiRec; in addition to those, we also focus on topical diversity and investigate:

RQ4 How does our reproduced SentiRec implementation compare to the MIND [ 2 ] baselines concerning user-centric and intra-list topical diversity? We adapt the user-centric sentiment alignment metrics and introduce user-centric topical alignment metrics, i.e., articles. Lower rize our analysis. The and @ , by considering the categorical membership of the news indicate higher diversity. The last three columns of Table 3 summa

model recommends the most topically diverse news articles to the users’ previously browsed news articles, except if the top 10 recommendations are considered, where the model excels. The

and the user-centric topical diversity than our reproduced baselines significantly reach better models while maintaining reasonable recommendation performance – demonstrating the competitiveness of the baseline models. If we consider intra-list topical diversity

@ (see Table 4 last two columns), which is defined by the pairwise categorical diferences within the recommendation list, the Model recommends the most diverse news articles. Our reproduction, models and is on par with the and baselines.

, outperforms the

6. Discussion

’s diversification approach. the by the Overall, we cannot confirm the findings of the original work, where they outperformed all baselines in efectiveness and user-centric sentiment diversity. We argue that the efectiveness and diversity discrepancies between the original SentiRec and our reproduction are due to dataset diferences highlighting the shortcomings of concerning generalizability. Our dataset contains five times more users and about 23K more news than the original paper; however, it contains relatively few positive feedback (i.e., clicks) and spans only over six weeks (compared to nine weeks). Thus, inherently more diverse behavior is contained in the used dataset than in the original paper. One might argue that the sentiment diversity issue in our sample is not as prevalent as in the sample of the original work. However, we demonstrate that baseline significantly outperforms our reproduction and gets close to the model’s performance. This highlights that there is room for improvement, which is not utilized As mentioned, the [ 10 ] model outperforms all other models (except the model) regarding user-centric sentiment diversity while maintaining comparable recommendation performance to our

reproductions. Besides the title of a news article, it also considers category, subcategory, and abstract. Thus, we reason that considering diferent modalities supports the diversification task. Note, in the original paper is fed with only one modality (i.e., title) - in this work denoted as .

Besides the user-centric view of sentiment diversity, we also analyze a more generic perspective, i.e., intra-list sentiment diversity. We demonstrate that our reproduction achieves an outstanding intra-list sentiment diversity, although optimized for user-centric sentiment diversity. Setting both perspectives alongside opens the room for the following question, which we will tackle in future work: Which view of sentiment diversity should we optimize while maintaining user satisfaction? Optimizing for the user-centric perspective is more conservative. This will rank news articles with an orthogonal sentiment to the overall sentiment of the user’s news consumption higher. Such an approach has a strong nudging power but might drop user satisfaction by recommending more the “unusual”. On the other hand, optimizing for the intra-list perspective is more relaxed by suggesting news articles with diferent sentiments. However, it bears the risk that users might still follow their previous behavior and consume, for example, only negative news articles.

Our final evaluation perspective, which the original work does not consider, is topical diversity. In particular, we consider categorical diferences between recommended news articles and the users’ browsed news, i.e., user-centric topical diversity and categorical diferences within the news articles in the recommendation list, i.e., intra-list topical diversity. In both measures, the

model achieves the most topically diverse recommendations. Setting aside the model, while in the user-centric perspective, our reproduction most baselines, in the intra-list perspective, it is on par or better than the baselines. With diferent sentiment distributions within news categories, we plan to analyze whether topical diversification already yields sentiment diversification and higher user satisfaction in future is outperformed by work.

7. Conclusion

This work aims to reproduce SentiRec [ 3 ] - a sentiment diversity-aware neural news recommendation model - without having access to the original source code and dataset. We re-implement SentiRec from scratch and make it publicly available. We use the MIND [ 2 ] dataset, which has the same source as the original paper, albeit a diferent time period. Overall, we can not confirm the significant findings of the SentiRec paper. The reproduced model does not outperform the random model in (user-centric) sentiment diversity while maintaining the best recommendation performance compared to the baselines as in the original work. Moreover, our results suggest that the baselines already perform well. In particular, the NAML [ 10 ] model delivers the most sentiment-diverse recommendations (w.r.t. to the users’ overall consumption behavior) apart from the random model while holding a comparable recommendation performance to all other baselines. We conclude that these discrepancies are due to dataset diferences high-lighting the shortcomings of SentiRec concerning generalizability.

In addition to the original paper, we also consider the topical diversity of the recommended list compared to the users’ previous user history. Similar to previously, we show that the baselines, particularly , significantly yield better topical diversity than our reproduced model.

In addition to a rule-based sentiment analyzer, as used by Wu et al. [ 3 ], we conducted our experiments with a pre-trained neural sentiment analyzer to study whether a neural model leads to better sentiment labels and thus to improved overall training performance. However, we do not observe improvements in recommendation performance or sentiment diversity.

While the original paper only focuses on sentiment diversity by comparing the users’ overall user history with the recommendation list (i.e., user-centric diversity), we also investigate the sentiment and the topical diversity between news articles within the recommendation list (intra-list diversity). In contrast to the user-centric evaluation, the intra-list evaluation shows that our reproduction significantly outperforms most baselines, while the strong baseline performs poorly.

We discuss our diferent evaluation perspectives (i.e., user-centric/intra-list sentiment and topical diversity). We plan to conduct ofline and online experiments to compare and combine them in future work. Furthermore, we plan to include other auxiliary information into the end-to-end recommendation model, such as emotion awarness and diversity. Ultimately, we want to create recommendation models that optimize for a broad range of goals and benefit society by more responsible recommendations.

Acknowledgments

This research is supported by the Christian Doppler Research Association (CDG), and has received funding from the EU’s H2020 research and innovation program (Grant No. 822670). Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 6389–6394. URL: https://www.aclweb.org/anthology/D19-1671. doi:1 0 . 1 8 6 5 3 / v 1 / D 1 9 - 1 6 7 1 . [13] H. Wang, Y. Fu, Q. Wang, H. Yin, C. Du, H. Xiong, A location-sentiment-aware recommender system for both home-town and out-of-town users, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, Association for Computing Machinery, New York, NY, USA, 2017, p. 1135–1143. URL: https://doi.org/10.1145/3097983.3098122. doi:1 0 . 1 1 4 5 / 3 0 9 7 9 8 3 . 3 0 9 8 1 2 2 . [14] P. Padia, K. H. Lim, J. Cha, A. Harwood, Sentiment-aware and personalized tour recommendation, in: 2019 IEEE International Conference on Big Data (Big Data), 2019, pp. 900–909. doi:1 0 . 1 1 0 9 / B i g D a t a 4 7 0 9 0 . 2 0 1 9 . 9 0 0 6 4 4 2 . [15] C. Orellana-Rodriguez, E. Diaz-Aviles, W. Nejdl, Mining afective context in short films for emotion-aware recommendation, in: Proceedings of the 26th ACM Conference on Hypertext & Social Media, HT ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 185–194. URL: https://doi.org/10.1145/2700171.2791042. doi:1 0 . 1 1 4 5 / 2 7 0 0 1 7 1 . 2 7 9 1 0 4 2 . [16] C. Musto, G. Rossiello, M. de Gemmis, P. Lops, G. Semeraro, Combining text summarization and aspect-based sentiment analysis of users’ reviews to justify recommendations, in: Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 383–387. URL: https://doi.org/10.1145/3298689.3347024. doi:1 0 . 1 1 4 5 / 3 2 9 8 6 8 9 . 3 3 4 7 0 2 4 . [17] D. Hyun, C. Park, M.-C. Yang, I. Song, J.-T. Lee, H. Yu, Review sentiment-guided scalable deep recommender system, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 965–968. URL: https://doi.org/10.1145/3209978.3210111. doi:1 0 . 1 1 4 5 / 3 2 0 9 9 7 8 . 3 2 1 0 1 1 1 . [18] A. Da’u, N. Salim, Sentiment-aware deep recommender system with neural attention networks, IEEE Access 7 (2019) 45472–45484. doi:1 0 . 1 1 0 9 / A C C E S S . 2 0 1 9 . 2 9 0 7 7 2 9 . [19] J. Urbano, H. Lima, A. Hanjalic, Statistical significance testing in information retrieval: An empirical analysis of type i, type ii and type iii errors, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 505–514.

URL: https://doi.org/10.1145/3331184.3331259. doi:1 0 . 1 1 4 5 / 3 3 3 1 1 8 4 . 3 3 3 1 2 5 9 . [20] M. D. Smucker, J. Allan, B. Carterette, A comparison of statistical significance tests for information retrieval evaluation, in: Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management, CIKM ’07, Association for Computing Machinery, New York, NY, USA, 2007, p. 623–632. URL: https://doi.org/10.1145/ 1321440.1321528. doi:1 0 . 1 1 4 5 / 1 3 2 1 4 4 0 . 1 3 2 1 5 2 8 . [21] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for word representation, in: Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543.

URL: http://www.aclweb.org/anthology/D14-1162. [22] S. Bird, E. Klein, E. Loper, Natural language processing with Python: analyzing text with the natural language toolkit, ” O’Reilly Media, Inc.”, 2009. [23] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint

References

[1] F. Ricci , L. Rokach , B. Shapira , Recommender Systems: Introduction and Challenges , Springer US , Boston, MA, 2015 , pp. 1 - 34 . URL: https://doi.org/10.1007/978-1- 4899 -7637- 6 _1. doi:1 0 . 1 0 0 7 / 9 7 8 - 1 - 4 8 9 9 - 7 6 3 7 - 6 \ _ 1 .

[2] F. Wu , Y. Qiao , J.-H. Chen , C. Wu , T. Qi , J. Lian , D. Liu , X. Xie , J. Gao , W. Wu , M. Zhou, MIND: A large-scale dataset for news recommendation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Online, 2020 , pp. 3597 - 3606 . URL: https://www.aclweb.org/anthology/2020. acl-main. 331. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 0 . a c l - m a i n . 3 3 1 .

[3] C. Wu , F. Wu , T. Qi , Y. Huang, SentiRec: Sentiment diversity-aware neural news recommendation, in: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing , Association for Computational Linguistics, Suzhou, China, 2020 , pp. 44 - 53 . URL: https://www.aclweb.org/anthology/2020.aacl-main. 6 .

[4] C. Hutto , E. Gilbert, Vader: A parsimonious rule-based model for sentiment analysis of social media text , Proceedings of the International AAAI Conference on Web and Social Media 8 ( 2014 ). URL: https://ojs.aaai.org/index.php/ICWSM/article/view/14550.

[5] D. Jannach , M. Zanker , A. Felfernig , G. Friedrich, Online consumer decision making , Cambridge University Press, 2010 , p. 234 - 252 . doi:1 0 . 1 0 1 7 / C B O 9 7 8 0 5 1 1 7 6 3 1 1 3 . 0 1 2 .

[6] R. El Baf , H. Wachsmuth , K. Al Khatib , B. Stein , Analyzing the Persuasive Efect of Style in News Editorial Argumentation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Online, 2020 , pp. 3154 - 3160 . URL: https://www.aclweb.org/anthology/2020.acl-main. 287. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 0 . a c l - m a i n . 2 8 7 .

[7] M. Sertkan , J. Neidhardt , H. Werthner , Documents, topics, and authors: Text mining of online news , in: 2019 IEEE 21st Conference on Business Informatics (CBI) , volume 01 , 2019 , pp. 405 - 413 . doi:1 0 . 1 1 0 9 / C B I . 2 0 1 9 . 0 0 0 5 3 .

[8] S. Zhang , L. Yao , A. Sun , Y. Tay , Deep learning based recommender system: A survey and new perspectives , ACM Comput. Surv . 52 ( 2019 ). URL: https://doi.org/10.1145/3285029. doi:1 0 . 1 1 4 5 / 3 2 8 5 0 2 9 .

[9] Y. Deldjoo , M. Schedl , P. Cremonesi , G. Pasi, Recommender systems leveraging multimedia content , ACM Comput. Surv . 53 ( 2020 ). URL: https://doi.org/10.1145/3407190. doi:1 0 . 1 1 4 5 / 3 4 0 7 1 9 0 .

[10] C. Wu , F. Wu , M. An , J. Huang , Y. Huang , X. Xie , Neural news recommendation with attentive multi-view learning , arXiv preprint arXiv: 1907 . 05576 ( 2019 ).

[11] M. An , F. Wu , C. Wu , K. Zhang , Z. Liu , X. Xie , Neural news recommendation with longand short-term user representations, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics , Florence, Italy, 2019 , pp. 336 - 345 . URL: https://www.aclweb. org/anthology/P19-1033. doi:1 0 . 1 8 6 5 3 / v 1 / P 1 9 - 1 0 3 3 .

[12] C. Wu , F. Wu , S. Ge , T. Qi , Y. Huang , X. Xie , Neural news recommendation with multihead self-attention , in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural