News recommendation systems face unique challenges, including the dynamic nature of user preferences and the need for diversity in recommended news articles. To address these challenges, we propose a deep neural network architecture that learns representations for both news items and users. Our approach uses an enhanced vector for each query and news item to facilitate information interaction between these entities. To overcome selection bias in implicit user feedback, we employ negative sampling. We also promote diversity in recommended news by aligning the uneven news category representations of items in a loss function. Experimental results on a benchmark dataset demonstrate the superiority of our proposed architecture over baselines, achieving both relevancy and diversity in the news recommendations.
Prominent news organizations such as Yahoo!, BBC, NYTimes, and CNN have introduced
online news portals, which users can access from any location to peruse a wide range of news
categories and stay up-to-date. However, with the abundance of information available on the
internet, locating pertinent news has become a challenging and time-consuming task. News
recommender systems (NRS) address the issue of information overload by presenting users with
personalized and intriguing recommendations, chosen from an extensive pool of available news
articles [
An NRS must balance between maintaining relevance to the user’s interests and introducing enough diversity to keep the user engaged. If the NRS recommends news articles that are too diverse or unrelated to the user’s interests, the user may lose interest and stop using the system. Conversely, if the NRS recommends only news articles that are closely related to the user’s interests, it may miss out on opportunities to introduce the user to new topics and categories. Therefore, finding the optimal balance between relevance and diversity is crucial for the success
In general, users’ preferences can be either long-term or short-term, with short-term
preferences defining their current interests. For example, a user may have a long-standing interest in
Joint Proceedings of 10th International Workshop on News Recommendation and Analytics (INRA’22) and the Third
International Workshop on Investigating Learning During Web Search (IWILDS‘22) co-located with the 45th International
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’22), July 15, 2022, Madrid, Spain
a particular topic, such as climate change, and read numerous articles related to this topic over
the course of their lifetime [
Short-term interests can also be identified by analyzing the user’s recent behavior on the NRS. For instance, if a user has been reading numerous articles on a particular topic over the past few days, the NRS can infer that the user is currently interested in that topic and adjust its recommendations accordingly. By incorporating short-term interests into the recommendation process, the NRS can provide more personalized and relevant recommendations to the user.
This paper presents an NRS that prioritizes both relevancy and diversity in the
recommendation process, achieving a more balanced and engaging user experience. The specific
contributions of this paper are:
• The proposed deep neural network is based on a query-candidate architecture [
Extensive experiments on a news dataset show that our proposed approach can provide both relevant and diversified news recommendations in an NRS.
Next, we discuss our approach.
The objective of news recommendation is to select relevant candidate news items from a news corpus, given a set of queries. The item set is represented as { }=1 and the query set as { }=1 . The recommendation problem, denoted as R, is learned from the query-item feedback represented by a matrix ℝ × . Each query is considered as feedback provided by the user. If query j gives positive feedback on news item i, then = 1; otherwise, it is considered as non-positive feedback.
We show our proposed architecture in Figure 1 and explain next.
The proposed model is a query-candidate model that consists of an embedding layer, an augmented layer, and two models (query and news item generation). The model takes into account diferent content features related to news items, such as news ID, title, body, and category, as well as contextual information.
The embedding layer is represented by an embedding matrix E ∈ ℝ× , which maps each piece of information (e.g., news item or user ID) in and to a low-dimensional dense vector ∈ ℝ .
The augmented layer creates two augmented vectors and by concatenating the IDs corresponding to two input feature vectors f and f , respectively. These augmented vectors are then concatenated with the original feature vectors f and f to obtain the augmented representations of query and news item . The fully connected layers with the ReLU activation function are applied to these concatenated vectors, and the output from the fully connected layers goes through the 2 normalization layer to obtain the augmented representations of query and news item .
The loss function of the proposed model is defined as the mean square error between the augmented vectors and and query/item embedding for each sample of which label equals 1. The augmented vectors and are used to fit all positive interactions in the model belonging to the corresponding query or item. The stop gradient strategy is applied to stop the gradient of and from flowing back into and , respectively. The output of the model is the inner product of the query embedding and news item embeddings.
To improve the generalization ability of the model and enable it to learn more transferable features across diferent news categories, we introduce an additional loss function during the training phase. This loss function aims to minimize the distance between the feature representations of news items from diferent categories. To achieve this, we first select the largest news category as the reference category. We then calculate the squared Euclidean distance between the feature representations of news items in this category and the feature representations of news items in the other categories. This loss function is added to the overall objective function, with a regularization parameter to control its relative importance.
The final loss function is calculated as the sum of the binary cross-entropy loss, the loss functions for query and news item representations, and the category-aware loss function.
To train the model for news recommendation, we treat the problem as a binary classification task. We use a random negative sampling technique, where for each positive query-item pair (the label = 1), we randomly select a set of N news items from the news corpus to create negative query-item pairs (the label = 0) for that query. This process results in a dataset with a balance of positive and negative samples. We then use binary cross-entropy loss to train the model by minimizing the error between the predicted scores and the ground truth labels.
We use the benchmark dataset MIND-small [
Following the standard evaluation methodology s in NRS [
Relevancy metrics: We use Mean Reciprocal Rank (MRR) and F1-score (harmonic mean of precision and recall) to evaluate the relevancy of news recommendations. MRR is the average of the reciprocal rank of the first relevant item in the recommendation list for a set of queries.
To compute MRR, we first calculate the reciprocal rank for each query:
Reciprocal Rank = 1
rank of first relevant item Then, MRR is calculated as the mean of the reciprocal ranks for all queries: (1) MRR = ∑|=| 1 Reciprocal Rank || where || is the number of queries.
Diversity metric: In addition to the relevancy metrics of MRR and F1-score, we also use the
GINI index to evaluate the diversity of news recommendations. GINI is a diversity metric that
measures the inequality of item distribution across diferent categories or clusters [
Tradeof metric : In the context of news recommendation systems, trade-ofs are necessary to balance multiple objectives, such as relevancy and diversity. Striking an optimal balance between these two aspects enhances user satisfaction and engagement. We use the trade-of metric, which is the product of the MRR and GINI scores, divided by their sum, multiplied by 2, to measure the balance between relevancy and diversity.
tradeof = 2 ∗ ( MRR ∗ GINI ) ( MRR + GINI )
The trade-of metric ranges from 0 to 1, with higher values indicating a better balance between relevancy and diversity. By evaluating both relevancy and diversity, we can ensure that our news recommendation system provides users with personalized, engaging, and informative news articles. The trade-of metric ranges from 0 to 1, with higher values indicating a better balance between relevancy and diversity.
We use the following baseline methods for evaluation.
Wide Deep [
DKN [
LightGCN [
SASRec [
NeuMF [
We implemented these models in TensorFlow. The embedding dimension and batch size were ifxed to 32 and 256. We use the Adam optimizer for training. Other hyperparameters of all models were individually tuned to achieve optimal results to ensure a fair comparison. The dimensions of augmented vectors were both set to d= 32, and the tuning parameter 1, 2 were set to 0.5 and 3 to 1. We set top@ k to 5 and 10 as it is normally good practice to retrieve a relatively large number of candidate news items to rank. (2) (3)
We evaluate the results for both relevancy and diversity and mainly evaluate the model
performance based on the tradeof score, as it shows a harmonic mean between relevancy and
diversity. We expect a good tradeof score to be above 50% as it is a balancing score between 2
diferent metrics [
Method Wide & Deep DKN LightGCN SASRec NeuMF Proposed Model
Overall, we observe in Table 1, our proposed approach outperforms all the other methods in terms of relevancy (MRR, F1, precision), diversity (GINI), and trade-of scores on the dataset. This indicates that the current model is capable of providing personalized news recommendations that are not only relevant to individual users but also diverse enough to expose them to a variety of content. The relevancy scores may not be optimal but we achieve balanced tradeof scores.
The improvements in relevancy, as demonstrated by the higher MRR and F1-score, suggest that the current model can better capture users’ preferences and deliver news articles that cater to their interests. This is particularly important in the news recommendation domain, where user engagement largely depends on the presentation of content that aligns with their interests and preferences.
The higher diversity, as represented by the GINI score, shows that the current model is able to recommend a more diverse set of news articles, helping users discover new and unexpected content. This is an essential aspect of a news recommender system, as it encourages users to explore diferent perspectives and broadens their understanding of various topics.
Moreover, the better trade-of score signifies that the current model strikes an optimal balance between relevancy and diversity, ensuring that users receive a well-rounded set of recommendations. This balance is crucial in maintaining user satisfaction and engagement, as it prevents filter bubbles and echo chambers from forming while still providing users with content that matches their interests.
These results also show that by using negative sampling, we are reducing the selection bias
[
Next, we showcase the relevancy-diversity trade-of achieved by our model in Figure 2.
In Figure 2, we observe several trends and relationships between relevancy, diversity, and the tradeof as the number of recommendations increases. The relevancy scores decrease as the number of recommendations increases. This observation suggests that it becomes more challenging for the recommender system to maintain high relevancy for all items as the recommendation list size grows. This challenge is a common trade-of in recommender systems, as providing more recommendations can increase the likelihood of including diverse items at the cost of potentially lower relevancy for some items.
Conversely, diversity scores increase with the number of recommendations. This trend indicates that the recommender system is capable of providing more diverse recommendations as the list size grows. Including a larger variety of items in the recommendation list can enhance the user experience, as it exposes users to a broader range of content that may match their interests.
The tradeof scores, which measure the balance between relevancy and diversity, remain relatively stable across diferent recommendation list sizes. This stability suggests that the recommender system is maintaining a reasonable balance between providing relevant and diverse recommendations. In the given example, the tradeof scores show a slight decrease as the number of recommendations increases, indicating a minor compromise in balancing relevancy and diversity for larger recommendation lists.
This analysis highlights the challenge of balancing relevancy and diversity in recommender systems as the number of recommendations increases. It is crucial to find a suitable balance to ensure an optimal user experience, providing recommendations that are both relevant to the user’s preferences and diverse enough to expose them to a variety of content.
To analyze the impact of negative sampling on our news recommendation system, we compared the performance of the model with and without negative sampling using precision and recall metrics at a recommendation list size of 10. As illustrated in Figure 3, employing negative sampling significantly improves both precision and recall scores. This finding highlights the importance of incorporating negative samples during the model training process, as it enables the system to provide more accurate and diverse recommendations for users.
In this paper, we present a deep neural network-based architecture designed to model the information interaction between query and news items. Our approach incorporates a variety of features in both the news item and query representations. Additionally, we introduce a loss function that selects distinctive news items across diferent news categories. We briefly discuss selection bias and demonstrate how using negative sampling can mitigate this bias by including random negatives from the news corpus.
Extensive experiments on a benchmark dataset showcase the superior performance of our
proposed method in achieving a balance between accuracy and diversity. For future work, we
plan to conduct experiments on additional real-world news datasets and explore the potential
of deeper neural networks. We also intend to incorporate more evaluation metrics to assess
relevancy, diversity, and novelty in the recommendation results. Furthermore, we aim to address
challenges such as mitigating biases [