News portals provide content to millions of users in current days, from topics like sports to politics. With such a wide range of themes and a massive number of possible articles to read, news recommender systems play a central role in filtering which items are more suited for a specific user at a given time [1]. However, several challenges still need to be overcome when building such systems, both in the societal and technical domains.
The definition of what is suited for a user is somewhat relative. Some may design news recommender systems to optimize for user engagement, which should lead to higher click rates or reading time on a platform [2,3]. However, in this scenario, news recommender may not be suited to keep the users informed on relevant aspects of society other than those they are more leaned to consume, which raises concerns about the democratic roles of these systems [4].
This algorithmic influence is amplified due to the fact that readers tend to engage more on contents that confirm their own worldview [5,6]. This phenomenon prompts recommenders to limit the diversity of suggested items, potentially leading to user segregation and biased opinions [7,8].
Other than these societal issues, technical challenges still need to be addressed on news recommender. As new items are added every minute with fresh information, old items are inactivated for recommendation, yielding a short item shelf life [9]. As a consequence, traditional user-item matrix used by algorithms are commonly very sparse, which sets additional challenges to model user's preferences [2,3]. This scenario is aggravated with anonymous users who usually have few past interactions in the system [10].
Besides these sparsity challenges, news recommenders heavily rely on rich feature engineering to represent items and model users' consumption from previous behaviors [11,2]. Although simpler forms of metadata can be used, such as news article categories, representing items from its textual content requires applying complex natural language processing techniques to the article's title or body content [12,13].
Creating solutions to these issues with news recommender systems requires contributions from both industrial and academic players. In this scenario, proper datasets need to be publicly available for researchers to discuss and explore solutions on a common ground. Several benchmarks have been proposed so far and each of them has had its own share of contribution towards research development.
However, most datasets so far lack the proper information structure for research on news recommender to be properly conducted [14]. To the best of our knowledge, the most suited released so far is the MIND [11] dataset, which has enabled several works to be published towards technical challenges [15,16,17]. Nonetheless, even this last benchmark still has its own limitations when it comes to normative diversity research [1].
In order to fill the gaps in previously published datasets, this paper introduces the News Portal Recommendations dataset, an improved version of a past dataset that aims to provide the necessary information for research on normative diversity in news recommender systems. Therefore, it is structured as follows.
Section 2 revisits related works published in the past related to public datasets used for research development on news recommender systems. Then Section 3 describes how the proposed dataset was constructed and what are, as well as its main characteristics that contribute to bridging the gap of past datasets. Later, Section 4 brings normative diversity metrics from the proposed dataset and, finally, all results are discussed and concluded in Section 5.
In the past years, a few datasets have been made public to foster research in news recommenders. A dataset from Globo.com [18] was built by sampling user interactions from G1 1 , a Brazilian news portal. It contains 3𝑀 records distributed in 46𝑘 news articles and 314𝑘 users extracted from October 1 to October 16, 2017. However, instead of having text information from the articles (like its title or body content), it contains the article's word embeddings generated by a neural model trained on classification tasks [18], which considerably limited the use of recent natural language processing tools and other types of diversity-oriented explorations.
The MIcrosoft News Dataset (MIND) [11] dataset was later published, providing news' textual information as metadata for 161𝑘 news items, MIND contains 24.1𝑀 logs for 1𝑀 randomly sampled users from Microsoft News2 who had at least 5 clicks in the period between October 12 and November 22, 2019. In addition, the dataset is associated with a public competition3 in which the goal was to predict the click scores of candidate news based on user interests.
A few other datasets were published before the aforementioned [14]. Plista [19] contains activity logs from 13 German news portals, recorded in June 2013 by ≈ 1𝑀 sampled records to ≈ 70𝑘 items. Adressa [20] included 27M click interactions from 3M users to 48k news articles, extracted in a ten weeks period from Adresseavisen4 . However, each of these datasets has its own limitations like size or lack of metadata, as thoroughly described in the MIND original article [11].
Considering these aforementioned datasets, MIND became a reference benchmark due to its size and textual components. Nonetheless, despite its contributions, some of its limitations towards recommender diversity were brought to light.
Firstly, as it contains a considerable amount of soft news, it may compromise research in normative diversity metrics which are more tailored towards the so-called "hard" news [1]. Secondly, the dataset is split among training, validation, and test sets, in which the validation set only contains data from November 15, 2019. In this case, it is unlikely how the users and the recommender's behavior towards those users change over time [21]. Finally, nearly half of the anonymous user IDs have unique visits, which makes it unlikely to model the long-term effects of the recommender diversity on user consumption.
In order to improve the research possibilities on news recommendations towards normative diversity, this article proposes the News Portal Recommendations (NPR) dataset, a restructured version of a previous dataset aiming to provide a qualified dataset for research purposes. In particular, we list the following main contributions compared to the MIND dataset:
• Focus on hard news • Ranked recommendation lists • Distinction between logged and returning users from anonymous ones • Longer periods of data
The NPR dataset was built by sampling users from G1 5 , the largest Brazilian news portal maintained by the Globo media company. It contains 1162802 randomly sampled users who received recommendations in the period between January 3rd, 2023, and May 1st, 2023, where nearly 73% are non-logged users. All users were anonymized in order to protect data privacy. NPR was developed following the same structure as the MIND [11] dataset. Therefore, it is composed of the following files: behaviors, articles, and impressions.
The behaviors file contains 1402576 impressions logs regarding which sequence of items was recommended at a given time and which items users consumed before receiving such recommendations. Unlike the previous dataset, NPR also includes statistics on user behavior regarding the articles' page, such as the number of clicks, time spent on the page, and scroll percentage.
The articles file contains metadata for 148099 news items that were either consumed or recommended to users in the previous behaviors file. It contains news URLs, their title text, and a list of topics associated with each article assigned by a specialized editorial board. The complete schema for the article's file is displayed in the dataset's repository in Kaggle.
Finally, the recommendations contains three files on 92700 randomly sampled recommendations generated by means of the following algorithmic approaches: Collaborative Filtering, the most Recent publications, and the Top consumed articles. Each file refers to recommendations provided by one specific algorithm, but all files share the same impression IDs and, as a consequence, refer to the same users.
Since textual information plays a central role in news datasets, Figure 1 displays the distribution from some of the article's features.
The top left and top right plots already show a language distinction between the NPR and MIND datasets. In terms of the number of words in the article's titles, NPR has an average of 14.9 words while its counterpart presents 11.52 [11]. However, the biggest difference is in the article's body length. While NPR has a single-modal and skewed distribution, with articles having 471.7 words in their body on average, MIND's body length is multimodal, with averages around 20 and 80 [11]. This indicates that NPR contributes to much richer textual information to be assessed with natural language processing techniques.
In addition, most news articles are associated with a topic assigned manually by the editorial board, which consists of multiple teams spread out in different geographic regions of the country. Such scenario potentially results in non-uniform categorization of news articles. Solutions to address this challenge are further discussed in the Future Work section. From a total of 94 topics (bottom left plot in Figure 1), most of the news is related to sp, mg, and rj, which are acronyms for Brazil's states. Since these are some of the most populated states in Brazil, it indicates the predominance of regional content. In fact, the topic's distribution is so unequal that it reaches a 74% Gini index [22] of distribution inequality. Other generic themes like "mundo" (world), "política" (politics), and "economia" (economy) also have a significant share of news articles.
Based on these topics, news articles can be associated with hard and soft news. As explored by Vrijenhoek [1], the MIND dataset has a higher share of soft news, which may not be the best scenario for research on normative diversity. In order to evaluate the differences in those news types, Table 1 shows comparisons between NPR and MIND datasets.
Considering all articles in the catalog, NPR presents 91.0% of hard news items while MIND Finally, the news in the G1 ecosystem has a short survival time. It can be observed in Figure 1, where the consumption of the news is more concentrated on the first days after the publication. In this case, more recent news is more consumed.
Despite having a higher share of hard news, items on NPR seem to last longer than the ones presented in MIND. Figure 1 shows on the bottom right plot the cumulative distribution function (CDF) of news survival time (number of days between the article's publish date and last click). It can be seen that 85% of items are clicked up to 7.95 days for hard news and 9.55 days for soft news, which reinforces the characteristic of soft news to last longer.
As aforementioned, news recommender systems play a central democratic role in keeping users informed by unlocking the diversity of online information [4,8]. However, the definition of diversity is plural, especially when contrasting the fields of computer science and normative literature. For instance, while technical metrics such as intra-list distance of recommended items [23,24] or gini index [25,26] may be a proxy to diversity in computer science, normative literature might lean towards concepts of democracy, freedom of expression and cultural inclusion [27,4].
To bridge the gap between technical and normative literature, a framework by the name of RADio (Rank-Aware Divergence Metrics) [28] has been proposed to translate normative goals into a set of quantifiable metrics grounded in democratic theory. The framework works under five metrics which are summarized as follows (for a thorough description of the metrics, refer to [28,27]):
• Calibration: assesses the degree to which the issued recommendations align with the user's preferences. The further from 0, the greater the deviation from the user's preferences. • Fragmentation: quantifies the level of overlap among recommendations presented to distinct users. The closer to 0, the greater the overlap. • Activation: gauges the extent to which the issued recommendations aim to motivate users into action. The closer to 0, the more neutral the content. • Representation: indicates how different opinions or perspectives are expressed. The closer to 0, the more balanced the content whereas higher scores measure larger discrepancies. • Alternative Voices: measures to which extent minority groups are represented in the content. The closer to 0, the fewer the presence of minority voices.
Based on the different values extracted from a news recommender for these five metrics, it can be assigned to four democratic models described by Helberger [4]: liberal, participatory, deliberative, and critical. A reference table overview for each model is documented in [28,27].
All five RADio metrics aforementioned were applied to the MIND dataset for 6 different algorithms [28]. However, given that some of these metrics rely on applying natural language processing techniques to extract aspects such as entity recognition of minority voices or content neutrality, we focus the analysis on the calibration metric.
The NPR dataset contains recommendations for three different kinds of algorithms. The first is an Alternating Least Square ("ALS") strategy, which is a classical recommendation algorithm based on the factorization of the user-item matrix [29]. The other two algorithms are the "Top" algorithm, which recommends the most consumed news articles from the past 48 hours, and "Recents", which recommends the most recent news articles published by the editorial board. Both of these later algorithms are non-personalized, meaning that all users who access the news portal at a given time receive the same recommendation.
Considering these three algorithms, Figure 2 provides two plots generated after extracting the calibration metrics on different recommendation scenarios. The left plot shows how calibration is distributed among three different algorithms after recommending 5 items, whereas the right plot shows the average calibration considering multiple recommendation list sizes. At first glance, no major differences can be observed between algorithms. A first hypothesis for such an overlap is that the "ALS" algorithm may converge to the "Top" or "Recents" strategy, especially when considering hard news with a short life span. Additionally, the dynamic behavior of hard news might make it more difficult to model user preferences, since users are likely to change their interests rapidly. For instance, users may consume distinct categories of hard news due to the fact that they are breaking news, yielding a more general type of consumption profile. This scenario can be addressed by more robust algorithms that are more suited for news recommendation, which are already in place in Globo portals. We discuss their use in the scope of this dataset later in the future work section.
However, the plot shows a noticeable difference when comparing recommendations between logged versus non-logged users. Recall that as calibration approaches 0, recommendations are more tailored towards the user's preferences. Since logged users tend to have more historical data, it is reasonable to see lower calibration values when compared to non-logged users, which can be seen by comparing the distributions' quantiles.
By expanding the analysis to ranks different than 5, the right plot in Figure 2 provides how the average calibration changes according to different recommendation list sizes. For longer lists, it is more probable to find items tailored to the users' preferences. Therefore, it is reasonable to observe a descending calibration variation as recommendation lists get larger. Since NPR contains up to 10 items in the recommendation lists, a lower limit of calibration can be observed around 78.9%.
Based on this lower limit, we can establish a calibration comparison between MIND and NPR using the results reported for the "top" algorithm in [28] (referred to as the "most popular"). By recommending a list of 10 items to the users, a calibration of 65.3% was observed after using news article topics. Therefore, we can observe that even a non-personalized algorithm may present significantly different calibration results depending on the dataset, which reinforces the need for several datasets to be employed as benchmarks when analyzing the diversity capabilities of a recommendation algorithm.
Besides the calibration analysis, Figure 3 also shows preliminary results on the representation (left plot) and fragmentation (right plot) metrics. It can also be seen from Figure 3 that no clear difference is observed among algorithms, which reinforces the aforementioned hypothesis that the "ALS" algorithm converges to the "Top" approach. Notice that, for all algorithms, the average representation is close to 50%, whereas average fragmentation approaches 85%. A hypothesis for such behavior is the lack of proper natural language processing tools towards Portuguese texts which better indicates how different opinions or perspectives are expressed.
Taking into account the need for proper datasets to be publicly available for both Academy and Industry researchers to discuss and explore solutions on common ground, this paper introduced the News Portal Recommendations (NPR) dataset. The dataset provides data on recommendation impressions, user behavior (consumption history), and also metadata about published articles.
Section 3 analyzed specific characteristics of the dataset and also compared it with the MIND dataset from Microsoft News [11]. Other related datasets are also described. Besides providing much richer textual information in comparison with MIND, NPR has a considerably greater proportion of hard news consumption than the MIND dataset. Subsequently, section 4 showed that the NPR dataset could be applied to the RADio framework [28], translating normative goals into quantifiable metrics.
The first version of the dataset is already publicly available, opening the horizon for continuous updates and improvements based on feedback from the community. Some improvements are already planned for Future Work.
When looking at news categories, the NPR dataset also presents acronyms for Brazillian states, indicating the predominance of regional content. As stated in section 3.2, the need for manual tagging potentially results in non-uniform categories. In this sense, we are currently developing automatic extraction of semantic metadata from news articles, which will enrich the current categories already in place in the dataset. In this context, we also aim to explore content-representation techniques in order to remove any possible differences resulting from the Portuguese language.
Finally, in addition to the ALS algorithm, we will also incorporate recommendation impressions resulting from other, and more advanced, personalization algorithms. Although other algorithms are already being employed for providing recommendations to the final user, engineering efforts are needed to extract multiple algorithms' outputs to the same users due to Globo's AB platform. Since in Globo any information delivered to the final user is subjected to an AB test, the recommendation algorithm employed, as well as its resulting impressions, will vary depending on the AB testing alternative that has been employed for that specific user.
Thanks to Mateo Gutierrez Granada, Johannes Kruse, and Gabriel Benedict for implementing the code that made it possible to run the RADio metrics on Globo's dataset. We would also like to acknowledge Globo for providing this dataset for the academic community, especially to the Recommendation team for preparing the original dataset from the G1 Portal.