In its technical conceptualization, 'diversity' prevents a recommender system from recommending the same type of content over and over again [2], and is one of the primary values considered in research on news recommender systems [3]. Diversity is usually expressed as an 'intra-list distance', measuring whether the items within the recommendation are sufficiently different from each other [4]. However, this definition based on intra-list distance does not fully reflect the requirements of a normative interpretation of diversity, which relates to the role the recommender system plays in society [5,6,7].
Diversity has characteristics of an essentially contested concept [8], and its interpretations can vary greatly among different people and implementations [9]. We may consider different aspects when talking about diversity, such as political viewpoints [10], different ethnicities [11] or diversity of category and topic [12]. We may also talk about diversity in different contexts: for example, the recommendation should reflect society [13], it should counter existing biases [14], or expose the reader to new things [15].
Most, if not all, of these conceptualizations are relevant to the domain of news recommendations. News recommenders may play a gatekeeping role in the type of news that is exposed to the public, and thus need to be capable of incorporating editorial values [16,17]. The different aspects could logically be incorporated in the intra-list distance formalization mentioned above with a sufficiently sophisticated way to calculate distance between articles. However, the different contexts cannot be captured by a metric that only considers the items within a recommendation. To solve this, Vrijenhoek et al. [1] proposed an alternative diversity metric, conceptualized as a rank-aware divergence metric. This was called the RADio framework, where RADio is short for Rank-Aware Divergence (plus -io).
With these divergence metrics, the presence of a certain diversity aspect in the recommendation is compared to the presence of that aspect in an external context distribution. When these distributions are similar, the divergence is low; when they are very different, divergence is high. There is no clearcut 'optimal' divergence score. In some cases one could strive for a recommendation similar to the context distribution (for example, be reflective of political voices in government), in others for a higher divergence score (for example, expose a reader to new perspectives). To show how this could work in practice RADio implemented the diversity metrics (DART) outlined in Vrijenhoek et al. [18]: Calibration, Fragmentation, Activation, Representation and Alternative Voices, which are inspired by democratic theory. The metrics were prototyped with news recommenders trained on the Microsoft News Dataset (MIND) [19].
In order to do justice to the normative underpinnings of the DART metrics, the RADio metrics needed metadata that was not included in MIND. This metadata would include things like which political viewpoints are expressed in an article, is the article written in a neutral or subjective tone, or does the article mention people from a minority background. This type of information is notoriously hard to extract from just a text, and often RADio needed to rely on proxies that were known not to be exactly right, but were necessary to prototype how the framework could theoretically function.
Despite the fact that they were simplifications, the data preprocessing and augmentation steps to identify these proxies were already quite elaborate. For example, political opinions would be approximated by the mention of political actors in the text. These actors would be identified by 1) scraping article body; 2) running Named Entity Recognition on the fulltexts; 3) attempting to match entities of type Person to their entry on Wikidata; 4) checking whether this person was a politician, and for which party. Without a Golden Standard it was not possible to evaluate the performance of this approach, but even just looking at the procedure makes it quite clear that there are a lot of ways in which this process can fail. Alterations in the spelling of a name (Barack Obama vs. President Obama) could leave a political actor unidentified, and new elections or party compositions would render past results invalid. This approach, or even one based on regular expressions and/or string matching, would probably work well enough in a contained experiment over a limited amount of time where the relevant actors are already known, such as in Michiels et al. [20] and Einarsson et al. [13]. An implementation that monitors an algorithm in real-time would probably benefit from a more sophisticated approach to viewpoint diversity, such as in Draws et al. [21]. Lastly, RADio's implementation of the DART metrics also distracts from the findings of Vrijenhoek et al. [9], which claims that diversity can be conceptualized in many ways, depending on the domain's requirements.
This work describes a repository that allows for easy implementation of the divergence-based metrics, by making the different diversity aspects and tactics configurable. The code can be found on Github. 1 This paper works under the assumption that whoever implements the framework has a data preprocessing or annotation pipeline that contains the required metadata for the metrics to work. While it still keeps the DART metrics in the repository to give examples of metric configuration, the framework can also accommodate domains beyond news recommendation. Keep in mind that the repository does not provide plug-and-play metrics, and that conceptualizing diversity within a news recommender system is still very much a matter of discussion with stakeholders from outside technical teams [22,23].
The repository consists of three primary components: a Jupyter notebook which showcases how metrics could be configured, a class for building the rank-aware distributions, and a class for calculating the divergence scores.
In this part of the framework, we aim to build the distributions for the recommendation and context respectively. In order to do this, we pass the framework the list of relevant articles (either in the recommendation or in the context), and tell it which feature to look for. When building the distribution, the framework can optionally account for the rank of an article in the recommendation. It will then count articles that appear higher up in the list more strongly than those that appear lower by weighing it with the harmonic number2 of the length of the list. Making a distribution rank-aware only makes sense when there is some sort of meaning in the ordering of the articles; for example, in a recommendation ranked by predicted relevance, or in a reading history when the most recently read articles are listed first. It does not make sense in cases where such a meaning cannot be found; for example, when considering all articles that have been published over the last few days. The framework can accommodate both categorical and numerical data. Categorical data can have both single and multiple values per article. In case of numerical data, the values will need to be discretized into bins. The number of bins to be used can be set, but defaults to 10. With this approach, we lose a lot of important information. For example, we will not know that certain bins may be closer to each other than others. Future work may look into alternative ways of calculating divergence for numerical data.
Within RADio, diversity is conceptualized as a rank-aware divergence score between a recommendation and a context:
where π₯ refers to the relevant feature to consider; π to the recommendation, and π to the context. As explained in the previous section, both the recommendation π and context π can be set up to be rankaware. For more details regarding the justification of setting up diversity as a rank-aware divergence score, see Vrijenhoek et al. [1]. Within this framework, we can calculate the divergence using both Kullback-Leibler (KL) divergence and the Jensen-Shannon (JS) divergence [24]. While KL Divergence is commonly known, JS Divergence has the added benefit of being 1) symmetric and 2) bound between 0 and 1, and is thus the default option within the framework.
While the repository contains instructions for configuring all the original RADio metrics, for this paper we will discuss the configuration and output of the Calibration metric in more detail as an example. While Calibration is from a normative perspective not the most interesting metric, it relies on data that is supplied in MIND itself, and therefore does not rely on complicated data augmentation to show meaningful results. The goal of Calibration is to measure to which extent a recommendation is tailored to a user's preferences. Thus, we want this score to show low divergence, meaning that there is actually a large overlap between the recommendation and what a user wants to see. In this setup, we approximate a users' preferences by looking at the categories of articles they have consumed in the past: their reading history. Note that this is just an example implementation, and that there are likely many better ways to express a users' interests than through categories in past reading behavior.
In summary, we configure the metric in the following way: We expect all recommendations to be represented in a DataFrame, with columns for 1) the impression ID; 2) the time of the impression; 3) the ID of the user this impression was shown to; 4) the reading history of that user; and 5) one or more generated recommendations, corresponding to different recommendation algorithms. We assume that an apply-method will be called to calculate the diversity metrics for each line, and thus for each of the different algorithms. We first configure a Metric: Here, 'feature_type', 'rank_aware_recommendation', 'rank_aware _context' and 'divergence' correspond to the information summarized in Table 1. The ππππ‘ππ₯π‘ parameter is there for efficiency. If the context is ππ¦πππππ, it will need to be calculated for every line. This is the case here, as we are looking at the users' reading history, which is of course different for every user. The context can also be π π‘ππ‘ππ, or the same for all users. This is the case when for example looking at all articles published, or when considering an external distribution. Next, we write a πππππ’πππ‘π_ππππππππ‘πππ function to pass the right recommendation and context to the framework: 1 def calculate_calibration(recommendations, history): We expect 'recommendations' to be a list where each entry in the list corresponds to a different algorithm. Each entry again consists of a list of article IDs. We also expect that these are ordered by which article is going to be recommended first according to that algorithm. Next, we tell the framework to retrieve the 'category' feature for each article in both the recommendations and the reading history. The resulting lists of features are given to the framework to, under the hood, build the corresponding distributions and calculate the divergence. The resulting 'scores' is a list of scores, each entry corresponding to one of the recommendation algorithms.
We run the configured metric on the news articles and recommendations of the 'MINDsmall_dev' dataset, which can be obtained from the Microsoft website 3 . We compare the recommendations generated by the LSTUR [25] and NRMS [26] algorithms, trained using the code supplied by Microsoft4 , to those from two simple baseline algorithms: a random selection, and a selection based on article popularity. For the most popular baseline, the popularity of an item is derived from the clicks recorded in the dataset. However, there are many articles with zero recorded clicks, and in case of a tie in the number of clicks the recommender makes a random selection.
It is quite hard to pinpoint what exactly a 'good' divergence score would be. However, when we compare the algorithm we are interested in to a baseline algorithm, we can draw some conclusions on how that algorithm impacts the behavior of the metric. In this example, the first difference in metric outcomes can already be observed from calculating basic statistics on the outcomes, visualized in Table 2. At each point, the divergence in the neural recommenders is lower than those for the baseline recommenders. As expected, the neural recommenders are more tailored to the users' preferences than the baselines. Note that this does not mean that generally neural recommenders are more diverse than baseline ones; it just means that in this conception of diversity, and in this setting, the neural recommenders show more of the desired behavior than the baseline does.
Figures 1 and 2, which can also be found in the repository, provide more details into the behavior of the metric. In Figure 1 we see that the neural recommenders show similar patterns, and that the baseline recommenders behave similarly between them too. It also shows the effect of the time of day; there may have been meaningful events that influence the type of articles a recommender system can choose from, and thus make the algorithm choose articles that diverge from the users' personal preferences. Note that this is not necessarily bad, if the primary goal of the recommender is to inform readers about important events happening in the world. In Figure 2, the neural recommenders have distinctly lower divergence, which means that the recommendations they generate are closer to the users' reading history. Given that there are clear differences between the baseline recommenders in this image but not in Figure 1, some meaningful consequences happen when aggregating scores per user; the most popular recommender may generate more Calibrated recommendations for some users than for others.
Section 3 explains how to technically implement the RADio-framework to measure normative diversity in recommendations. The example metrics are tailored towards news recommendation, but the framework can be adapted to suit a wide range of applications. Yet, this does not yet answer the question of how one should go about conceptualizing diversity for their application. This can be exceptionally challenging for technical teams that, while they are the ones that need to implement the metric, often do not have all the domain knowledge necessary for making such decisions. As such, it is important that all relevant stakeholders are brought to the table. In the case of news recommendation, these would include editorial, but also strategic and business roles [22]. Readers themselves also bring a different perspective on what they value in their news, and why they would choose to read certain items but others not [27,28,29,30]. Lastly, one should not underestimate the effects of interface design on users' reading behavior. Even a perfectly built and diverse algorithm may not accomplish what it is intended to do due to position bias or simply differences between users [31]. Vrijenhoek et al. [9] interviewed professionals in the media sector, and noted all the different ways they spoke about diversity. The taxonomy that resulted from this, which is split into goals, aspects and contexts of diversity, could serve as a starting point for other implementations; at the very least, it should facilitate discussion and ease the identification of domain-specific needs and requirements. Furthermore, one could take inspiration from literature beyond the technological domain. For example, those working on news recommendation could look into how social scientists conceptualize diversity, and draw inspiration from democratic theory and the role news plays in society [6,18]. However, while democratic theory is directly relevant to news, it should not be blindly applied to other domains. Rather, we would encourage those from other domains to invest time choosing or building their own normative framework [7].
Without a doubt, conceptualizing and implementing diversity in any kind of recommender system is a complicated process, and it is unlikely that a perfect (or even a good) solution will be attained in a single iteration. One could argue that aiming for one would only prevent any progress from happening. Rather, perhaps we should aim for imperfect solutions; ones that we fully understand, and where we can exactly pinpoint what the metric does and does not do. As such, we would also urge readers not to resort to opaque solutions such as off-the-shelf Large Language Models, which may be easy to implement but are not under the control and full understanding of your organization. Solutions that we know are simplified, perhaps even 'stupid', can be discussed and criticized, and thus be improved upon. It is our hope that the RADio-codebase will make at least the technical part of the process more straightforward.
This work builds upon the code of the original RADio framework, which was a collaboration between the author of this work, Gabriel BΓ©nΓ©dict and Mateo Gutierrez Granada. I would like to thank Savvina Daniil for testing and reviewing the code, Johannes Kruse for making the repository's very first pull request, and Lucien Heitz and Alain Starke for proofreading.