=Paper=
{{Paper
|id=Vol-3898/paper3
|storemode=property
|title=Diversifying for Democracy: On the Framing of Diversity in the NRS Design and the Normative Consequences for Journalism
|pdfUrl=https://ceur-ws.org/Vol-3898/paper3.pdf
|volume=Vol-3898
|authors=Jannie Møller Hartley,Elisabetta Petrucci
|dblpUrl=https://dblp.org/rec/conf/normalize/HartleyP24
}}
==Diversifying for Democracy: On the Framing of Diversity in the NRS Design and the Normative Consequences for Journalism==
https://ceur-ws.org/Vol-3898/paper3.pdf
Diversifying for Democracy: On the Framing of Diversity
in the NRS Design and the Normative Consequences for
Journalism
Jannie Møller Hartley1,*† and Elisabetta Petrucci1,†
1
Roskilde University, Universitetsvej 1, 4000 Roskilde, Denmark
Abstract
The concept and umbrella term news diversity has been introduced in the literature in Social Sciences and
Computer Sciences as a way to counter these phenomena and to produce well-functioning news
recommender systems appropriate for a democratic society. The aim of this article is to critically analyse
what kind of normative assumptions about the role of journalism emerge from the framing of diversity in
the literature on news recommender systems. We do so by conducting a framing analysis of the concept of
diversity in the literature in the fields of Social Sciences and Computer Science (N=57). We show that
diversity is framed as an essential component of a well-functioning NRS, as it is framed as the solution to
the filter bubble and potential polarization problem. However, diversity is most often operationalized as
diversity of news topics and political viewpoints on a left/right wing scale across the literature. Further the
analysis shows how the cause of diversity´s importance changed over time in the literature from initially
being rooted in the individual users’ lives, connected to their potential boredom and information overload,
to the more recent literature framing diversity in NRS as connected to the democratic role of news. The
implicit role of democracy is linked to liberal models in the operationalization of diversity, while more
deliberate and participatory models of democracy are less prominent.
Keywords
Diversity, Recsys, News, Framing Analysis, Literature Review, Democracy 1
1. Introduction
Since the early days of Journalism, it has been the task of journalists to choose between a number of
given possible news item of the day and turn them into stories informing the public. This selection
and gatekeeping process has been highly linked to normative ideals of the role of the press [49],
which has historically also changed from an omnibus press system to a more segmented press, at
least in a Western context [55].
With the digital turn in the field of journalism, users are not only segmented, but constructed as
aggregated datapoints with AI-driven recommender systems [39], increasingly influencing how and
what news gets produced and distributed to the public, leading to questions of how this role of
journalism as cultivators of democratic publics can be safeguarded.
In the newsroom today decisions are increasingly made on the basis of large amounts of auto-
generated big data of audiences [3, 11, 42], and it is these data that are now also being utilized in
personalization projects [8] and for implementing and evaluating recommender systems. A report
shows that personalization of recommender system is a common area for use of AI in newsrooms
[47]. This trend marks a shift from journalism’s traditional focus and orientation toward shared
NORMalize 2024: The Second Workshop on the Normative Design and Evaluation of Recommender Systems, October 18, 2024,
co-located with the ACM Conference on Recommender Systems 2024 (RecSys 2024), Bari, Italy.
∗
Corresponding author.
†
These authors contributed equally.
jath@ruc.dk (J.M. Hartley); elpa@ruc.dk (E. Petrucci)
0000-0001-5784-2864 (M. Hartley); 0009-0009-6464-0895 (E. Petrucci);
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
importance and the public sphere [18, 38, 39] to emphasising highly individualized news experiences
where the news distribution is responsive and based on algorithmic surveillance and the
interpretation of individuals’ past behaviours [10].
Both in the academic and public debate there seems to be no doubt that this development will
affect the democratic role and contribution of the press, however, it is still unclear and debated how
[19, 27, 29]. On the more optimistic side of the debate, scholars have been advocates for the potential
positive effects of increasing the general engagement with news, for news to become more
responsive to audiences, to counteract negative effects of information overload and for
personalisation to offer new business models that can ensure the survival of an otherwise challenged
news industry [2, 27]. Critics have expressed concerns of increased audience fragmentation and
polarisation creating echo-chambers or filter bubbles, in which audiences are exposed to content
they are likely to agree with and that strengthens pre-existing beliefs, risking the loss of a shared
public sphere [12, 19, 43, 50].
The concept and umbrella term news diversity has been introduced in the literature in Social
Sciences and Computer Sciences as a way to counter these phenomena and to produce well-
functioning news recommender systems appropriate for a democratic society. We take as starting
point Helberger´s [27] claim that the design and evaluation of recommender systems can be done
according to the perceived and aspired role of Journalism, but given that Journalism historically has
many different roles and functions in democratic societies according to Strömbäck [49], it remains
important to investigate which role and function of journalism is taken as normative when
recommender systems are designed and argued for in academic scholarship. This is important
because this research feeds into the media industries, who currently design and implement NRSs
across their sites and services. Furthermore, much of the existing literature focusing on recommender
system design presents diversity as solution to the filter bubble issue by means of design, influencing
the way problems in journalism are defined and which actors are designated to solve them, which
remains imperative to examine. While previous literature has done a good job in mapping and
discussing potential and shortcomings of algorithmic news recommendation in democratic society,
it has done so without clearly explicating neither the normative role of journalism assumed to be
desirable, nor the impact of algorithmic recommendation might have on such role.
Thus, the aim of this article is to contribute to this line of research into diversity in algorithmic
news curation, but rather than looking for gaps and important avenues for future research into how
algorithmic curation affects users or journalists in the newsroom, our aim is to critically analyse
what kind of normative assumptions about the role of journalism emerge from the framing of
diversity in the literature on news recommender systems. As this vast body of literature often feeds
into the design and the implementation of news recommenders, it is important to investigate and
discuss how such normative assumptions about journalism matter for this development and thus
also for the future of news curation, as power is delegated to specific actors and specific ways of
framing diversity in news are presented as the solution to the problems in the news industry.
We do so by conducting a framing analysis [17] of the concept of diversity in the literature in the
fields of Social Sciences and Computer Science. Framing theory has proven valuable in pointing
towards implicit ways of framing problems and solutions, which normatively privilege some
perspectives while others are ignored or not perceived as possible avenues. Indeed, if the design and
evaluation of news recommender systems says something about the implicit normative role of
journalism, analysing the framing of the conceptualization and operationalization of diversity in
news recommendation will allow us to gain a sense of the normative assumptions about the
democratic role of journalism in the literature. In the next section we present and position our study
in relation to previous studies on news recommenders in general and on diversity in particular.
Following a section presenting the methods and the data we proceed with the results. Finally, we
discuss our results in relation to the consequences of this framing for the future of recommender
systems in the news industry.
2. State of the Art
NRSs are algorithmically driven personalization systems that make use of implicit user data to make
assumptions about what news readers find interesting and relevant. It is the reliance on implicit user
data, rather than on specific user choices that makes algorithmic personalization considerably
different from customization [4]. In the literature both personalization efforts that deal with the
domain of news more generally (such as news aggregators like Google News or the recommendation
of news on social media) and personalization efforts by the news industry have been dealt with under
the umbrella term of news recommendation. However, in this context, we use the term news
recommender systems (NRSs) to indicate algorithmically driven distribution systems specifically
employed by the news industry.
NRSs have become a way to sell news to audiences, but arrived in the news industry from the
domain of social media, where they were originally meant to “sell ads to the audiences and audiences
to advertisers” [8, 15]. In this sense, it is not surprising that their entrance in the domain of news
was followed by concerns about issues of filter bubbles and polarization [43, 50]. These concerns
have now lost traction in the literature, which recognizes the theory of filter bubbles as too naïve
and simplistic [8, 40]; at the same time, more general concerns about the invisibility of the
technological mediation influencing the users´ choice architecture, which is a de facto digital nudge,
seem to remain relevant [31].
Helberger [27] worked specifically to construct a framework to evaluate the role of news
recommender systems in democratic society and argued that, as a tool, they are neither good nor
bad; it is the way they are used by the media organizations that makes them more or less beneficial
to democracy, she argues. Similarly, Bodó et al. [9] calls for the relevance of choice of goals and
consequent KPIs and for the need of framing such goals and KPIs according to the specific goals of
news organizations, which are considerably different from the goals of platforms. The algorithmic
recommendation of news not only has domain specific goals but presents also domain specific
challenges, both technically and ethically. Technically, the algorithmic recommendation of news
faces challenges connected to the time sensitivity of news content and the difficulty of modelling
user profiles. Ethically, news personalization tends to work with and abide by different logics than
the logics of journalism as a profession; for instance, Møller [40] describes how the reliance on
audience metrics needed for algorithmic news recommendation is producing “an increasingly
metrics-oriented mindset among reporters that comes into conflict with longstanding journalistic
norms and ideals about journalists´ professional autonomy from market pressure”. Being accurate is
not enough – news recommender systems need to be able to fulfill not only their role as mediating
technology, but their journalistic role as well. In this sense, the function of design has become key,
and research has increasingly focused on the introduction of journalistic values in the design of news
recommender systems [4, 27]. One of the values that seems to be key to the design (and evaluation)
of news recommender systems, together with transparency and serendipity, is diversity, which is
the focus of this article.
The term diversity is widely used in connection with media and news, most likely because the
link between news media, diversity and democracy has been extensively researched and
substantiated [33, 34, 41]. However, the term diversity not only can refer to different things, such as
media ownership, sources, contents, and viewpoints, but can also be conceptualized in different
ways. The fluid adoption of the term diversity has spilled in the interdisciplinary domain of
recommender systems. Helberger et al. [28] has worked to try to systematize the way the term is
used in relation to news recommendations. In her earlier work, she made a first distinction between
diversity of supply, or the variety of content and sources media provide, and diversity of exposure,
or the actual content audiences are exposed to, in order to argue that recommender systems may
provide an opportunity for public service media to expose audiences to more diverse content, which
they would usually not consume. For Helberger et al. [28], diversity of exposure is not a goal in itself,
but a means to cultivate informed citizenship and therefore fulfill the democratic role of journalism.
News recommender systems, then, have a democratic role [27] and can and should be designed with
such democratic role in mind.
Conducting a literature review of the concept of news diversity Joris et al. [32] found twenty-six
different conceptualizations of the term in the studies they analyzed, which they argue are most
frequently employed in the literature because relatively easy to measure, which is usually done in
the context of “internal diversity” (p. 618), meaning on the single platform or site. In their literature
review on media diversity Loecherbach et al. [36] also found a tight link between the
conceptualizations and operationalizations of diversity; however, they claim that empirical
operationalizations of diversity tend not to refer back to specific conceptualizations of the term.
Moreover, while it is common to stress the relevance of diversity in connection to democracy,
“further specifications as to the type of democracy that is used” are not given (p. 615).
Vrijenhoek et al. [55] analyze how practitioners at three different public service media
organizations in the Netherlands conceptualize diversity within the scope of their recommender
systems. They show that even within this limited domain, conceptualization of diversity greatly
varies, and argue that it is unlikely that a standardized conceptualization will be achieved.
It is this line of research that our study is based within; however, since previous literature reviews
[32, 36] found that diversity is rarely defined and clearly conceptualized, in our study we focus not
on how its defined, but rather how diversity is operationalized and framed in the literature.
Moreover, we intentionally employed only the keyword ‘diversity’ for our search, without attaching
it to or supplementing the search with specific terms, such as ‘source’ or ‘opinion’ or ‘viewpoint’, in
order to get a sense of what is the implicit framing of diversity in the literature. This tells us what
normative role of journalism is implicitly guiding the studies, designs, and evaluations of such
systems, which is important for the future role of such system for the news industries. Our study
takes as starting point Helberger’s connection between the normative role of journalism and the
design of news recommender systems; however, rather than taking diversity by design as a given
good for news organization, we seek to critically asses what implicit assumptions we find in the
literature, investigating how the literature implicitly frames what sort of problem NRSs and diversity
by design are seen as the solution to, as well as what sort of moral judgements and delegation of
power to specific actors are happening along the way.
This leads us to asking, the following 3 research questions:
RQ1: How is diversity framed across Social Science and Computer Science literature on diversity
in NRS?
RQ2: What does this framing tell us about the implicit normative role of journalism?
RQ3: What frames are left out and what might be the consequences for the design of NRS?
3. Methodology
The empirical material on which our analysis is based is a corpus of journal articles from the fields
of Social Sciences and Computer Science. To construct such corpus of documents, we conducted two
literature searches, using the same keywords - “diversity” AND “news” AND “recommender system”
- in two different databases. The choice of keywords was guided by previous research, such as
Loecherbach et al.’s systematic literature review of conceptualizations and operationalization of
‘media diversity’ in different fields [36]. While Loecherbach et al. used a variety of synonyms of
diversity in their search string, we decided to only use the term ‘diversity’ because it “is more used
in communication and computational science” [36] (p. 611), which are the fields we were primarily
interested in. Moreover, such search would allow to capture both articles containing the critical
perspectives on NRS and articles operationalizing diversity into data science models for testing and
evaluations.
The search was conducted in the ACM Digital Library (Computer Science database) and Scopus
covering the period 2011 to August 2024, where 65 results were found. Most of these results come
up in the period 2022-2024 showing a dramatic increase in research dealing with diversity (29 articles
in this period vs 35 in the period from 2011 to 2022). After reading the abstracts eight articles from
the Computer Science field were filtered out because they did not pertain to the domain of NRS.
Thus, we ended up with a sample of 57 articles. We acknowledge that the reduced size of the corpus
may be a limitation of our study; however, the study does not aim to conduct a systematic review of
the issue of diversity in the domain of recommender systems, but rather to investigate how the issue
of diversity has been framed in the literature and what such framing reveals about the perceived role
of journalism.
Once the corpus of literature was gathered, the articles were imported in NVivo, in order to
conduct inductive thematic coding, with the aim of identifying the framing of diversity both on a
conceptual and operational level. After the coding process was completed, a framing analysis of the
conceptualization and operationalization of diversity following Entman’s approach was conducted
[17]. We found framing analysis an ideal method to investigate the gap between conceptualization
and operationalization of diversity, as “the conceptual definition of diversity and its normative
underpinnings significantly affect how it is operationalized” [36] (p. 607). Entman [17] suggests that
scholars conducting framing analysis should mine for four different properties of a frame: the
proposal of a particular problem definition, interpretation, causal relation, moral evaluation and/or
recommendation/proposed solution. This meant coding the articles for how they implicitly define
the problem that diversity is supposed to solve, what are the causes of this problem, and how
different moral evaluations take place as a part of this problem definition, paying attention to the
delegation of power to specific actors and processes along the way as other divergent framings are
left out.
4. Analysis
Our analysis shows that the Social Sciences literature includes articles written by scholars in the
fields of Journalism, Media, and Communications Studies, as well as Information and
Communication Law. This literature is published in journal article format, where literature
review/state of the art sections tend to include accounts of diversity as a multidimensional concept.
The following sections will analyze how diversity is framed in both the Social Sciences and Computer
Science. We apply Entman’s [17] conceptualization of frames as promoting a certain problem
definition, cause, moral evaluation and solution.
5. Framing of Diversity in the Literature
Across the articles diversity is understood as an umbrella term, including different kinds of diversity,
such as viewpoints, content, sources, and ownership, which usually tend to be defined or connected
to specific academic definitions, such as the definition provided by [41]. In this definition diversity
is seen as a multi-dimensional concept, including (a) source diversity, (b) content diversity, and (c)
exposure diversity (p. 11). Boundaries between the different kinds of diversity tend to be blurred and
differ within the literature; for instance, some authors include viewpoint diversity within content
diversity, meaning that content diversity is achieved by providing a variety of content on a topic
from different perspectives. While in the theoretical grounding of the literature ownership and
source diversity are acknowledged, they tend to fade out in the practical logistics of the
operationalization of news recommendations for concrete experiments.
Two overall framings of the problem of NRSs can be observed. The first one frames diversity as
the solution to a problem of ‘information overload’. One article illustrates this by describing how:
‘Automated content recommendations are a common way to provide users of online media platforms
a way of navigating the abundance of information’ [7]. Another article states that: ‘With this
abundance of available content and the rapid pace of publishing, it becomes increasingly difficult for
readers to filter out and distinguish what is relevant to them’ [53]. The introduction of NRS in the
news industry thus brought up a clash of modus operandi: while on the one hand recommender
systems are designed to give users more of what they want (according to accuracy measures), with
the ultimate aim of avoiding information overload, on the other hand the role of journalism is to
provide individuals with an overview of different kinds of news. This clash is exemplified by this
article, which states that: “in a fragmented and rich information environment, algorithm-based
recommendation systems help users find relevant content” [5]. Thus, the problem is framed as one
of too much news for the user to navigate, and the proposed solution is NRS.
Secondly, the literature frames the problem as linked to problems of ‘filter bubbles’. Many, if not
all of the articles refer either directly or indirectly to the issue of filter bubbles, stemming from
Pariser’s book ‘the filter bubble’ from 2011. An indirect referring can be found in an article
highlighting that: ‘Systems have been criticized for introducing biases and being a potential threat
to an informed citizenry and the democratic discourse’ [5]. Another article highlights how: “During
the development of news recommender systems, there is currently a large focus on the predictive
power of an algorithm. However, this may unduly promote content similar to what a user has
interacted with before and lock them in loops of “more of the same.” [3].
The Computer Science literature rarely discusses the concept of diversity in itself; however, it is
widely recognized to be an important aspect for a well-functioning NRS, and efforts are deployed to
investigate how to better measure it and implement it. Just like in the Social Science literature there
is no consensus about a definition of diversity, in the Computer Science literature there is no
consensus about which metrics and formulas are more useful in measuring and implementing
diversity. In general, diversity in this literature is understood as the opposite of similarity, and it is
usually measured in terms of mathematical distance between topics or opinions. For example, a study
argues that their ‘findings can contribute to diversity-aware NRS’s to nudge users toward the
consumption of diverse topics and viewpoints [46]. Another example is an article, which ‘propose a
pre-filtering graph-based approach of extending the user profile to nudge him/her along a path
toward unseen news topics.’ [53]. The article further states that: ‘Our algorithm’s aim to improve
the topical diversity concerning the content that an individual interacts with’ [53]. Another article
compared two groups of readers in a real-time experiment on the Danish site eb.dk and explains that:
‘Drawing on the agenda-setting framework, we analyzed the coverage of political news, the salience
of political issues and actors prevalent during the election, and the diversity in exposure to topics,
actors, and issues between the two groups’ [15]. What is important to note is that most of the
literature deals with the issue of diversity within one single news site; therefore, the kinds of
diversity most discussed in relation to how news recommender systems work are content and
viewpoint diversity. Source diversity is mentioned in practical terms in connection with the
workings of recommender systems on news aggregating sites, or sites that recommend news from
(a variety) of different sources.
If, as Entman [17] explains, frames involve the managing of ´selection and salience´, the
prominence of discussions about how to best achieve a sufficiently diverse list of recommendations,
and how to balance accuracy and diversity measures signals that diversity is a prerequisite of a well-
functioning NRS. Only later articles concern themselves with evaluating the way diversity is
measured, what kind of diversity is implemented, which kinds of measures are employed and to what
effect. The most recent literature also takes into account other normative values when designing
NRS, such as the RADio framework introduced in [54], in which the authors develop values from
normative democratic theory and evaluate the NRS mathematically according to these normative
criteria, measuring for example alternative voices or the degree of fragmentation. But also, in these
more recent articles diversity as a prerequisite for a well-functioning democracy is rarely discussed
and most often assumed.
6. The Framing of the Normative Role of Journalism in the Literature
The normative role of journalism has traditionally been centered around its connection to
democracy. In 2005, Strömbäck [49] called the relationship between journalism and democracy a
‘social contract’ (p. 332), necessary because “Democracy requires a system for the flow of
information, for public discussion and for a watchdog function independent of the state” (p. 332). He,
then, proposed to leverage this connection to evaluate journalism´s work, by linking different models
of democracy to different normative expectations for journalism. Overall, the literature analyzed in
this article have divergent framings of the normative role of journalism.
While the framing of diversity as a solution seems to be quite stable in the Social Sciences
literature, the framing of diversity in the Computer Science literature evolved throughout the years
we have in our sample. The earlier literature (2011 until 2018) framed diversity as a necessary
solution because recommender systems tend to recommend more of the same content and users get
bored of receiving more of the same; recommender systems, then, needed to work to balance
accuracy measures (giving users content similar to the one they already interacted with) with
diversity measures. In this period, we see that the role of journalism is limited to not bore people,
essentially building the NRS according to market-driven values of making a better product, by
making users click on more news than they did before the NRS was implemented. The solution to
the problem of boredom is finding out what people are interested in and recommending more of that
content. An example is the article by Hsieh et al. [30] in which they “infer user’s interest from their
digital traces and create a user profile that has strong predictive power to the kinds of items that the
user will be interested in. Abbar et al. [1] incorporate sentiments and entities in the comments and
in the articles for the articles to achieve a more diverse recommendations.
Later articles, specifically the articles published after 2018, frame diversity as needed because
recommending more of the same content to users impedes the democratic role of news; optimizing
solely for clicks (or according to the content users have previously clicked) becomes not only boring
for the user, but also dangerous for democracy, because aiding both the spread of sensationalist
content and misinformation and the creation of filter bubbles. An article by Sonoda et al. [48]
explains how filter bubbles, which only provide biased information to users, are generated due to
excessive recommendations. There are several issues related to filter bubbles, including the biased
nature of the information obtained and the tendency for society to become polarized and divided
[48]. Similarly, Heitz et al. [26] argue that while little evidence of filter bubbles in online news
environments is found, user’s selective exposure and avoidance of opposing views could re-enforce
particular political stances. Thus, we see increasingly that the framing of the problem changes with
the introduction of Pariser’s filter bubble hypothesis [43] into NRS research.
An article even labels this ‘the normative turn in computer science, which scrutinizes the ethical
and societal consequences of recommender systems’ [23] This article is one of several focusing more
on transparency and users’ experiences of NRSs. In an experiment the researchers provided user
control mechanisms and through focus groups and think aloud interviews with the users showed
that reading history and flexible preference settings were valued by users [23].
Many of these articles are rooted in Helberger´s argument that diversity is not a goal in itself, but
a means to the goal of citizens´ participation in democracy [28], and that recommender systems can
be designed according to different democratic models [27]. This is especially visible in the articles
conceptually exploring the possibility of nudging users towards more diverse consumption patterns.
These articles celebrate the possibility recommender systems allow for and conceive it to be set in
motion when users do not click on diverse enough content. An article for example tested different
nudging mechanisms and concluded that these would enhance user’s sensemaking of the algorithmic
curation and the intention to seek out more diverse news was increased when the NRS processes
were explained to the user in the front end design [46]. Another example is an article based on a
systematic literature review distinguishing between different dimensions of news diversity,
including topical and source diversity, but also other dimensions such as viewpoints and structural
diversity [36]. Building on the notion of viewpoint diversity, recent works have attempted to use
stance detection to capture the viewpoint across multiple topics [14], as well as propose
multidimensional representations beyond stance [13]. The framing of diversity thus changed over
the years, from a perception of diversity as necessary for not driving the user away from the site to
more normative perceptions of diversity as a pre-requisite for a well-functioning NRS for citizen’s
democratic participation, including exposing the user to a broader set of views. Such a normative
view of journalism can be seen as related to normative role of journalism in the participatory and
the deliberative models of democracy [49].
Despite the more normative turn, many of the articles operationalize diversity as exposure to
diverse set of news items, while the more deliberate and participatory measures found in the article
by Loecherbach et al. [36] are rare. For example, an article by Lu et al. argues that the values of news
organizations should be included in the design of the NRS to counter the click driven optimization,
but they define the values as ‘the ability to surprise, to provide fresh and timely news, to yield more
diverse reading behavior, increasing the number of items read´ [37]. As such the article remains on
the level of providing more diverse topics, while normative values such as focusing on problem
solving, engaging citizens in public life or linking citizens together – all journalistic values from
participatory and deliberate models of democracy – are absent across the literature.
7. What frames are Left out and with What Consequences
The findings above point to a specific way of framing diversity in relation to a very specific
democratic role of journalism and a consequent way of operationalizing it in terms of exposure
diversity, or exposing individuals with a variety of news, on a specific news site. A framing of
diversity related to exposure to diverse topics can be seen as fulfilling the competitive model of
democracy, which Strömbäck describes as focusing on ‘political actors’ and ‘journalism acting as a
watchdog’ [49]. This means that much of the literature in our sample is leaving out frames of viewing
journalisms normative role in the deliberative or participatory models of democracy, which tend to
focus more on letting the citizen set the agenda, linking active citizens together or mobilizing
citizens’ participation in public discussions [49] (p. 341). Furthermore, the normative role of the
critical watchdog, which is a part of Strömbäck’s competitive and minimal normative model is not
taken into account in the literature, as diverse content exposure limited to topics and viewpoints
cannot secure exposure to critical journalism, exposing those in power for example.
While designing for diversity is important and necessary in the domain of news, the problem with
such an approach is also that exposing individuals to a variety of news does not guarantee that they
will click on and read diverse news. Audience research has historically shown that there is often a
gap between the encoded intent of the media production and the decoding of this content by the
audiences, who for example often resist this intent of the media organizations. Moreover, as
suggested by Einarsson et al. [16] thinking between the boundaries of one specific news site does
not take into consideration that individuals tend to have wider news repertoires and be aware of the
editorial lines of the media they consume. The news diet of individuals may already be quite diverse
in itself, not requiring ulterior diversification on the singular news sites. If an individual is used to
reading about politics on one news site and about sport on another news site, recommending them
with diverse news that do not relate to such topics may actually dissuade them from using the sites.
As noted by Bodó et al. [9]: “underlying the filter bubble discourse is thus yet another, more implicit
assumption, namely that diversity, and exposure to diverse news is inherently a good thing. It is
worth noting that this assumption is not self-evident. Diversity can compete with other, not less
important public or economic values, such as the need for reducing complexities, personal autonomy
of the audience, and the provision of information of personal importance to the audience” (p.208).
Operationalizing the framing of diversity solely in terms of the competitive democratic role of
journalism on one news site also comes with the normative assumption that especially political news
needs to be part of the individual´s main news diet. Indeed, concerns in the literature tend to be
centered around individuals being exposed to enough politically relevant news, so they can fulfill
their role as democratic citizens. The definition of ‘the political’ is however somewhat narrow, both
in discussions and operationalizations of content diversity and viewpoint diversity. The political
domain is actually the only domain where viewpoint diversity is operationalized, which usually
tends to be done by making a distinction between leftwing and rightwing news. For example, Heitz
et al. explains how they: “assessed two political dimensions—left/right and liberal/conservative—for
each user based on more than 20 questions” [26] (p.1715). An exception is an article by [25], which
proposes two-dimensional viewpoint representation consisting of a viewpoint’s stance on a nuanced
level which reflects the degree to which a viewpoint opposes or supports a given topic, and a
viewpoint’s logic of evaluation, which reflects the perspective behind the stance. Nevertheless,
securing exposure to different standpoints and thus multiple voices, does not take into account the
voices not even heard in the article dataset, which research into the lack of representation in media
research has highlighted for quite some years [20].
Even though the narrow distinction is most likely due to the limited ability of algorithms to detect
and recognize other kinds of political views, such as pro-choice/pro-life or body-positive/body-
shaming, it does not allow for an accurate operationalization or evaluation of the full scope of
diversity of viewpoints which are implicitly embedded in news articles and play a role in the opinion
formation of the individual. For example, the long history of framing analysis of media content has
shown how issues are framed in often very subtle ways through specific use of language across a
body of articles, for example framing women as emotional or immigrants as criminals. Such a
framing occurs across the media landscape over time and thus not only in the individual article and
provides the user with ‘different windows to the world’ [52], which is not taken into account in the
diversity-by-design approaches. A similar argument can be found in the work of Sax who found that
much of the literature draws on a narrow set of theories of liberal and deliberate democracy,
emphasizing how conflicts should be solved in a rational manner, while ignoring more agonistic
democratic models. Such an approach would stress the ineradicably of conflict and the need to make
conflict productive [44].
Moreover, the focus on leftwing/rightwing news tends to privilege and overestimate the
importance of the recommendation of hard political news, as opposed to other kinds of news, such
as cultural, lifestyle and service. This seems to remediate the struggle of scholars working with
different kinds of journalism for the recognition of the legitimacy of these kinds of journalism as
public spheres hosting democratic discussions [24]. The evaluation of journalism strictly in
connection to democracy where journalism is seen as watchdog, providing users with a variety of
content to stay informed, means that only specific kinds of journalism, often connected to hard news
reporting, have received thorough scholarly attention [51], while other kinds of journalism, such as
service or lifestyle journalism, have been “relegated to the margins of academic inquiry” [22] (p. 5).
However, some scholars have been arguing for the role of other kinds of journalism in the flourishing
of the public sphere. Bird [6] argues that the softer kinds of news serve as incentive for broader social
discussions and that “in its ritualistic, community-building role, trivial news allows us to interrogate
the morality and dialogue with others about shared values” (p. 501). Some scholars, such as Thomas
[51] have been working to anchor journalism in a different normativity than democracy, arguing
that the conception of journalism centered around hard news reporting “does not reflect the different
things that different journalisms do for different people” (p. 365). He proposes that the “first and
most basic objective of journalism is to be helpful, where to be helpful is to expand and improve the
opportunities of others” [51] (p. 365, emphasis in the original). Therefore, journalism can be
considered helpful when it opens up opportunities for people, which did not exist or were not
accessible before, through new knowledge or new skills [51]. In this we see the role of journalism
described as more participatory and deliberative, which is a framing left out in most of the literature
on NRSs, possibly because such forms of diversity are harder to measure and translate into
computational code. As Vrijenhoek et al. [55] note: “An increasing number of media organizations
have acknowledged the difficulties in translating their editorial norms into metrics that can inform
recommender system design.
Different kinds of journalism have different roles in the lives of individuals and what emerges
from our analysis is that diversity has been studied, designed, and evaluated only in connection to
one of those roles, the democratic participatory one, but very much related to how to stay informed
on a broad set of issues and topics. As such the operationalization of diversity secures a pluralist
media system on the media site level. As much as it is true that people do read the news to get
informed, it is also true that people read the news in order to form connections with their
communities and to stay updated on their personal interests. In this sense, news does not only cover
a strictly political and democratic role in the lives of citizens, but also a more “ritualistic, community-
building role” as argued by Elizabeth Bird [6] (p. 501). This kind of normative role is left out in by
the way diversity is framed in NRS literature, which instead only focus on the strictly political and
democratic role of news by making sure that individuals are exposed to enough hard political news
or different political opinions. According to [22]: “There has long been a normative focus” (p. 2) on
the link between journalism and politics, with a narrow understanding of what constitutes politics,
and an underestimation of the role that softer kinds of journalism (such as service and lifestyle) can
have on individuals and their contribution to the public sphere. As [21] claims, the agenda-setting
role of news works also with the more “private problems” brought up in lifestyle journalism, “which
means that the selection of specific problems and solutions has a civic impact” (p. 15). Such a framing
is less prevalent in the literature on NRS.
Lastly, framing the solution in terms of diversity-by-design implies that the need of diversity in
algorithmic news recommendation is an issue to be solved by computer scientists, rather than by
journalists and editors. Journalists have historically sought to balance the distribution of news
according to market driven values and more normative democratic values of ‘informing the public’
[39], but recent research suggests that such values are hard to integrate into the NRS design beyond
the exposure to different topics. Such an exposure does not secure though, however diverse it might
be, that news users are exposed to a commonality of issues across, and the risk might be that NRS
re-enforce the tendency for audiences to end up in different segmented ‘audience-islands’ [35], with
no bridges to connect them across the consumed content. In short diversity in exposure to topics or
opinions does not hinder the emergence of filter bubbles across the ‘inherently cross-mediated’ [45]
news landscape of the individual user even if framed as such in the literature. An article by
Vrijenhoek et al. [55] highlights how the data science literature frames this issue as related to one
specific site. The authors operationalize this as fragmentation, referring to ‘the extent that users are
frequently recommended the same items and therefor has a similar understanding of the content in
the system’ [5] (p.5). Ironically, exposing users to very diverse directions might have the opposite
effect, as the user loses track of the issues of larger concern, or it might push the user away from a
site, if the user has expectations of very niche content. This is likely the reason why news
organizations limit NRS to specific spaces on the site and leave the top spaces for editors to curate
manually, keeping ‘the human in the loop’ as a part of implementing algorithmic curation [40].
8. Conclusion
In this article, we have examined how diversity is framed in the large body of literature from
Computer Science and Social Science. Answering RQ1 we showed how diversity is framed as an
essential component of a well-functioning NRS, as it is framed as the solution to the filter bubble and
potential polarization problem. However, diversity is most often operationalized as diversity of news
topics and political viewpoints across the literature, of course with exceptions in more recent
literature in particular. This seems to indicate an increasing focus on other and broader set of values
as a part of the NRS design and evaluation, and future research should look more into how more
qualitative normative values can be operationalized. Some research has already started looking into
this, for example Lu et al. [37] investigated how organizational journalistic values can be
implemented in the NRS design. An issue with designing NRSs in response to the filter bubble
problem might be that user’s use multiple sites and platforms for news, which means that the
problem should not be solved on a single news site level.
The second section answered RQ2 and showed how the cause of diversity´s importance changed
over time in the literature from initially being rooted in the individual users’ lives, connected to their
potential boredom and information overload, to the more recent literature framing diversity in NRS
as connected to the democratic role of news. In this literature NRS are framed as providing an armor
against filter bubbles and bias in information seeking. An issue is that some commonality in the
recommendations is likely to ensure access to a broad public sphere, and future research should
examine how to balance commonalities in content across sites and services with potential bias in the
user’s information seeking practices.
Finally, the third section answered RQ3 by showing how the framing of diversity-by-design as a
solution, leaves other frames out for a more journalistic evaluation of their recommendation, and the
designing of NRS according to journalistic norms and values. But also more deliberative and
participatory democratic models should be taken into account, when designing NRS, thus looking
beyond mere diversity by exposure and towards diversity by consumption, and how the different
NRSs allows for more community-building roles in societies. Deliberate and participatory models of
democracy focus on letting the citizen set the agenda, linking active citizens together or mobilizing
citizens’ participation in public discussions, which might be important factors for news organizations
to take into account when developing NRSs. Beginning from the lived experiences of individuals
with journalism and the context in which news consumption takes place, for instance, would allow
researchers to move beyond the analysis of diversity within a single site, to take into account the
cross mediated news repertoires of users, as well as to gain a clearer picture of how individuals
engage with the collective concerns beyond the consumption of hard political news and the impact
of NRSs on such engagement.
Acknowledgements
This article was supported by Villum Foundation Denmark, Synergy Grant. We are grateful for the
feedback from the anonymous reviewers and from the feedback on earlier drafts provided by the
Journalism and Democracy research group at Roskilde University.
References
[1] Sofiane Abbar, Sihem Amer-Yahia, Piotr Indyk, and Sepideh Mahabadi. 2013. Real-time
recommendation of diverse related articles. In Proceedings of the 22nd international conference
on World Wide Web, May 13, 2013. ACM, Rio de Janeiro Brazil, 1–12.
https://doi.org/10.1145/2488388.2488390
[2] Eytan Adar, Carolyn Gearig, Ayshwarya Balasubramanian, and Jessica Hullman. 2017.
PersaLog: Personalization of News Article Content. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems, May 02, 2017. ACM, Denver Colorado USA, 3188–
3200. https://doi.org/10.1145/3025453.3025631
[3] Amelia H. Arsenault. 2017. The datafication of media: Big data and the media industries.
International Journal of Media & Cultural Politics 13, 1–2 (2017), 7–24.
https://doi.org/10.1386/macp.13.1-2.7_1
[4] Mariella Bastian, Natali Helberger, and Mykola Makhortykh. 2021. Safeguarding the Journalistic
DNA: Attitudes towards the Role of Professional Values in Algorithmic News Recommender
Designs. Digital Journalism 9, 6 (July 2021), 835–863.
https://doi.org/10.1080/21670811.2021.1912622
[5] Abraham Bernstein, Claes de Vreese, Natali Helberger, Wolfgang Schulz, Katharina Zweig,
Christian Baden, Michael A. Beam, Marc P. Hauer, Lucien Heitz, Pascal Jürgens, Christian
Katzenbach, Benjamin Kille, Beate Klimkiewicz, Wiebke Loosen, Judith Moeller, Goran
Radanovic, Guy Shani, Nava Tintarev, Suzanne Tolmeijer, Wouter van Atteveldt, Sanne
Vrijenhoek, and Theresa Zueger. 2021. Diversity in News Recommendations.
https://doi.org/10.4230/DagMan.9.1.43
[6] S. Elizabeth Bird. 2011. ARE WE ALL PRODUSERS NOW?: Convergence and media audience
practices. Cultural Studies 25, 4–5 (September 2011), 502–516.
https://doi.org/10.1080/09502386.2011.600532
[7] Sina Blassnig, Edina Strikovic, Eliza Mitova, Aleksandra Urman, Anikó Hannák, Claes De
Vreese, and Frank Esser. 2024. A Balancing Act: How Media Professionals Perceive the
Implementation of News Recommender Systems. Digital Journalism (January 2024), 1–23.
https://doi.org/10.1080/21670811.2023.2293933
[8] Balázs Bodó. 2019. Selling News to Audiences – A Qualitative Inquiry into the Emerging Logics
of Algorithmic News Personalization in European Quality News Media. Digital Journalism 7, 8
(September 2019), 1054–1075. https://doi.org/10.1080/21670811.2019.1624185
[9] Balázs Bodó, Natali Helberger, Sarah Eskens, and Judith Möller. 2019. Interested in Diversity:
The role of user attitudes, algorithmic feedback loops, and policy in news personalization.
Digital Journalism 7, 2 (February 2019), 206–229. https://doi.org/10.1080/21670811.2018.1521292
[10] Joshua A. Braun. 2015. This Program Is Brought to You By...: Distributing Television News
Online. Yale University Press.
[11] Angèle Christin. 2020. Metrics at Work: Journalism and the Contested Meaning of Algorithms.
Princeton University Press.
[12] Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson. 2014. Echo chamber or public sphere?
Predicting political orientation and measuring political homophily in Twitter using big data.
Journal of communication 64, 2 (2014), 317–332.
[13] Tim Draws, Oana Inel, Nava Tintarev, Christian Baden, and Benjamin Timmermans. 2022.
Comprehensive Viewpoint Representations for a Deeper Understanding of User Interactions
With Debated Topics. In ACM SIGIR Conference on Human Information Interaction and
Retrieval, March 14, 2022. ACM, Regensburg Germany, 135–145.
https://doi.org/10.1145/3498366.3505812
[14] Tim Draws, Karthikeyan Natesan Ramamurthy, Ioana Baldini, Amit Dhurandhar, Inkit Padhi,
Benjamin Timmermans, and Nava Tintarev. 2023. Explainable Cross-Topic Stance Detection for
Search Results. In Proceedings of the 2023 Conference on Human Information Interaction and
Retrieval, March 19, 2023. ACM, Austin TX USA, 221–235.
https://doi.org/10.1145/3576840.3578296
[15] Árni Már Einarsson, R. Helles, and S. Lomborg. 2024. Algorithmic agenda-setting: the subtle
effects of news recommender systems on political agendas in the Danish 2022 general election.
Information, Communication & Society (March 2024), 1–21.
https://doi.org/10.1080/1369118X.2024.2334411
[16] Árni Már Einarsson, Elisabetta Petrucci, Jannie Møller Hartley, and Stine Lomborg. In press. “I
must have clicked on something”: assesing news recommendations from an everyday
perspectives. Journalism Practice (In press).
[17] Robert M. Entman. 1993. Framing: Toward Clarification of a Fractured Paradigm. Journal of
Communication 43, 4 (December 1993), 51–58. https://doi.org/10.1111/j.1460-
2466.1993.tb01304.x
[18] Natalie Fenton. 2010. Drowning or Waving? New Media, Journalism and Democracy. In New
Media, Old News: Journalism & Democracy in the Digital Age. SAGE Publications Ltd, London,
United Kingdom, 3–16. https://doi.org/10.4135/9781446280010.n1
[19] Seth Flaxman, Sharad Goel, and Justin M. Rao. 2016. Filter Bubbles, Echo Chambers, and Online
News Consumption. PUBOPQ 80, S1 (2016), 298–320. https://doi.org/10.1093/poq/nfw006
[20] Elfriede Fürsich. 2010. Media and the representation of Others. Int Social Sci J 61, 199 (March
2010), 113–130. https://doi.org/10.1111/j.1468-2451.2010.01751.x
[21] Elfriede Fürsich. 2012. LIFESTYLE JOURNALISM AS POPULAR JOURNALISM: Strategies for
evaluating its public role. Journalism Practice 6, 1 (February 2012), 12–25.
https://doi.org/10.1080/17512786.2011.622894
[22] Folker Hanusch. 2012. BROADENING THE FOCUS: The case for lifestyle journalism as a field
of scholarly inquiry. Journalism Practice 6, 1 (February 2012), 2–11.
https://doi.org/10.1080/17512786.2011.622895
[23] Jaron Harambam, Dimitrios Bountouridis, Mykola Makhortykh, and Joris Van Hoboken. 2019.
Designing for the better by taking users into account: a qualitative evaluation of user control
mechanisms in (news) recommender systems. In Proceedings of the 13th ACM Conference on
Recommender Systems, September 10, 2019. ACM, Copenhagen Denmark, 69–77.
https://doi.org/10.1145/3298689.3347014
[24] John Hartley. 2000. Communicative democracy in a redactional society: the future of journalism
studies. Journalism 1, 1 (April 2000), 39–48. https://doi.org/10.1177/146488490000100107
[25] Taha Hassan. 2019. Trust and Trustworthiness in Social Recommender Systems. In Companion
Proceedings of The 2019 World Wide Web Conference, May 13, 2019. ACM, San Francisco USA,
529–532. https://doi.org/10.1145/3308560.3317596
[26] Lucien Heitz, Juliane A. Lischka, Alena Birrer, Bibek Paudel, Suzanne Tolmeijer, Laura
Laugwitz, and Abraham Bernstein. 2022. Benefits of Diverse News Recommendations for
Democracy: A User Study. Digital Journalism 10, 10 (November 2022), 1710–1730.
https://doi.org/10.1080/21670811.2021.2021804
[27] Natali Helberger. 2019. On the Democratic Role of News Recommenders. Digital Journalism 7,
8 (September 2019), 993–1012. https://doi.org/10.1080/21670811.2019.1623700
[28] Natali Helberger, Katharina Kleinen-von Königslöw, and Rob Van Der Noll. 2015. Regulating
the new information intermediaries as gatekeepers of information diversity. info 17, 6
(September 2015), 50–71. https://doi.org/10.1108/info-05-2015-0034
[29] Derina Holtzhausen. 2016. Datafication: threat or opportunity for communication in the public
sphere? Journal of Communication Management 20, 1 (January 2016), 21–36.
https://doi.org/10.1108/JCOM-12-2014-0082
[30] Cheng-Kang Hsieh, Longqi Yang, Honghao Wei, Mor Naaman, and Deborah Estrin. 2016.
Immersive Recommendation: News and Event Recommendations Using Personal Digital Traces.
In Proceedings of the 25th International Conference on World Wide Web, April 11, 2016.
International World Wide Web Conferences Steering Committee, Montréal Québec Canada, 51–
62. https://doi.org/10.1145/2872427.2883006
[31] Mathias Jesse and Dietmar Jannach. 2021. Digital nudging with recommender systems: Survey
and future directions. Computers in Human Behavior Reports 3, (January 2021), 100052.
https://doi.org/10.1016/j.chbr.2020.100052
[32] Glen Joris, Frederik De Grove, Kristin Van Damme, and Lieven De Marez. 2020. News Diversity
Reconsidered: A Systematic Literature Review Unraveling the Diversity in Conceptualizations.
Journalism Studies 21, 13 (October 2020), 1893–1912.
https://doi.org/10.1080/1461670X.2020.1797527
[33] Kari Karppinen. 2009. Rethinking media pluralism and communicative abundance. Observatorio
(OBS*) 3, 4 (2009).
[34] Kari Karppinen. 2013. Uses of democratic theory in media and communication studies.
Observatorio 7, (June 2013), 1–17.
[35] Johan Lindell and Jan Fredrik Hovden. 2017. Distinctions in the media welfare state: audience
fragmentation in post-egalitarian Sweden. Media, Culture & Society 40, 5 (2017), 639–655.
https://doi.org/10.1177/0163443717746230
[36] Felicia Loecherbach, Judith Moeller, Damian Trilling, and Wouter Van Atteveldt. 2020. The
Unified Framework of Media Diversity: A Systematic Literature Review. Digital Journalism 8, 5
(May 2020), 605–642. https://doi.org/10.1080/21670811.2020.1764374
[37] Feng Lu, Anca Dumitrache, and David Graus. 2020. Beyond Optimizing for Clicks: Incorporating
Editorial Values in News Recommendation. In Proceedings of the 28th ACM Conference on User
Modeling, Adaptation and Personalization, July 07, 2020. ACM, Genoa Italy, 145–153.
https://doi.org/10.1145/3340631.3394864
[38] Brian McNair. 2018. 8. Journalism as Public Sphere. In Journalism, Tim P. Vos (ed.). De Gruyter,
Berlin, Boston, 149–168. https://doi.org/10.1515/9781501500084-008
[39] Jannie Møller Hartley, David Mathieu, and Jannick Kirk Sørensen. 2023. Datapublics-the
construction of publics in datafied democracies (Bristol ed.). University Press.
[40] Lynge Asbjørn Møller. 2022. Recommended for You: How Newspapers Normalise Algorithmic
News Recommendation to Fit Their Gatekeeping Role. Journalism Studies 23, 7 (May 2022), 800–
817. https://doi.org/10.1080/1461670X.2022.2034522
[41] Philip M. Napoli. 1999. Deconstructing the Diversity Principle. Journal of Communication 49, 4
(December 1999), 7–34. https://doi.org/10.1111/j.1460-2466.1999.tb02815.x
[42] Philip M. Napoli. 2014. Automated Media: An Institutional Theory Perspective on Algorithmic
Media Production and Consumption: Automated Media. Commun Theor 24, 3 (August 2014),
340–360. https://doi.org/10.1111/comt.12039
[43] Eli Pariser. 2011. The filter bubble: How the new personalized web is changing what we read
and how we think. Penguin.
[44] Marijn Sax. 2022. Algorithmic News Diversity and Democratic Theory: Adding Agonism to the
Mix. Digital Journalism 10, 10 (November 2022), 1650–1670.
https://doi.org/10.1080/21670811.2022.2114919
[45] Kim Christian Schrøder. 2011. Audiences are inherently cross-media: Audience studies and the
cross-media challenge. Communication management quarterly 18, 6 (2011), 5–27.
[46] Donghee Shin and Shuhua Zhou. 2024. A Value and Diversity-Aware News Recommendation
Systems: Can Algorithmic Gatekeeping Nudge Readers to View Diverse News? Journalism &
Mass Communication Quarterly (June 2024), 10776990241246680.
https://doi.org/10.1177/10776990241246680
[47] M. Simon. 2024. AI in the News: Retooling, Rationalizing, and Reshaping Journalism and the
Public Arena. Tow Center for Digital Journalism at Columbia University.
[48] Atom Sonoda, Yoshifumi Seki, and Fujio Toriumi. 2019. Analysis of Factors that affect Users’
Behavioral Changes in News Service. In IEEE/WIC/ACM International Conference on Web
Intelligence - Companion Volume, October 14, 2019. ACM, Thessaloniki Greece, 35–42.
https://doi.org/10.1145/3358695.3360926
[49] Jesper Strömbäck. 2005. In Search of a Standard: four models of democracy and their normative
implications for journalism. Journalism Studies 6, 3 (August 2005), 331–345.
https://doi.org/10.1080/14616700500131950
[50] Cass R. Sunstein. 2009. Going to Extremes: How Like Minds Unite and Divide. Oxford University
Press.
[51] Ryan J. Thomas. 2019. Helpfulness as Journalism’s Normative Anchor: Addressing blind spots
and going back to basics. Journalism Studies 20, 3 (February 2019), 364–380.
https://doi.org/10.1080/1461670X.2017.1377103
[52] Gaye Tuchman. 1978. MAKING NEWS: A STUDY IN THE CONSTRUCTION OF REALITY. Free
Press.
[53] Stefaan Vercoutere, Glen Joris, Toon De Pessemier, and Luc Martens. 2024. Improving selection
diversity using hybrid graph-based news recommenders. User Model User-Adap Inter (June
2024). https://doi.org/10.1007/s11257-024-09399-w
[54] Sanne Vrijenhoek, Gabriel Bénédict, Mateo Gutierrez Granada, Daan Odijk, and Maarten De
Rijke. 2022. RADio – Rank-Aware Divergence Metrics to Measure Normative Diversity in News
Recommendations. In Proceedings of the 16th ACM Conference on Recommender Systems,
September 18, 2022. ACM, Seattle WA USA, 208–219. https://doi.org/10.1145/3523227.3546780
[55] Sanne Vrijenhoek, Savvina Daniil, Jorden Sandel, and Laura Hollink. 2024. Diversity of What?
On the Different Conceptualizations of Diversity in Recommender Systems. In The 2024 ACM
Conference on Fairness, Accountability, and Transparency, June 03, 2024. ACM, Rio de Janeiro
Brazil, 573–584. https://doi.org/10.1145/3630106.3658926
[56] Ida Willig. 2010. Constructing the audience: a study of segmentation in the Danish press.
northern lights: film and media studies 8, 1 (October 2010), 93–114.
https://doi.org/10.1386/nl.8.93_1
[57] Frederik J. Zuiderveen Borgesius, Damian Trilling, Judith Möller, Balázs Bodó, Claes H. de
Vreese, and Natali Helberger. 2016. Should we worry about filter bubbles? Internet Policy
Review 5, 1 (2016), 1–16. https://doi.org/10.14763/2016.1.401