Understanding How News Recommender Systems Influence Selective Exposure

Abstract

News recommender systems (NRSs) ofer benefits in the realm of information consumption and personalized news delivery. Yet, some critics argue that overly personalized news recommendations can pose a threat to democracy as these systems can potentially increase the occurrence of selective exposure, where individuals seek out political news that confirms their opinions at the expense of political news that contradicts their opinions. However, the conditions under which NRSs amplify or reduce selective exposure and the extent to which this happens are still poorly understood. Therefore, we ask: To what extent can NRSs influence the selective exposure behavior of news users? We present a preregistered online experiment to empirically test the impact of structural factors on selective exposure. We track user behavior on a news website equipped with two diferent versions of custom-made NRSs that are specifically designed to present news articles in such a way that we assume to nudge users towards increased or decreased selective exposure to like-minded or cross-cutting news. The findings indicate that the positioning and size of news articles have a notable impact on participants' behavior. Specifically, larger articles placed at the top tend to be more attractive for selection when they align with participants' attitudes. On the other hand, smaller articles placed at the bottom are less likely to capture participants' attention if they are attitude-consistent. The findings provide evidence that it may be possible to program NRSs to reduce selective exposure by promoting certain factors in the design of NRSs.

Keywords

• eol>News recommender system
• selective exposure
• survey experiment

1. Introduction and Related Work

At the heart of current debates on the ethical challenges of news recommender systems (NRSs) lies an important puzzle. Why do NRSs sometimes pose democratic detrimental efects, while other times do not show any signs of such consequences [ 1 ]? In this paper, we focus on one central aspect of such democratic consequences: NRS’ role in exacerbating or reducing selective exposure, that is, people’s tendency to seek out information in line with their political beliefs [ 2 ]. We build on a growing body of literature on NRS’ influences on selective exposure [ 3, 4, 1 ] and ask: To what extent can NRSs influence the selective exposure behavior of news users? Our contribution lies in conducting one of the initial empirical investigations of how custom-made NRSs can causally influence people’s tendency to seek out political news that aligns with their existing political beliefs.

For many years, social scientists have believed that individuals have a natural inclination to seek out information that aligns with their beliefs rather than challenges them [ 5, 2, 6 ]. The former is referred to as attitude-consistent, the latter as attitude-inconsistent. The phenomenon that news users, given the choice, tend to select attitude-consistent stories is commonly referred to as “selective exposure”. This is not to say that they never select attitude-inconsistent stories (after all, especially if an outrageous position is voiced, one may want to read it) – but on average, this is just less likely to happen. Researchers who study selective exposure have frequently voiced concerns that individuals, when presented with a variety of choices, tend to seek out political information they agree with, and as a possible consequence, isolate themselves in “echo chambers” where they mainly encounter content that aligns with their political beliefs [ 7 ]. Moreover, Pariser [ 8 ] and Dylko [ 9 ] argued that the use of algorithms to select content can increase the chance that selective exposure occurs by leading to a self-reinforcing feedback loop in which users are exposed solely to information that confirms their pre-existing belief. Pariser labeled these spaces, where individuals encounter like-minded content, “filter bubbles” [ 8 ].

Recommendation systems leverage machine learning algorithms to increase the level of relevant content over the noise that continuously grows as more and more content becomes available online [ 10 ]. The main evaluation metrics for the recent recommendation approaches are accuracy metrics [ 11, 12 ]. These metrics measure the algorithm’s performance by comparing its prediction against a known user interaction with an item, which means that the algorithm will recommend more and more items that resemble those a user has read before, and omit irrelevant information. If users engage in selective exposure behavior and mainly select political news content that aligns with their preexisting beliefs, and the recommender system mainly recommends such news to the user, the decisions made by this kind of recommender system can potentially increase the chance that selective exposure will occur over time [ 9 ]. Consequently, some argue that, at the societal level, accuracy-centric algorithms and overly personalized news recommendations can have negative efects such as filter bubbles, echo chambers, and political polarization, and caution that overly personalized recommendations can cause users to avoid counter-attitudinal information [ 13, 8 ]. This type of behavior, at the societal level, poses a potential threat to democracy [ 14 ].

While the negative consequences of NRSs on democracy are acknowledged, there is a significant body of research suggesting that these outcomes are not necessarily predetermined. For instance, studies combining insights from the field of computer science with psychology have shown that diversity in recommendation sets increases user satisfaction [ 14 ]. Empirical research on selective exposure [ 15 ] also indicates that news readers, in some instances, tend to consume both pro- and counter-attitudinal news information to gain knowledge about other perspectives. Another large-scale user study [ 16 ] indicates that selective exposure already exposes users to counter-attitudinal political beliefs. Several approaches were proposed over the past years to enhance the diversity of the recommendations created by a system [ 17, 18, 19, 20, 21, 22 ]. The news topics, writing styles, tags, perspectives, contexts, and ideologies are some of the factors that can be diversified in NRSs [ 14 ]. Möller and colleagues [ 7 ] proposes to incorporate diversity in NRSs as a function of news in democracy identifiable in topics, the tone in which topics are represented, categories, tags, the ratio of politically relevant content, and writing styles. In addition, there is a significant body of research suggesting that the extent to which NRSs pose a threat to democracy could depend on the conditions and factors under which NRSs amplify or reduce selective exposure. They claim that NRSs can be programmed to promote factors that reduce selective exposure because the NRSs are programmed by human beings and are thus dependent on the decisions surrounding the implementation and design of the technology [ 23, 24 ].

We still have only a limited empirical understanding of when and how NRSs influence selective exposure (under which conditions), and to what extent NRSs can do so [ 25, 1 ]. The exposure of a news reader to an article can be afected by many factors and conditions. For example, it can depend on the time when the recommendation happens, or readers can be more interested in news that happens closer to their location. Many moderators and factors of selective exposure have been studied in prior works such as efects of source cues, community ratings, and the number of views [ 26 ], user expectations regarding content diversity and depth of the recommendations [ 23 ], relative distance to assess how diverse recommendations set [ 7 ], and other factors summarized in [ 5 ].

Instead of solely focusing on whether NRSs causally amplify or reduce selective exposure, we argue that it is key to consider the factors and conditions under which these systems causally influence the likelihood of selective exposure occurring. To do so, we follow a categorization, based on a synthesis of the literature, which we have elaborated on in previous work [ 1 ]. Each category represents a diferent set of anticipated outcomes or viewpoints regarding how NRSs influence selective exposure.

The first perspective suggests that NRSs increase selective exposure. It is, for instance, supported by a randomized controlled trial demonstrating that Facebook’s algorithm was less likely to show content from outlets that users disagreed with, even when users actively expressed interest in those outlets [ 27 ]; and by an experiment using an online mock news site that found that NRSs indeed increased selective exposure [ 4, 28 ].

The second perspective argues that although selective exposure does occur in online services utilizing NRSs, NRSs themselves do not causally contribute to increased selective exposure. For example, even though not focusing on NRSs specifically, correlational evidence suggests that biases in news consumption primarily result from self-selection [ 29 ]. A recent literature review suggests that algorithmic selection ofered by digital platforms like search engines and social media generally leads to slightly more diverse news consumption, contradicting the “filter bubble” hypothesis [ 30 ].

The third perspective argues that NRSs can actively reduce selective exposure. Various studies support this viewpoint [ 31, 32, 33, 24 ]. For example, Bozdag and Hoven [ 31 ] argue that personalized information services can be designed in three diferent ways that efectively enhance the diversity of individuals’ media exposure choices. One approach involves ofering exclusively high-quality challenging items since readers may prioritize a high-quality, non-reinforcing item over a lower-quality reinforcing item. Another approach is to provide challenging information only when people show interest in it. Thus, when the personalization service identifies interest in a particular topic that the user has not recently explored, the service could recommend relevant items representing alternative viewpoints. The third approach involves reducing the cognitive dissonance that arises from encountering challenging information by facilitating easy access to information that supports individuals’ existing views whenever they are presented with challenging information. By providing challenging information alongside supporting information, individuals can navigate and reconcile conflicting perspectives efectively.

The shared notion among these perspectives is that NRSs can causally increase or decrease selective exposure under specific conditions. Crucially, all three perspectives can be valid, and a better understanding is needed on which factors and conditions their occurrence depends.

In our own earlier work, we proposed the Recommender Influenced Selective Exposure framework (RISE) [ 1 ], which addressed the challenges associated with conceptualizing NRSs as a causal variable. The framework proposed that the impact of an NRS on selective exposure is conditional upon its design objectives. It introduced a counterfactual causal question, exploring the conditions in which NRS amplifies or reduces selective exposure in online news environments, considering its intended goals. In other words, the extent to which an NRS increases or decreases the likelihood of selective exposure occurring depends on what it is intended to achieve – a point also echoed in interdisciplinary work on how to build human values into recommender systems [ 34 ], or in work that proposes new metrics to specifically translate normative perspectives on content diversity into recommender systems [ 35 ], thereby equipping them to reduce selective exposure on various levels.

Recently, work that is interested in understanding the conditions under which news selections happen in NRSs has moved towards study designs in which custom news sites or apps are made, and participants are – after filling in a questionnaire – asked to use these sites, while their interactions with its content are logged [ 36, 3 ]. Following this trend, we developed a platform that incorporated front-end and back-end systems for the experiment. The platform introduces an online news website featuring a front page with several news items. Users are provided with two options for each item: they can click on it to read immediately or add it to their read-later list for future access. We expect the processes of immediate reading and saving for later reading to be similar – selective exposure theory would not predict any diference. There are two reasons for studying both actions. First, bookmarking features are very common in modern web applications, yet have not been studied explicitly in the context of selective exposure. Second, pragmatically speaking, within a short-term experiment, requiring the participants to really read all articles they find interesting is not feasible, which makes it necessary to give them another possibility to signal that they would want to read the article. Arguably, immediately reading is a stronger signal than adding to a read-later list. Using this platform, we conduct an empirical test to examine the impact of nudging the salience moderator variable by making subtle adjustments to the presentation of choices in order to influence individuals’ decision-making processes. Salience refers to the level at which news articles are noticeable, prominent, and visible to users. It involves how news stories are featured, highlighted, or positioned within the website’s interface, influencing the attention of users. Salience can be influenced by multiple factors, including position, size, placement, or formatting of news articles.

For example, previous research has indicated that users’ first clicks are largely located in the top three positions when using list and grid layouts (e.g., [ 37 ]). Readers read a web news page by beginning in the top left, moving to the right diagonally downward, and the items located in the top left are viewed first and for a longer duration, and clicked more [ 38, 39, 40, 36 ]. Moreover, larger news items are viewed significantly earlier than smaller ones [ 41, 42 ].

Eriksson and Lundberg propose eight design recommendations for online newspapers, based on identified features that mediate a specific purpose and use between the publisher and the audience [ 43 ]. One of the design recommendations is to provide news valuation through text positioning and markers. The purpose of this recommendation is to indicate and recognize the value of news, and the form includes headline size and images. The recommendation suggests that the top position should be reserved for the highest-valued news content.

In addition, salience played a crucial role in our initial empirical test of the RISE framework as the framework was empirically demonstrated using simulated NRSs that aimed to influence selective exposure by enhancing the salience of a particular article.

Our hypothesis is that if the salience factor is applied, then there will be a higher probability of users clicking on, reading, and/or adding the treated articles to their read-later list, compared to a scenario when the salience factor is not applied. Drawing from the evidence on how salience influences the selection of news items and the degree of selective exposure, we first hypothesize, as a baseline, that: • H1: Readers are more likely to selectively expose themselves to attitude-consistent stories over attitude-inconsistent stories in terms of (a) clicking on and/or (b) adding articles to their read-later list when the order of news stories is random.

Compared to the baseline hypothesized in H1, we hypothesize that: • H2: Readers are more likely to selectively expose themselves to attitude-consistent stories over attitude-inconsistent stories in terms of (a) clicking on and/or (b) adding articles to their read-later list when they are placed at the top with a larger size compared to when the order of news stories is random (i.e., the baseline). • H3: Readers are less likely to selectively expose themselves to attitude-consistent stories over attitude-inconsistent stories in terms of (a) clicking on and/or (b) adding attitudeconsistent stories to their read-later list when they are placed at the bottom with a smaller size compared to when the order of news stories is random (i.e., the baseline).

If the findings of this experiment support the hypotheses, it will provide evidence that promoting certain design factors, such as salience in this case, in NRS can directly influence users’ selective exposure behavior. Consequently, it may be possible to program NRSs to reduce selective exposure by promoting the salience moderator, as mentioned in the presented hypotheses.

This paper is organized as follows: Section 2 presents our methodology, including the research design and the recommendation approach. The results of the online experiment are presented in Section 3, and their implications are discussed in Section 4, along with the limitations and future work.

2. Method

2.1. Dataset

The study builds upon the well-known TREC Washington Post corpus [ 44 ], which has been also used in prior studies on news recommendation systems [ 45, 46 ]. We used a single news source as we focused on selective exposure on the article level [ 47, 48 ]. The corpus contains 728,626 news articles and blog posts from January 2012 through December 2020. Because we mainly focused on how users choose news articles that favor their preferred political party, known as party cue [ 49, 50 ], we only considered articles categorized under the “political” category for determining selective exposure, reducing the corpus to 39,415 articles. We used the pool of excluded articles to pull so-called filler articles to fill non-political slots on the news site (see below). The articles are stored in JSON format and include the title, byline, kicker (a section header), author and their bio, date of publication, article text broken into paragraphs, and links to embedded images and multimedia (for 2012-2017 documents). The selected news articles were published on the Washington Post news website between 2013 and 2016 and had a text length between 295 and 1246 words (M=625, SD=238.9) with an estimated average reading time of 2.7 minutes (SD=0.9)—calculated based on the normal or typical reading rate of 300 words per minute, which has been widely cited in prior works (e.g. [ 51, 52, 53, 54 ]).

To measure whether the political articles contained information about a political party, we searched for news titles that included the names of political parties. The study was conducted in the United States, which has a dominant two-party system consisting of the Republican and Democratic parties. While other third parties exist, such as The Green Party, Libertarians, Constitution Party, and Natural Law Party, our research specifically concentrated on the Republican and Democratic parties, excluding third parties.

2.2. Participants

A total of 901 participants between ages 20 and 72 years were recruited via MTurk out of which 7441 qualified to be included in the analysis. The rest of the participants were excluded as they did not interact with the system at all, as requested in the task description. The participants were compensated on average with 1.8 USD which significantly exceeds the US minimum wage. Participants rated the study good on TurkerView2.

Out of the 744 participants ( = 37.4 years, = 9.99, Male = 55.5 %), 64.8% identified themselves as Democrats, 24.6% identified as Republicans, and the remaining participants (10.6%) were independents. Among the independents, 67.1% showed a leaning towards the Democratic Party.

2.3. Research Design

The study was conducted in the form of an online experiment presenting an artificial news website that features a front page with multiple news items. The front page is organized in a grid system, in which every article is displayed in its separate box, making it easy to find and clearly delineated. The boxes are arranged in rows and columns with three items per row, except for the first three rows. The first row contains one large item, while the second and third rows contain two smaller items each. The remaining rows contain items that are all the same size.

1Due to the logistics of the data collection and our available resources, we could not collect as many observations as originally planned in the pre-registration form.

2https://turkerview.com/requesters/ARXG1WHV70CR3-christoph-trattner

Each news item consists of a news headline, a picture, and a button to add the article to the read-later list (see Figure 1). The participant can either click on the news item to read the full story or click the button read-later to add the article to the read-later list without having to read the full story. The top of the page features a header bar where a countdown timer is located on the right-hand side indicating the remaining time until the experiment ends. While ideally, one would want to give people unlimited time, in the context of an MTurk experiment, the duration needs to be fixed. Without a minimum time, most workers would not spend enough time on the site, but without a maximum, a few of them would spend much more time than others, introducing unwanted biases in the data. Meanwhile, a dropdown menu labeled “Read later” is positioned on the left side which holds the news articles that the user had previously included in their list. The front page is visually represented in Figure 1.

The front page displays fifty news items. Each news item typically features a headline in a larger font size and a picture with a caption placed at the top (the same headline and picture used on the front page). Additionally, there is a byline that indicates the author of the article and is located below the headline and the body of the article which is divided into sections or paragraphs. The article pages also feature a header bar at the top of the page where the countdown timer is located on the right-hand side (the same counter on the front page), a website logo is placed in the center and can be clicked by users to return to the front page of the website and a back button is positioned on the left side for returning to the front page. The article pages also have a button labeled “Read Later”, which allows users to save the current article to a list for future reading. The article page is visually represented in Figure 2.

2.4. Procedure

The experiment starts with a welcome page that displays the title and the purpose of the research and explains the procedure and duration of the experiment. The purpose of the study was described to participants as follows: "We aim to study people’s news use. We ask you for your help by participating in an online study, in which you simply have to answer a few background questions about yourself and use a prototype of a news site we made." When users click the start button, they are taken to the following page that features a survey on demographics and political preferences. Once the survey is completed, users can click on a button to proceed to the next page where they will be directed to the front page of the website and given five minutes to browse, click on and read articles, or add some articles to their read-later list. According to the RISE framework [ 1 ], implementing a time constraint motivates participants to be more selective, which is crucial when studying selective exposure behavior. A time constraint is also important in terms of our desire to strike a trade-of between the cost of the experiment and the workload of the Mturk task on each participant. Upon the completion of the five-minute time limit, users will be automatically redirected to a page thanking them for participating and confirming that they have finished the study.

2.5. Recommendation Approach

We utilized a knowledge-based recommender system [ 55 ] as our approach for making recommendations. To construct the knowledge-based NRS for the experiment, we built up a dictionary consisting of the names of US politicians and their corresponding political parties, along with associated keywords for each party. Two political datasets were utilized to compile this dictionary. The first dataset, known as the U.S. Presidential Elections dataset [ 56 ], provides state-level returns for U.S. presidential elections spanning from 1976 to 2020. The second dataset is derived from Twitter usernames of American politicians [ 57 ]. From both datasets, Republican and Democratic candidates were extracted, along with their respective political party afiliations. After cleaning the datasets and removing duplicates, a total of 1954 politician names remained. Among these names, there were 1035 associated with the Democratic party and 919 associated with the Republican party.

We employed the created dictionary to select news articles from the Washington Post corpus. Specifically, we focused on articles that included the names of political parties or politicians mentioned in the dictionary within their titles or texts. For these selected articles, we enriched the news fields in each article by extracting the relevant names and adding new columns to the news article determining the number of occurrences of Democratic names in both the article title and text, as well as the number of occurrences of Republican names in the title and text.

The subsequent step requires determining whether the headline is positive or negative towards the mentioned party. To develop our system, three human annotators were employed to manually annotate the entire dataset. We asked them to indicate whether the headline was in favor of (pro) or against the party mentioned in the headline. This approach ensured that the news articles displayed on the front page accurately represented the intended political party and were framed in the desired manner. The inter-annotator agreement, measured using Krippendorf’s , resulted in a value of 0.62. This indicates a moderate level of agreement among the three human annotators. We only considered articles where all annotators agreed for our experiment. We opted for this approach to ensure that there is as little bias and uncertainty as possible in the experimental manipulation.

We also explored how using an of-the-shelf dictionary would perform this task. Given the known limitations of such approaches compared to more modern machine-learning approaches, this could be seen as an estimate for a lower bound of how automating this step would perform. (e.g., [ 58 ]). For this, we computed the polarity score for the article titles using NLTK’s PreTrained Sentiment Analyzer, VADER [ 59 ]. The polarity score, which falls within the range of -1.0 to +1.0, denotes the sentiment expressed in the titles. To capture selective exposure, we required articles that either positively or negatively framed a specific political party. Consequently, articles with a polarity score of 0 were eliminated from the dataset. Hence, if the title of an article contains more keywords related to one political party compared to the other, we assume that the polarity score is indicative of the sentiment towards that particular party. To generate the recommendation list, we divided the news articles into four categories: pro-Democratic, against-Democratic, pro-republican, and against-republican. For each category, we selected articles that had a higher occurrence of keywords related to the corresponding political party in their title compared to the other party. In the case of the pro-parties lists, we specifically chose articles with a polarity score greater than or equal to 0.5, indicating a predominantly positive sentiment towards the desired party. Conversely, for the against-parties lists, we selected articles with a polarity score less than or equal to -0.5, indicating a predominantly negative sentiment towards the desired party. As a result of this process, the recommendation lists consisted of 614 pro-Democratic news articles, 522 against-Democratic news articles, 1082 pro-Republican news articles, and 1096 against-Republican news articles. As our evaluation against the annotated dataset shows (Precision=0.45 and Recall=0.82), this approach does not perform equally well for both parties, which would be a requirement to rely on it for testing our hypotheses.

Consequently, as a final step, we required that all annotators and the automated approach agree. We selected 5 news articles that were labeled as pro-Democratic (pro-D) and 5 news articles that were labeled as pro-Republican (pro-R) from the annotated dataset. The selection criteria for these articles were that all three human annotators agreed on the same party afiliation, and their annotations matched the automatic labeling.

2.6. Treatment Conditions

We built one baseline and two knowledge-based NRSs that are designed to either increase or decrease selective exposure to like-minded news. Participants are divided into six diferent groups based on their political party favorability which are either “pro-Democratic” or “proRepublican”, If a participant self-identified as “independent” instead, we asked them in a followup question whether they lean more toward the Democratic Party or the Republican Party, and assigned them accordingly.

The baseline condition showed the fifty news articles in random order on the front page of the news site. The first recommender system (REC-1) was designed to increase selective exposure to pro-Democratic news and decrease selective exposure to pro-Republican news by featuring the five pro-Democratic news articles on the top, mixed with five filler news articles, and giving a larger size to the top 3 pro-Democratic news. While the five pro-Republican news articles featured on the bottom of the page mixed with five filler news articles.

The second recommender system (REC-2) was designed to increase selective exposure to pro-Republican news and decrease selective exposure to pro-Democratic news by featuring the ifve pro-Republican news articles on the top, mixed with five filler news articles, and giving a larger size to the top 3 pro-Republican news. While the five pro-Democratic news articles featured on the bottom of the page mixed with five filler news articles.

The front page features the same news articles in both NRSs and baseline conditions. ProDemocratic participants are divided randomly into three groups. The first group is assigned to the baseline, the second group is assigned to (REC-1), and the third group is assigned to (REC-2). The same for pro-Republican who are divided into three groups, the first one is assigned to the baseline, the second one is assigned to (REC-2), and the third group is assigned to (REC-1).

The six diferent conditions are: • D-BLINE: Pro-Democratic participants who are assigned to the baseline. • Pro-D-increase: Pro-Democratic participants who are assigned to (REC-1) aiming to increase their selective exposure to pro-Democratic news. • Pro-D-decrease: Pro-Democratic participants who are assigned to (REC-2) aiming to decrease their selective exposure to pro-Democratic news. • R-BLINE: Pro-Republican participants who are assigned to the baseline. • Pro-R-increase: Pro-Republican participants who are assigned to (REC-2) aiming to increase their selective exposure to pro-Republican news. • Pro-R-decrease: Pro-Republican participants who are assigned to (REC-1) aiming to decrease their selective exposure to pro-Republican news.

The procedure of the online experiment is depicted in Figure 3.

2.7. Measures

To assess selective exposure (represented by article clicks and adding to the read-later list), we asked participants to indicate their preferred political party, and we coded each political news article with its political afiliation to be able to match the news articles with the participant’s political attitudes 3. Then we recorded (a) clicks on the article to read it, and (b) clicks to add the article to the read-later list. We did so for each of the ten political news articles separately. We recoded these variables so that we have a measure where 1 is coded for articles that are attitude-consistent for the respondent and 0 is coded for articles that are attitude-inconsistent for the respondent. Thus, we can compare whether there is a diference in clicking on, or adding articles to the read-later list, between 1 “attitude-consistent articles” and 0 “attitude-inconsistent” articles.

To test the hypotheses, we used logistic regression models to estimate how articles click and add to the read-later list (dependent variables) vary by the attitude-consistency of the articles on the experimental treatment conditions (independent variable).

To address H1, we applied separate logistic regression models for the two dependent variables, used attitude-consistency as the independent variable, and restricted the observations to the baseline condition. H1a is considered supported if the participants demonstrate a statistically significant preference for clicking on articles that are labeled as attitude consistent rather than attitude inconsistent. H1b is considered supported if the participants demonstrate a statistically significant preference for adding articles that are labeled as attitude-consistent to their read-later list rather than attitude-inconsistent. This analysis addresses hypothesis H1.

We applied similar models to answer hypotheses H2 and H3. We ran two separate logistic regression models for the two dependent variables (articles click and add to the read-later list). For each of these models, we used two independent variables: attitude-consistency and treatment conditions (using the baseline as the reference category) and inserted an interaction term between the two independent variables. If participants are significantly more or less likely to click on and/or add articles that align with the hypothesized attitude to their read-later list, compared to a baseline condition, then hypotheses H2 and H3 are considered supported.

Participants were excluded if they had missing data on the dependent variables, which indicates that they did not click on any articles or did not add any articles to their read-later list.

3. Results

We collected the data for clicking behavior and adding to the read-later list behavior that shows how users interacted with each of the fifty articles. With this data, we were able to restructure and organize the information in a way that created multiple data points for each participant. Thus, we used multilevel GLM (logistic, Bernoulli) models, as we have several observations for each participant nested within their interactions with each individual article.

3We asked participants: "In politics, as of today, do you consider yourself a Republican, a Democrat or an independent?" with the options 1 "a Republican", 2 "a Democrat", 3 "an independent". If a respondent chose "3 an independent", we asked the following follow-up question: "As of today, do you lean more to the Democratic Party or the Republican Party?" with the options: 1 "lean Republican" or 2 "lean Democrat". We included leaners in the analysis but the results remain substantively similar if we exclude leaners

3.1. Selective exposure without NRS (H1)

Hypothesis (H1a) predicted a higher likelihood of clicking on attitude-consistent articles, in contrast to the attitude-inconsistent articles, when we focus solely on the baseline treatment condition.

While our results point in the expected direction (i.e., more likely to click on the attitudeconsistent than the attitude-inconsistent articles in the baseline condition), the efect is not statistically significant ( b=0.112, SE=0.125, p=0.37)4.

Analogous to H1a, we also expected to observe a higher likelihood of adding attitudeconsistent articles to the participants’ read-later list when examining only the baseline treatment condition (H1b). The results show that participants are indeed significantly more likely to add attitude-consistent articles to their read-later list rather than attitude-inconsistent articles (b=0.386, SE=0.138, p=0.005).

In sum, the results do not provide support for Hypothesis (H1a); however, they provide support for (H1b). Panel (a) in Figure 4 illustrates this with a plot of the marginal predicted mean of selective exposure on the article clicks and adding to the read-later list in the baseline condition.

4Note, however, that this result is sensitive to the decision to either include or exclude observations on clicks on the filler articles. If we exclude filler articles from the analysis, the efect is statistically significant ( b=0.82, SE=0.21, p<0.001). (Intercept) consistency treatment_increase treatment_decrease consistency× treatment_increase consistency× treatment_decrease Note: * < 0.1; * < 0.05; * < 0.01

Article click

Read later b

3.2. Selective exposure with salience-increasing NRS (H2)

Hypothesis 2 (H2) expected a greater probability of selective exposure when using an NRS that increases the salience of attitude-consistent article, compared to the baseline condition.

To test this hypothesis, we again estimated GLM multilevel models with attitude-consistency (0/1) as the independent variable. However, this time, we included additional binary variables for the treatment variables as well as for the interaction between treatment and consistency (Table 1). The coeficients are relative to the random baseline condition (see previous section). Panel (b) in Figure 4 illustrates the interaction efects between attitude-consistency and treatment conditions, where the dots represent the point estimates of the efects and the dots without a bar represent the reference category.

We can clearly see that participants are significantly more likely to click on and add attitudeconsistent articles to their read-later list rather than attitude-inconsistent articles, compared to the baseline condition, when attitude-consistent articles are placed at the top with a larger size. Thus, the findings provide support for H2a and H2b.

3.3. Selective exposure with salience-decreasing NRS (H3)

H3 expected a lower probability of selective exposure compared to the baseline condition when we employ an NRS that specifically decrease the salience of like-minded items.

Table 1 and Panel (b) in Figure 4 show that this is indeed the case. The positive efects of salience-increasing NRS found in H2 are clearly mirrored by the negative efects of saliencedecreasing NRS. Participants are significantly less likely to click on and add attitude-consistent articles to their read-later list rather than attitude-inconsistent articles, compared to the baseline condition, when attitude-consistent articles are placed at the bottom with a smaller size. Therefore, the findings provide support for H3a and H3b.

4. Discussion

News recommender systems are often criticized for the risk of reducing users’ exposure to content they disagree with, limiting their perspectives, and ultimately posing a threat to democracy. At the same time, recent work has also highlighted that it is possible to design NRSs in such a way that they avoid such detrimental efects [ 23, 24, 60, 35 ]. However, it is still not fully understood under which conditions NRSs promote or demote selective exposure [ 1 ].

To add another piece of evidence to this puzzle, we designed a system that manipulates the salience of selected articles, and measure how this afects selective exposure. In doing so, we add to recently emerging work that studies user selection processes in NRSs by creating news sites and news apps that closely resemble real-world news sites [ 36, 3 ]. This allows us to calculate an estimate of (a) the extent to which selective exposure occurs, and (b) how far this can be influenced by creating an NRS that increases or decreases the salience of the articles.

Our results demonstrate that participants are more likely to click on and add attitudeconsistent articles to their read-later list compared to attitude-inconsistent articles when the attitude-consistent articles are promoted through a salience recommendation. The placement and size of attitude-consistent articles play a significant role in influencing participants’ behavior, with larger and top-placed attitude-consistent articles being more appealing for selection. Conversely, smaller and bottom-placed attitude-consistent articles are less likely to attract participants’ attention.

The results of examining selective exposure behavior without the presence of an NRS (H1) provide insights into the clicking behaviors and adding attitude-consistent articles to the readlater list behaviors of participants in the absence of a recommender system. While we do not identify a statistically significant efect for clicking behavior, we observe a statistically significant efect for adding attitude-consistent articles to the read-later list. In accordance with the selective exposure literature [ 5, 2, 6 ], this suggests that in the absence of a recommender system, users display a preference for attitude-consistent articles. However, the absence of a statistically significant efect for clicking behavior implies that, when news articles are in random order, there may be other factors that are more important for influencing users’ immediate engagement with attitude-consistent or attitude-inconsistent articles. For instance, we do identify a statistically significant efect for clicking behavior if we exclude observations of clicking behavior on filler articles. To further study how the efect size of selective exposure relates to other factors, future work could take multiple routes. For example, next to political attitudes, the questionnaire could also ask for psychological traits. Alternatively, one could systematically vary other factors than the (textual) content, but also study how far recommending diferent pictures to illustrate the news stories afects user interactions.

The analyses of the efects of NRSs designed to increase (H2) or decrease (H3) selective exposure ofer evidence that supports the notion that diferent perspectives regarding the causal efect of NRSs on selective exposure can be valid under specific conditions. In line with literature that argues NRSs increase selective exposure [ 27, 4, 28 ], and others that argue NRSs can actively reduce selective exposure [ 31, 32, 33, 24 ], the results confirm that the impact of NRSs on selective exposure is conditional upon their design objectives. These results align with the premise of the RISE framework, which suggests that the extent to which an NRS increases or decreases the likelihood of selective exposure depends on what the NRS is designed to achieve. The results show that diferent design objectives can lead to diferent outcomes in terms of selective exposure. Specifically, the findings show that the NRS that is designed to increase the salience of attitude-consistent articles significantly increases selective exposure, while the NRS that is designed to decrease the salience of attitude-consistent articles significantly decreases selective exposure.

We see the main merit of our findings in the fact that they provide empirical evidence and a clear estimate of the extent to which selective exposure is influenced by diferent NRSs. While prior work on the RISE framework has tested simulated NRSs [ 1 ], we contribute the first test of the RISE framework that employs custom-made NRSs.

Despite the valuable insights gained from our study, there are certain limitations that should be acknowledged. In light of these limitations, our findings provide a fruitful starting ground for further investigations. First, our study focused on political news and participants’ preferences towards political parties as the target domain for examining selective exposure. However, it is essential to recognize that selective exposure behavior extends beyond political party favorability [ 2 ]. Moreover, the front page of the real online news site displays stories covering diverse topics on its homepage, not solely limited to those related to political parties. Future research should examine the generalizability of our findings across diferent domains to obtain a more comprehensive understanding of selective exposure tendencies.

Second, the moderate sample size ( N ≈ 230 in each condition) in our study may have implications the statistical power of the study, making it harder to detect significant efects or relationships. In addition, while Mturk provide a diverse sample of participants, it is not representative of the American adult population. Future research with larger and more diverse participant samples would help strengthen the validity and generalizability of the study’s conclusions.

In this study, we focused only on testing one factor (salience) in a relatively constraint setting. The next step should examine and assess a wide range of factors that can moderate selective exposure, and incorporate the most relevant factors into NRSs, thus enabling empirical testing of their impact on selective exposure.

Furthermore, this study has primarily examined the influence of NRSs on individuals’ selective exposure behavior within the controlled environment of an online experiment. Future research could expand the scope through field experiments to investigate how NRSs afect selective exposure in real-world settings over extended periods. This would provide valuable insights into the long-term efects of personalized NRSs on individuals’ media diets and political attitudes. Using field experiments would also solve another potential issue with our experiment. As it relied on a static corpus of news from 2013 to 2016, the news is by definition outdated. Even though the participants were instructed to use the news site just like they would use any other news site, they may have behaved slightly diferently because they may already have known some of the content. While using real-time news may alleviate this issue, it is considerably more complex to implement, as the pool of candidate stories of attitude-consistent and attitudeinconsistent news within a timeframe of one day is by definition orders of magnitude smaller than in our dataset.

Additionally, although this study has focused on the structural design of NRSs, there is a need to consider the role of algorithmic factors in shaping selective exposure. While we examined the influence of NRSs on selective exposure by modifying the interface, future research could explore how diferent recommendation algorithms impact users’ exposure to like-minded or cross-cutting news. By comparing the efects of various algorithms, such as content-based filtering, collaborative filtering, or hybrid approaches, researchers can identify algorithmic designs that mitigate selective exposure while still providing personalized news recommendations.

Mindful of these limitations, we would emphasize that our study demonstrates that NRS can reinforce selective exposure, which in turn could lead to democratically detrimental consequences where users are increasingly fragmented online. Hence, we encourage future work on responsible media technology [ 13 ] to develop NRS that does not increase selective exposure. Our study provides a possible roadmap, as we demonstrate that NRS could use similar recommendation techniques to demote rather than promote like-minded content, as the efect of placing such content at the bottom of the page is almost exactly the inverse of the efect of promoting it on top. Alternatively, NRS can promote attitude-inconsistent content on top. NRS development should consider implementing such mechanisms to make NRS align with democratic norms.

Finally, we conclude our discussion section by ofering a set of novel research questions as a roadmap for future work: • What is the impact of the choice of the recommender algorithm on users’ exposure to like-minded news? • To what degree can the research findings be generalized and applied in real-world settings? • How can other types of interface nudging strategies be integrated with NRSs to mitigate the influence of NRSs on selective exposure?

Acknowledgments

This work was supported by the NEWSREC Project (project number: 324835) and by industry partners and the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the Centers for Research-based Innovation scheme (project number: 309339).

References

[1] E. Knudsen , Modeling news recommender systems' conditional efects on selective exposure: evidence from two online experiments , Journal of Communication 73 ( 2022 ) 138 - 149 . doi: 10 .1093/joc/jqac047.

[2] S. Knobloch-Westerwick , Choice and preference in media use: Advances in selective exposure theory and research , Routledge Communication Series, Routledge, London, England, 2014 . doi: 10 .1017/9781315771359.

[3] L. Heitz , J. A. Lischka , A. Birrer , B. Paudel , L. Laugwitz , A. Bernstein , S. Tolmeijer , Benefits of Diverse News Recommendations for Democracy : A User Study , Digital Journalism 0 ( 2022 ) 1 - 21 . doi: 10 .1080/21670811. 2021 . 2021804 , publisher: Routledge.

[4] I. Dylko , I. Dolgov, W. Hofman , N. Eckhart , M. Molina , O. Aaziz, The dark side of technology: An experimental investigation of the influence of customizability technology on online political selective exposure , Computers in Human Behavior 73 ( 2017 ) 181 - 190 . doi:https://doi.org/10.1016/j.chb. 2017 . 03 .031.

[5] S. Smith , L. Fabrigar , M. Norris, Reflecting on six decades of selective exposure research: Progress, challenges, and opportunities , Social and Personality Psychology Compass 2 ( 2008 ) 464 - 493 . doi: 10 .1111/j.1751- 9004 . 2007 . 00060 .x.

[6] M. Lodge , C. S. Taber , Cambridge studies in public opinion and political psychology: The rationalizing voter , Cambridge University Press, Cambridge, England, 2013 .

[7] J. Moeller , D. Trilling , N. Helberger , B. Es , Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity , Information, Communication and Society 21 ( 2018 ) 1 - 19 . doi: 10 .1080/1369118X. 2018 . 1444076 .

[8] E. Pariser , The Filter Bubble: What The Internet Is Hiding From You , Penguin Books Limited , 2011 .

[9] I. B. Dylko , How Technology Encourages Political Selective Exposure, Communication Theory 26 ( 2015 ) 389 - 409 . doi: 10 .1111/comt.12089.

[10] L. B. Marinho , A. Nanopoulos , L. Schmidt-Thieme , R. Jäschke , A. Hotho , G. Stumme, P. Symeonidis , Social tagging recommender systems , in: F. Ricci , L. Rokach , B. Shapira , P. B. Kantor (Eds.), Recommender Systems Handbook , Springer US, Boston, MA, 2011 , pp. 615 - 644 . doi: 10 .1007/978-0- 387 -85820-3_ 19 .

[11] A. Gunawardana , G. Shani , A survey of accuracy evaluation metrics of recommendation tasks , Journal of Machine Learning Research 10 ( 2009 ) 2935 - 2962 .

[12] J. Herlocker , J. Konstan , L. Terveen , J. C. Lui , T. Riedl, Evaluating collaborative filtering recommender systems , ACM Transactions on Information Systems 22 ( 2004 ) 5 - 53 . doi: 10 . 1145/963770.963772.

[13] M. Elahi , D. Jannach , L. Skjaerven , E. Knudsen, H. Sjøvaag , K. Tolonen , Holmstad, I. Pipkin , E. Throndsen , A. Stenbom , E. Fiskerud , A. Oesch , L. Vredenberg , C. Trattner , Towards responsible media recommendation , AI and Ethics 2 ( 2022 ). doi:10.1007/ s43681-021-00107-7.

[14] S. Raza , C. Ding , News recommender system: a review of recent progress, challenges, and opportunities , Artificial Intelligence Review 55 ( 2022 ) 1 - 52 . doi: 10 .1007/ s10462-021-10043-x.

[15] J. Brundidge , Encountering “diference” in the contemporary public sphere: The contribution of the internet to the heterogeneity of political discussion networks , Journal of Communication 60 ( 2010 ) 680 - 700 . doi: 10 .1111/j.1460- 2466 . 2010 . 01509 .x.

[16] S. Flaxman , S. Goel , J. Rao , Filter bubbles, echo chambers, and online news consumption , Public Opinion Quarterly 80 ( 2016 ) 298 - 320 . doi: 10 .1093/poq/nfw006.

[17] K. Bradley , B. Smyth , Improving Recommendation Diversity, in: D. O'Donoghue (Ed.), Proceedings of the 12th National Conference in Artificial Intelligence and Cognitive Science , Maynooth, Ireland, 2001 , pp. 75 - 84 .

[18] C.-N. Ziegler , S. M. McNee , J. A. Konstan , G. Lausen, Improving recommendation lists through topic diversification , in: Proceedings of the 14th International Conference on World Wide Web, WWW '05 , ACM , New York, NY, USA, 2005 , pp. 22 - 32 . doi: 10 .1145/ 1060745.1060754.

[19] J. Wasilewski , N. Hurley , Incorporating diversity in a learning to rank recommender system ., in: Z. Markov , I. Russell (Eds.), The Florida AI Research Society , AAAI Press, 2016 , pp. 572 - 578 .

[20] N. Yadav , R. Kumar , A. Singh , S. Pal , Diversity in recommendation system: A cluster based approach , in: A. Abraham , S. K. Shandilya , L. Garcia-Hernandez , M. L. Varela (Eds.), Hybrid Intelligent Systems , 2021 , pp. 113 - 122 . doi: 10 .1007/978-3- 030 -49336-3_ 12 .

[21] M. C. Willemsen , M. P. Graus , B. P. Knijnenburg , Understanding the role of latent feature diversification on choice dificulty and satisfaction, User Modeling and User-Adapted Interaction 26 ( 2016 ) 347 - 389 . doi: 10 .1007/s11257-016-9178-6.

[22] M. Kunaver , T. Pozrl, Diversity in recommender systems, a survey , Knowledge-Based Systems 123 ( 2017 ) 154 - 162 . doi: 10 .1016/j.knosys. 2017 . 02 .009.

[23] B. Bodó , N. Helberger , S. Eskens , J. Möller , Interested in diversity, Digital Journalism 7 ( 2019 ) 206 - 229 . doi: 10 .1080/21670811. 2018 . 1521292 .

[24] N. Mattis , P. Masur , J. Mller , W. van Atteveldt , Nudging towards news diversity: A theoretical framework for facilitating diverse news consumption through recommender design , New Media & Society ( 2022 ). doi: 10 .1177/14614448221104413.

[25] H. Han , C. Wang , Y. Zhao , M. Shu , W. Wang , Y. Min, SSLE: A framework for evaluating the “Filter Bubble” efect on the news aggregator and recommenders , World Wide Web ( 2022 ). URL: https://doi.org/10.1007/s11280-022-01031-4. doi: 10 .1007/s11280-022-01031-4, publisher: Springer US ISBN: 0123456789 .

[26] S. Winter , N. C. Krämer , A question of credibility - efects of source cues and recommendations on information selection on news sites and blogs , Communications 39 ( 2014 ) 435 - 456 . doi: 10 .1515/commun-2014-0020.

[27] R. Levy , Social media, news consumption, and polarization: Evidence from a field experiment , American Economic Review 111 ( 2021 ) 831 - 70 . doi: 10 .1257/aer.20191777.

[28] I. Dylko , I. Dolgov, W. Hofman , N. Eckhart , M. Molina , O. Aaziz , Impact of customizability technology on political polarization , Journal of Information Technology & Politics 15 ( 2018 ) 19 - 33 . doi: 10 .1080/19331681. 2017 . 1354243 .

[29] R. Fletcher , A. Kalogeropoulos , R. K. Nielsen , More diverse, more politically varied: How social media, search engines and aggregators shape news repertoires in the united kingdom , New Media & Society ( 2021 ). doi: 10 .1177/14614448211027393.

[30] A. Ross Arguedas , C. Robertson , R. Fletcher , R. Nielsen , Echo chambers, filter bubbles, and polarisation: a literature review , Technical Report , 2022 .

[31] E. Bozdag , J. van den Hoven, Breaking the filter bubble: democracy and design , Ethics and Information Technology 17 ( 2015 ) 249 - 265 .

[32] R. K. Garrett , P. Resnick , Resisting Political Fragmentation on the Internet, Daedalus 140 ( 2011 ) 108 - 120 . doi: 10 .1162/DAED_a_ 00118 .

[33] N. Helberger , K. Karppinen , L. D'Acunto, Exposure diversity as a design principle for recommender systems , Information, Communication & Society 21 ( 2018 ) 191 - 207 .

[34] J. Stray , A. Halevy , P. Assar , D. Hadfield-Menell , C. Boutilier , A. Ashar , L. Beattie , M. Ekstrand , C. Leibowicz , C. M. Sehat , et al., Building human values into recommender systems: An interdisciplinary synthesis , arXiv preprint arXiv:2207.10192 ( 2022 ). doi: 10 .48550/arXiv.2207.10192.

[35] S. Vrijenhoek , G. Bénédict, M. Gutierrez Granada , D. Odijk , M. De Rijke, RADio - RankAware Divergence Metrics to Measure Normative Diversity in News Recommendations, in: Sixteenth ACM Conference on Recommender Systems , ACM, Seattle WA USA, 2022 , pp. 208 - 219 . doi: 10 .1145/3523227.3546780.

[36] F. Loecherbach , K. Welbers , J. Moeller , D. Trilling , W. Van Atteveldt , Is this a click towards diversity? Explaining when and why news users make diverse choices , in: 13th ACM Web Science Conference 2021 , ACM, New York, NY, USA, 2021 , pp. 282 - 290 . doi: 10 .1145/3447535.3462506.

[37] L. Chen , H. Tsoi , Users' decision behavior in recommender interfaces: Impact of layout design , CEUR Workshop Proceedings 811 ( 2011 ).

[38] S. Leckner , Presentation factors afecting reading behaviour in readers of newspaper media: an eye-tracking perspective , Visual Communication 11 ( 2012 ) 163 - 184 . doi: 10 . 1177/1470357211434029.

[39] D. Cerepinko , K. Hajdek , Impact of position and layout of news articles inside simulated ipad newspaper application on reading , Tehnicki Vjesnik 26 ( 2019 ) 941 - 946 . doi: 10 . 17559/TV-20171117130325.

[40] D. Zambarbieri , E. Carniglia , C. Robino , Eye tracking analysis in reading online newspapers , Journal of Eye Movement Research 2 ( 4 ) ( 2008 ) 1 - 8 . doi: 10 .16910/jemr.2. 4 .7.

[41] D. Lundqvist , K. Holmqvist , Bigger is better: How size of newspaper advertisement and reader attitude relate to attention and memory , in: ECEM Conferences , 2001 .

[42] K. Holmqvist , C. Wartenberg , The role of local design factors for newspaper reading behaviour - an eye-tracking perspective , volume 127 , 2005 .

[43] C. Ihlström Eriksson , J. Lundberg , A genre perspective on online newspaper front page design , Journal of Web Engineering (JWE) 3 ( 2004 ) 50 - 74 .

[44] NIST , Trec washington post corpus, 2019 . Data retrieved from: http://data .worldbank.org/ indicator/SP.DYN. LE00.FE .IN.

[45] A. Starke , S. Larsen , C. Trattner , Predicting feature-based similarity in the news domain using human judgments , in: 9th International Workshop on News Recommendation and Analytics (INRA 2021 ) co-located with 15th ACM Conference on Recommender Systems (RecSys 2021 ), 2021 .

[46] J. H. Jeng , A. D. Starke , C. Trattner , Towards attitudinal change in news recommender systems: A pilot study on climate change , 2023 .

[47] S. Knobloch-Westerwick , J. Meng , Looking the other way: Selective exposure to attitudeconsistent and counterattitudinal political information , Communication Research 36 ( 2009 ) 426 - 448 . doi: 10 .1177/0093650209333030.

[48] E. Knudsen , M. Johannesson , S. Arnesen , Selective exposure to news cues: Towards a generic approach to selective exposure research , Working Paper ISSN 2535-3233 , University of Bergen, Norway, 2017 .

[49] S. Iyengar , K. S. Hahn , J. A. Krosnick , J. Walker , Selective exposure to campaign communication: The role of anticipated agreement and issue public membership , The Journal of Politics 70 ( 2008 ) 186 - 200 . doi: 10 .1017/S0022381607080139.

[50] M. F. Mefert , T. Gschwend, When party and issue preferences clash: Selective exposure and attitudinal depolarization , in: Annual conference of the International Communication Association , Phoenix., 2012 .

[51] P. Aaron , Dyslexia and hyperlexia: Diagnosis and management of developmental reading disabilities , volume 1 , Springer Science & Business Media , 2012 .

[52] S. Andrews , From inkmarks to ideas: Current issues in lexical processing , Psychology Press, 2010 .

[53] A. Whimbey , J. Lochhead , R. Narode , Problem solving & comprehension, 7 ed., Routledge , 2013 . doi: 10 .4324/9780203130810.

[54] J. Yaworski , Getting ahead: Fundamentals of college reading, Longman Publishing Group, 2005 .

[55] A. Felfernig , G. Friedrich , D. Jannach , M. Zanker , Developing constraint-based recommenders , in: F. Ricci , L. Rokach , B. Shapira , P. B. Kantor (Eds.), Recommender Systems Handbook , Springer US, Boston, MA, 2011 , pp. 187 - 215 . doi: 10 .1007/ 978-0- 387 -85820- 3 _ 6 .

[56] M. E. Data , S. Lab , U.S. President 1976 - 2020 , 2017 . doi: 10 .7910/DVN/42MVDX, data retrieved from: https://doi.org/10.7910/DVN/42MVDX.

[57] A. Samoshyn , Us politicians twitter dataset, 2020 . Data retrieved from: https://www.kaggle. com/datasets/mrmorj/us-politicians -twitter-dataset.

[58] W. van Atteveldt , M. A. van der Velden , M. Boukes, The Validity of Sentiment Analysis:Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms , Communication Methods and Measures 00 ( 2021 ) 1 - 20 . URL: https: //doi.org/10.1080/19312458. 2020 . 1869198 . doi: 10 .1080/19312458. 2020 . 1869198 , publisher: Routledge.

[59] C. Hutto , E. Gilbert, VADER: A parsimonious rule-based model for sentiment analysis of social media text , Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media 8 ( 2014 ) 216 - 225 . doi: 10 .1609/icwsm.v8i1. 14550 .

[60] S. Vrijenhoek , M. Kaya , N. Metoui , J. Möller , D. Odijk , N. Helberger , Recommenders with a mission: assessing diversity in newsrecommendations , in: Proceedings of ACM Conference (Conference'21) , Association for Computing Machinery, 2021 . URL: http: //arxiv.org/abs/ 2012 .10185.