mender Systems - Results From an Experimental User Study

Abstract

Recommender systems play a pivotal role in curating personalized news environments based on user preferences. However, there is a growing demand to go beyond mere accuracy and consider the societal impact of these systems. In this paper, socially responsible news recommender systems are designed to promote news consumption diversity. This approach incorporates digital nudges, leveraging cognitive heuristics and biases, to steer user decisions toward diverse news articles. The study evaluated the efectiveness of feedback and social norms nudges in a simulated news recommender environment. A sample of n = 117 participants completed an online survey and engaged in news article selection trials. An A/B test was designed based on a set of diversified articles and the target and its selection rates were compared within the control, feedback nudge, and social norms nudge groups. The findings reveal that the feedback nudge did not significantly impact article selection, potentially due to methodological limitations. However, the study demonstrated that the social norms nudge significantly influenced the selection of the target article. These results suggest that digital nudging can efectively increase the consumption of diverse news in recommender systems, aligning with democratic values. The findings emphasize the importance of deepening the discussion on digital nudges to promote diversity and tolerance in news consumption. By fostering responsible news consumption, recommender systems can positively impact society.

Keywords

• psychology-aware recommendations
• user-interface design
• digital humanism

-

CEUR ceur-ws.org

1. Introduction

Recommender systems (RSs) collect, analyze, and integrate data about users’ likes, interests, and previous online behaviors to filter and customize the information presented to them, thereby facilitating navigation and decision-making in digital spaces [ 1 ] [ 2 ]. RSs therefore organize digital choice environments according to data-based user-profiles and thereby have the potential to influence people’s decisions and behaviors in virtual spaces. The state-of-the-art indicates that traditionally, the primary objective of automated recommendations has been to predict items of maximal relevance and interest to users [ 3 ] by focusing on the accuracy of the suggestions, thereby optimizing personalization and increasing recommendation satisfaction [ 1 ] [ 4 ]. The CEUR Workshop Proceedings focus on generating accurate recommendations has been particularly critical in traditional news recommender systems (NRSs), which are designed to maximize user satisfaction and engagement, and at least partly, to increase revenue for the news provider [ 5 ] [ 4 ]. NRSs “automatically (de)select and (de)prioritise news articles [...], [thus] increasingly determine the accessibility of digital media content” [ 6 ], and, as a result, influence what news content is consumed by individuals [ 6 ].

However, when solely considering users’ existing likes and interests to generate maximally accurate recommendations, users may find themselves in echo chambers and filter bubbles, which may cause a lack of media pluralism [ 6 ], i.e., a lack of exposure to and consumption of diverse information, ideas, and viewpoints [ 6 ]. Selective exposure in recommender systems can undermine democratic and socially relevant values, leading researchers to advocate for the development of humanistic and psychology-aware systems that go beyond accuracy and address issues such as diversity [ 5 ] [ 7 ] [ 8 ] [ 9 ]. Incorporating ”socially responsible designs” in recommender systems can benefit individuals and society [ 5 ] [ 10 ] [11] [ 8 ]. More specifically, diversified NRSs may promote a deliberative, or discursive, model of democracy, which encourages civic interest, critical thinking, and open debate about diverse viewpoints and opinions that extend beyond the individual [ 7 ]. Within this model, the responsibility of media outlets exceeds the presentation of purely personalized information by additionally aiming at providing the user with content that would give them a broader perspective on any given issue to improve their tolerance and open-mindedness and stimulate active deliberation within a society [ 7 ]. In light of this, it is assumed that recommendations can be enhanced by employing personalized nudges, for example by incorporating normative messaging with the purpose of making diversity norms more salient to the user [ 4 ], that is, to remind people of certain behaviors that are regarded as desirable in the society they are part of. Additionally, recommendations can reveal the divergence between the user’s behavior and socially approved, democratic behavior via feedback that is visible to the user and thereby prompts them to shift course in their actions and decisions.

While recommender systems already influence decision-making through recommendations, the use of digital nudging can further impact user choices [ 3 ] [ 10 ] [ 4 ] [11] [12]. Thus far, nudges have primarily been employed and studied in ofline settings [ 3 ] [13]. However, some researchers have argued that RSs themselves can be regarded as nudges [ 3 ]. This paper aims to explore how digital nudges can guide users toward more diverse news consumption, particularly in the context of news recommender systems.

This submission summarizes the findings of an interdisciplinary user study on feedback nudges and social norm nudges, two diferent types of nudges that were investigated for their potential to diversify news consumption [ 4 ].

From our analysis, the following efects of digital nudges on diversified news consumption emerged and are summarized in the following: • Socially responsible RSs interventions led to a shift in user behavior toward diversified news consumption. An increase in the selection of diversified articles was observed following the implementation of diversity-enhancing digital nudges in news recommendations, indicating the efectiveness of nudges on user decisions in digital spaces. • Behavior change regarding news choices was shown more strongly in response to normative messaging in comparison to behavior feedback. Whereas reminding users of social norms significantly impacted diversified news consumption, generalized feedback on biased consumption tendencies only slightly increased user choices for diversified items.

This paper contributes to the field of recommender systems by introducing socially responsible news recommender systems that promote news consumption diversity. The study evaluates the efectiveness of feedback and social norms nudges in a simulated news recommender environment, therefore allowing for insights into user behavior and providing an understanding of the potential of digital nudging to increase the consumption of diverse news, aligning with democratic values.

2. Related Work

2.1. Digital Nudging

The concept of nudging was introduced by Richard Thaler and Cass Sunstein and refers to the deliberate design of a choice environment to influence and guide people towards specific decisions and behaviors that are better for the individual while still preserving their freedom of choice [13]. Nudges thus “alter people’s behavior in a predictable way” [13] by designing a decision environment in which certain situational and contextual factors steer decisionmakers towards pre-defined options, thereby afecting the choices they make [ 11]. Nudging as conceptualized by Thaler and Sunstein can be considered a values-based approach to behavioral change as it aims at leading to behaviors that are more beneficial than previous, habitual modes of action for the target audience [13]. Thus, nudging constitutes a way of influencing people’s behavior towards what is perceived as beneficial [ 14]. This highlights the fact that nudges are always tied to an overarching goal, which is to be achieved through the nudge intervention. The external context of a decision situation, i.e., the choice architecture, or choice environment, can activate specific cognitive patterns that can result in the adoption of the nudged behavior [ 15]. Nudges therefore harness people’s cognitive limitations by establishing choice environments that appeal to cognitive heuristics and biases, thereby increasing the likelihood that individuals perform specific pre-defined decisions and behaviors [ 11]. In digital environments, nudges are primarily employed at the user interface [14] [16]. As users are always afected by the architecture of online recommendations [16], nudges can be implemented rather easily in the digital world by making subtle changes to the organization and presentation of recommendations, i.e., to the choice environment. Thus, digital nudges can be defined as subtle design elements in the user interface. However, it needs to be noted that digital choice environments should only be regarded as nudges when they utilize design elements that purposefully influence users in predictable and meaningful ways. This can be achieved by not simply providing suggestions but by ”making the user stretch” [ 10 ], thereby changing the user’s typical behavior in a beneficial way that is in line with the nudging goal. To summarize, digital nudging in recommender systems provides an opportunity to steer users toward relevant items while simultaneously enhancing their decisions and behaviors to achieve an overarching goal.

2.2. Nudges in News Recommender Systems

According to Mattis et al. [ 4 ], article selection in virtual news environments strongly depends on the presentational factors of recommendations. Considering this, digital nudges are critical in guiding users’ news consumption behavior through altering the user interface and can be designed to facilitate consumption diversity [ 4 ]. In their study, Mattis et al. [ 4 ] propose a theoretical framework for tailored diversity nudges to encourage the consumption of diversified news on platforms that autonomously curate content. According to their research, labeling a news article with information that its content counts as diverse and suggesting through feedback that the article would broaden the reader’s perspective may facilitate the selection of more diverse articles, resulting in a more balanced news diet. In addition, a diversity-aware news recommender algorithm’s utility and side efects were examined in an experimental study by Heitz et al. [ 5 ]. They found that diversity-optimized recommendations outperform user preference-based approaches and are related to more tolerance for competing viewpoints, especially among politically conservative users. According to their study, diversity in news recommendation algorithms could depolarize democracies.

Feedback Nudges Utilizing feedback nudges appears particularly promising for NRSs due to the baseline assumption of selective exposure, or confirmation bias. Selective exposure theory in the context of news recommendations refers to research on cognitive dissonance [17]. The theory posits that people prefer articles that resonate with their opinions and tend to avoid information that could question their prevailing beliefs in order to prevent an uncomfortable feeling, defined as cognitive dissonance. As a result, people will adhere to confirmation bias [ 4 ], which becomes an individually and socially relevant issue when considering the repercussions of such selectivity on diversity, tolerance, and empathy – values that can be regarded as essential in an increasingly complex world. By providing users with feedback in relation to their previous reading behavior, feedback nudges may encourage exploration outside of confirmatory filter bubbles, resulting in more diverse news consumption [ 6 ].

Social Norm Nudges Social norm nudges appeal to people’s fundamental desire to follow others’ behavior, i.e., to conform to the majority [ 3 ] [ 4 ]. By providing information that reminds people of common diversity values when presenting diversified news recommendations, individuals are guided towards selecting the recommended item as the presented cues influence their decision-making by providing social proof and grounds for justification for conducting a particular behavior, i.e., selecting the recommended item [18] [ 3 ] [11]. NRSs can, for example, incorporate pop-ups, messages, or labels when recommending diverse news articles that highlight that these articles align with norms such as good citizenship and tolerance [ 4 ], thereby indicating that diverse news consumption is regarded as socially desirable [19].

3. Method

3.1. Hypotheses

To examine the efectiveness of digital nudges for increasing news consumption diversity, a research experiment was conducted to test the proposed hypotheses. To evaluate if feedback and social norm nudges influence users’ news consumption diversity, an experiment was conducted via a digital survey. For the design of the nudges needed for the proposed experiment, Meske and Potthof’s [ 14] DINU-Model was consulted. This three-phase model was designed to guide choice architects in the process of constructing nudges and segregates the development and implementation of a digital nudge into the following steps: analyzing, designing, and evaluating [14]. Moreover, each of these phases combines a variety of steps that can also be found in other models, primarily Schneider et al.’s [12] four-step model. As part of the analysis stage, increasing news consumption diversity was defined as the primary objective of this study’s nudge implementation. Moreover, developing a diversity-sensitive news recommender nudge must include examination of the behavioral patterns, i.e., the underlying information processing mechanisms that “subconsciously influence people’s behavior and decision making” [ 10 ]. This study thus addresses confirmation bias on the one hand and herd instinct bias on the other hand. Due to methodological limitations, the experiment could not be conducted using a fully interactive website as would be the case in traditional A/B testing. Instead, a feedback and social norm nudge were designed and implemented in an online survey format to test the efect of these diversity-enhancing nudges, and the evaluation was conducted by utilizing experimental A/B testing via SoSci Survey [20]. In line with the research question and hypotheses of this paper, the following nudges were designed for the suggested experiment, see Figure 1. Additionally, to gain some insight into trends regarding the news sources utilized most frequently for news consumption in today’s media landscape, a brief questionnaire was presented to study participants in which they reported their primary news source. Moreover, in the questionnaire, participants were also asked to evaluate the importance of diverse news presentation and consumption, which would then be assessed as indicators of readers’ value orientation.

Feedback Nudge The desired article was presented with a feedback prompt providing information on users’ previous reading behavior to nudge people away from confirmation bias and towards broadening their perspective in terms of diverse news consumption. Social Norm Nudge The desired article was marked with a textual prompt, reflecting the descriptive norm in combination with the presentation of a symbol of unification (image of joined hands), marking the injunctive norm as a united improvement of socially and democratically relevant issues. Thus, injunctive and descriptive norms concerning diverse news consumption were employed to trigger the herd instinct bias.

Firstly, we looked into the potential of feedback nudges. A greater diversity of news sources may be consumed as a result of feedback nudges that stimulate interest outside of confirming iflter bubbles [ 6 ]. Combining a recommendation for a more diverse article with a reflective message about the individual’s prior news consumption could cause them to reflect on their article selection and push them in the direction of selecting a more balancing news item because people are frequently unaware of the biases and efects of their news consumption. Considering this, the following hypothesis was derived for further analysis within the research experiment. • Feedback nudges H1: Diversity-enhancing feedback nudges increase consumption of diverse news in news recommenders.

Secondly, social norm nudges build upon the empirical evidence harnessed mainly from psychological studies in the last decades that people are inclined to orient themselves towards the behavior of others to gain social approval and avoid disapproval [11] [19]. Normative messaging can be incorporated into recommendations to make diversity norms salient and remind individuals of desirable behaviors in their society [ 4 ]. Social norm nudges, which involve reminding people of injunctive norms, i.e., what people think they should do based on their perception of approved behaviors by others [? ] and descriptive norms, i.e., what people actually do in a given situation based on the observed behaviors of others [? ], to appeal to their inclination to conform, can guide individuals towards selecting diversified news recommendations [ 3 ] [ 4 ]. To increase news consumption diversity, such nudges could be employed to mark articles that difer from the users’ typical preferences, thereby nudging individuals to select articles that cover a broader range of topics and viewpoints than they would typically consume in a traditional recommendation environment. Highlighting and reminding readers of society’s and other people’s values may therefore be an efective tool to influence individuals’ decisions in news recommenders, which led to the development of the following hypothesis: • Social norm nudges H2: Diversity-enhancing social norms nudges increase consumption of diverse news in news recommenders.

3.2. Study Design

Experiment To test the research hypotheses, a web-based controlled experiment, also commonly referred to as a single-factor A/B test was conducted within a survey. A/B testing was selected as it is frequently employed in digital environments to compare the outcome of a control group to the outcome of a variant group to explore a change in the dependent variable that can be attributed to a modified independent variable [ 21]. As such, this A/B test was conducted to examine the causal relationship between article selection and the two nudges under examination [21]. Two separate A/B tests were conducted in this online experiment using SoSci Survey [20] to examine the impact of individual nudges on article selection. Test one compared the control group with the feedback nudge group (H1), and test two compared the control group with the social norm nudge group (H2). The experimental survey was designed to simulate a virtual news environment by presenting a variety of articles. Participants were randomly assigned to the control or experimental groups, and instructed to browse through the presented article recommendations (titles and brief introductions) and select the article they would like to read fully. To obtain more valid results, participants ran through three consecutive trials, each on a separate topic: climate, migration, and veganism. All groups were presented with the same four articles per topic but difered in target article display depending on group assignment, i.e., the inclusion of the examined nudges for the target article. Whereas in the control condition, the diversified article, or target article, was simply presented amid the other articles without a nudge, either a feedback or social norms nudge was displayed alongside the target article in experimental conditions 1 and 2 respectively. To control for serial position bias, in which the ordering of items may trigger a primacy or recency efect [ 12], the target article was presented as option 3 across all groups and trials. The target article selection rate, as tracked by SoSci Survey, was measured as the metric of interest to analyze user behavior and statistically test the influence of the examined nudges on article selection. Due to its data analytical limitations, SoSci Survey was primarily utilized to collect the raw data for further evaluation of conversion rates using CXL’s A/B test calculator [22]. Since the experiment was conducted to examine whether there is a diference between two independent groups (control and feedback nudge in test 1, and control and social norms nudge in test 2) and the measured dependent variable is not metric, the statistical test selected for the sample size calculation as well as for further analysis was a Mann-Whitney U-Test.

Sample Participation in the study was voluntary and the participants were recruited via snowballing, primarily over private messaging and social media channels. To be included in the study, participants had to be older than 18 years. Inspired by Joris et al.’s [23] research design, to collect data for the study, the experimental survey was designed to simulate a virtual news magazine in which a list of articles was recommended on a given topic. This simulated news site can be considered the choice environment within which study participants’ news consumption decisions and behaviors were monitored and measured by tracking their article selection. Looking at previous research and adhering to conventional statistical metrics, a total sample size of n = 74 per test, for an even distribution of n = 37 per group was calculated given a power of .8, a statistical significance of .05, and a moderate efect size (d) of 0.6 [ 24] [23]. Material The stimulus material, i.e., the news articles, were extracted from the digital archives of the Austrian weekly newspaper Der Falter on the basis of topic-specific keywords, e.g. Klimakrise (climate crisis), Klimakatastrophe (climate catastrophe), Migration in Österreich (migration in Austria), Flüchtlingsintegration (integration of refugees), Veganismus (veganism), pflanzliche Ernährung (plant-based diet). Overall, this filtering yielded 1072 results for articles related to the topic of climate, 97 articles for migration, and 591 articles for veganism. To narrow down the final selection, only the 50 most recently published articles for each topic were considered. As the aim of this study was to determine if digital nudges would influence news consumption diversity, the extracted articles were qualitatively analyzed and evaluated regarding their content diversity, which was defined as “the heterogeneity of media content in terms of one or more specified characteristics” [ 25]. The characteristics considered in this study were genre, language tone, political perspective, and viewpoint. The objective was to gather three articles per topic - climate, migration, and veganism - that would be considered homogenous, i.e., non-diverse in terms of the above-mentioned properties, and one article that deviated from this homogeneity. Though a variety of attributes were considered for the final evaluation and article selection, it is important to note that Der Falter follows a rather clear editorial line in its content and coverage, thereby limiting the scope of diversity examined in the study to a certain extent.

4. Results

Overall, 222 data sets were collected. Following the removal of incomplete interviews and interviews completed under the time limit of 3 minutes 40 seconds, 117 data sets remained for further analysis. Within the valid data sets, experimental conditions were distributed among participants as follows: control (n = 38), feedback nudge (n = 48), and social norms nudge (n = 31). The mean age of participants was 35.02 years (SD 14.62). Examining the first hypothesis, the target article was selected 10 out of 114 times (8.77%, conversion rate, M = .088, SD = .284) in the control and 21 out of 144 times (14.58%, 66.25% lift, M = .146, SD = .354) in the feedback nudge condition. The result of the Mann-Whitney U-Test revealed an insignificant diference between the control and feedback nudge groups (U(114,144) = 7731, p = .078, r = -0.058). Thus, participants presented with a feedback nudge did not select the target article significantly more frequently than the group that was not nudged (see Figure 2), leading to a rejection of the first hypothesis.

Looking at the second hypothesis, the target article was selected 10/114 times in the control group (8.77% conversion rate, M = .088, SD = .284), and 22/93 times in the experimental condition (23.66% conversion rate, 169.68% lift, M = .237, SD = .427). Here, the Mann-Whitney U-Test revealed a significant diference between the two groups (U(114,71) = 4512, p = .002, r = - 0.149), indicating that participants selected the target article significantly more frequently when the target article was presented in combination with a social norm nudge, see Figure 2. Thus, the second hypothesis can be preliminarily accepted.

Regarding their primary source for news consumption (primary news medium used), most participants (n = 52, 44.44%) reported consuming news primarily through online media. This was followed by social media (n = 41, 35.04%), TV and radio (n = 9, 7.69% each), print media (n = 4, 3.42%), and podcasts (n = 2, 1.71%).

5. Discussion

Contribution The paper aspires to encourage RSs designers to consider the psychological and socially relevant aspects and implications of their design choices more thoroughly, thereby providing an opportunity to develop more ethical and beneficial recommenders that extend beyond accuracy. One result from the survey on news reading behavior that accompanied the A/B testing herein presented shows that most people consumed news online, either via traditional digital news platforms or via social media. This underlines the significant role that digital media play in today’s news landscape. Moreover, the dominant status of online media is particularly critical when considering the seemingly increasing politicization and polarization across the cultural and news landscapes in recent decades, which appears to be strongly driven and perpetuated by news filtering techniques. As people’s opinion formation and worldview are strongly impacted and guided by the content they consume, media outlets should therefore turn to more value-based approaches in the recommendation algorithms they employ to ensure content pluralism, thereby promoting deliberative models of democracy. While the study was limited to the context of news recommenders, the results shall constitute a broader knowledge gain on the efectiveness of digital nudges for the pursuit of individually and societally relevant goals. Thus, the aim of this paper is not only to suggest that digital nudges, particularly feedback and social norm nudges should be utilized more frequently in NRSs as they can increase news consumption diversity, but also to enlighten readers on a more general scale that RSs and their designs have considerable behavioral impacts on individuals, and, consequently, society at large. Limitations Due to methodological limitations, this study was unable to fully examine personalized digital nudges in recommender systems, but generic nudges appealing to universal psychological patterns were employed instead [ 3 ] [ 1 ]. As the study design did not enable access to user data, nor did study participants have the opportunity to freely browse through and consume news content prior to being nudged via feedback, participants in the feedback nudge group only received feedback on their ”hypothetical” behavior, which may explain the observed limited influence of this type of nudge. A potential resumption of the study could alter the procedure by utilizing an interactive website that monitors users’ behavior prior to presenting them with a personalized feedback nudge based on the gathered user data, which may be more likely to afect article choices based on more comprehensive and accurate feedback. Nevertheless, although the experiment did not provide data on the efect of personalized nudges, diferences in consumption behavior across treatment groups using diferent nudges were still observed and evaluated.

Future research potential Particularly given the fact that digital media outlets heavily utilize recommendation algorithms and these systems traditionally primarily consider optimal personalization as the incentive to improve news recommendation, this survey’s results accentuate the urgency for incorporating humanistic and foundational psychological values into the design of recommender systems. Moreover, since people rated it highly important that news on a given topic is presented from a variety of viewpoints and that a diverse set of news should be consumed by individuals, the study also revealed that diversity is a strongly valued and sought-after aspect in the news domain. Thus, more humanistic recommender designs should continuously be developed in the pursuit of both societally and individually relevant objectives. By further bringing research based on insights from nudging theory to the digital world and implementing nudges in recommender systems, there is great potential for changing people’s online decisions. Nevertheless, there are certain ethical questions and concerns that remain when implementing interventions designed to change people’s behavior. Therefore, Nudges should only be employed to increase behaviors that positively impact both individuals and society.

Acknowledgments

This research is supported by the Christian Doppler Research Association (CDG). [11] T. Mirsch, C. Lehrer, R. Jung, Digital nudging: Altering user behavior in digital environments (2017). [12] C. Schneider, M. Weinmann, J. Vom Brocke, Digital nudging: guiding online user choices through interface design, Communications of the ACM 61 (2018) 67–73. [13] R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions About Health, Wealth, and

Happiness, Penguin, 2009. [14] C. Meske, T. Potthof, The dinu-model–a process model for the design of nudges (2017). [15] R. Hertwig, T. Grüne-Yanof, Nudging and boosting: Steering or empowering good decisions, Perspectives on Psychological Science 12 (2017) 973–986. [16] M. Weinmann, C. Schneider, J. v. Brocke, Digital nudging, Business & Information Systems

Engineering 58 (2016) 433–436. [17] S. Munson, S. Lee, P. Resnick, Encouraging reading of diverse political viewpoints with a browser widget, in: Proceedings of the international AAAI conference on web and social media, volume 7, 2013, pp. 419–428. [18] A. Caraban, E. Karapanos, D. Gonçalves, P. Campos, 23 ways to nudge: A review of technology-mediated nudging in human-computer interaction, in: Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–15. [19] Y. Nahmias, O. Perez, Y. Shlomo, U. Stemmer, Privacy preserving social norm nudges,

Mich. Tech. L. Rev. 26 (2019) 43. [20] D. J. Leiner, Sosci survey (version 3.1. 06), Computer software (2019). [21] R. Kohavi, R. Longbotham, D. Sommerfield, R. M. Henne, Controlled experiments on the web: survey and practical guide, Data mining and knowledge discovery 18 (2009) 140–181. [22] 2020. URL: https://cxl.com/ab-test-calculator/. [23] G. Joris, S. Vercoutere, O. De Clercq, K. Van Damme, P. Mechant, L. De Marez, Nudging towards exposure diversity: Examining the efects of news recommender design on audiences’ news exposure behaviours and perceptions, Digital Journalism (2022) 1–22. [24] P. Cross, Statistical power: What it is and how to calculate it in a/b testing, 2021. URL: https://cxl.com/blog/statistical-power/. [25] A. Masini, P. Van Aelst, T. Zerback, C. Reinemann, P. Mancini, M. Mazzoni, M. Damiani, S. Coen, Measuring and explaining the diversity of voices and viewpoints in the news: A comparative study on the determinants of content diversity of immigration news, Journalism Studies 19 (2018) 2324–2343.

References

[1] J. Bobadilla , F. Ortega , A. Hernando , A. Gutiérrez , Recommender systems survey , Knowledge-based systems 46 ( 2013 ) 109 - 132 .

[2] F. Alyari , N. Jafari Navimipour , Recommender systems: a systematic review of the state of the art literature and suggestions for future research , Kybernetes 47 ( 2018 ) 985 - 1017 .

[3] M. Jesse , D. Jannach , Digital nudging with recommender systems: Survey and future directions , Computers in Human Behavior Reports 3 ( 2021 ) 100052 .

[4] N. Mattis , P. Masur , J. Möller , W. van Atteveldt , Nudging towards news diversity: A theoretical framework for facilitating diverse news consumption through recommender design, new media & society ( 2022 ) 14614448221104413 .

[5] L. Heitz , J. A. Lischka , A. Birrer , B. Paudel , S. Tolmeijer , L. Laugwitz , A. Bernstein , Benefits of diverse news recommendations for democracy: A user study , Digital Journalism 10 ( 2022 ) 1710 - 1730 .

[6] J. Vermeulen , To nudge or not to nudge: news recommendation as a tool to achieve online media pluralism , Digital Journalism 10 ( 2022 ) 1671 - 1690 .

[7] N. Helberger , On the democratic role of news recommenders , Digital Journalism 7 ( 2019 ) 993 - 1012 .

[8] J. Stray , I. Vendrov , J. Nixon , S. Adler , D. Hadfield-Menell, What are you optimizing for? aligning recommender systems with human values , arXiv preprint arXiv:2107.10939 ( 2021 ).

[9] T. N. T. Tran , A. Felfernig , N. Tintarev , Humanized recommender systems: State-of-the-art and research issues , ACM Transactions on Interactive Intelligent Systems (TiiS) 11 ( 2021 ) 1 - 41 .

[10] R. Karlsen , A. Andersen , Recommendations with a nudge , Technologies 7 ( 2019 ) 45 .