Negativity Sells? Using an LLM to Affectively Reframe News Articles in a Recommender System Jia Hua Jeng1 , Gloria Kasangu2 , Alain Starke3 , Erik Knudsen4 and Christoph Trattner5 1 MediaFutures, University of Bergen, Lars Hilles Gate 30, Bergen, Vestland, 5008, Norway 2 MediaFutures, University of Bergen, Lars Hilles Gate 30, Bergen, Vestland, 5008, Norway 3 Amsterdam School of Communication Research, University of Amsterdam, P.O. Box 15791, Amsterdam, 1001 NG, Netherlands 4 MediaFutures, University of Bergen, Lars Hilles Gate 30, Bergen, Vestland, 5008, Norway 5 MediaFutures, University of Bergen, Lars Hilles Gate 30, Bergen, Vestland, 5008, Norway Abstract Recent developments in artificial intelligence allow newsrooms to automate journalistic choices and processes. In doing so, news framing can impact people’s engagement with news media, as well as their willingness to pay for news articles. Large Language Models (LLMs) can be used as a framing tool, aligning headlines with a news website user’s preferences or state. It is, however, unknown how users perceive and experience the use of a platform with such LLM-reframed news headlines. We present the results of a user study (𝑁 = 300) with a news recommender system (NRS). Users had to read three news articles from The Washington Post from a preferred category (abortion, economics, gun control). Headlines were rewritten by an LLM (ChatGPT-4) and images were replaced in specific affective styles, across 2 (positive or negative headlines) x 3 (positive or negative image, or no image) between-subject framing conditions. We found that negatively framed images and text elicited negative emotions, while positive framing had little effect. Users were also more willing to pay for a news service when facing negatively framed headlines and images. Surprisingly, the congruency between text and image (i.e., both being framed negatively or positively) did not significantly impact engagement. We discuss how this study can shape further research on affective framing in news recommender systems and how such applications could impact journalism practices. Keywords News Recommender System, Affective framing, Large Language Models, User Engagement, User Study 1. Introduction Recommender systems are an essential tool to deliver news content to people these days. They are typically an integral part of any news website. These systems personalize the news experience, tailoring content to align with users’ preferences and behaviors [1]. At the same time, large language models (LLMs) more and more find their way into the news production cycle, from the back end to the front end [2, 3, 4, 5]. For example, in Norway and the UK, LLMs are used to summarize news articles to make them easier to digest and engaging [6, 7]. These AI-driven approaches can help newsrooms to become more efficient by providing new ways to present information. Although LLMs developments demonstrate considerable potential, they have yet to be widely im- plemented across newsrooms, particularly local newspapers, which have struggled to engage readers as effectively as social media platforms in recent years [8]. However, the introduction of LLMs in the newsroom has shown the potential to assist journalists in enhancing reader engagement [9, 10]. In this paper, we specifically examine the potential to emotionally reframe news with LLMs to increase user engagement. The framing of news articles has been the subject to the expertise of journalists. Framing refers to the representation of reality by a group of individuals or organizations, in line with specific beliefs or Proceedings of the International Workshop on News Recommendation and Analytics co-located with the 2024 ACM Conference on Recommender Systems (RecSys 2024), October 14–18, 2024, Bari, Italy Envelope-Open jia-hua.jeng@uib.no (J. H. Jeng); gloria.kasangu@student.uib.no (G. Kasangu); a.d.starke@uva.nl (A. Starke); erik.knudsen@uib.no (E. Knudsen); christoph.trattner@uib.no (C. Trattner) Orcid 0009-0008-5225-5757 (J. H. Jeng); 0009-0000-8585-3966 (G. Kasangu); 0000-0002-9873-8016 (A. Starke); 0000-0002-7046-9424 (E. Knudsen); 0000-0002-1193-0508 (C. Trattner) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings persuasive intents [11]. This practice is relatively common in political and economics news, where specific angles are attributed to a story [10], and play a crucial role in shaping public understanding and perception [12, 13, 11, 14]. Recent years have seen an increase in research on the practice of affective framing, which refers to the strategic use of emotional tone in communication—particularly in news articles—and its impact on reader engagement [15, 16, 17, 18]. This link between affective reframing in news and the engagement levels of readers in turn taps into their engagement levels and whether they would like to pay for subscriptions to news platforms [19]. A primary limitation to these findings is that it is unclear whether they also apply to a news recommender scenario. Arguably, if a news article already matches one’s preferences, the specific frame might not be the main predictor of news consumption. This study examines the effects of AI-driven affective framing through text and images in the context of a news recommender system (NRS). We introduce a novel methodology that diverges from the conventional, hands-on framing techniques. By utilizing OpenAI’s ChatGPT-4, a Large Language Model (LLM), we investigate its effectiveness in altering the construction of news narratives in affective news framing. Existing literature suggests that these models can craft detailed narrative frameworks that closely resemble human-authored content in news articles, to the extent that it becomes difficult for readers to discern the origin of the articles [20, 21]. Our research questions are the following: • RQ1: To what extent do different affective news frames in text and images affect emotional states among readers? • RQ2: How does image-based and text-based affective re-framing affect the intention to pay for a news service, and should these frames be emotionally aligned (i.e., congruent)? • RQ3: To what extent are the effects of affective reframing on intention to pay mediated by user engagement? We focus on two primary affective frames of news headlines: (a) ‘Positive’ and (b) ‘Negative’. We examine their impact on readers’ engagement and intention to pay for news when combined with imagery. If the emotional valence of images is not congruent with what is presented in the text, we expect users to be less inclined to engage with a news recommender service. In doing so, we address critical gaps in existing studies. First, we currently do not understand how readers’ emotional states are influenced by affectively reframed content. Not in the context of NRSs, nor in the context of Large Language Models. Secondly, while studies have focused on the effects on users’ emotions, there is little known about how it would affect users’ levels of engagement and their intention to pay for such a news service. We observed that negative images and text triggered more negative emotions than positive framing. Interestingly, when exposed to negatively framed headlines and images, participants tend to subscribe to a news service. Surprisingly, alignment between the affective framing of the text and image did not significantly influence engagement. This article is organized as follows: Section 2 presents the related work of News recommender systems, users’ preference and affective Framing. Section 3 outlines our research methods, including data collection and the utilization of ChatGPT-4 in the news framing experiment. The research results depict in Section 4, each subsection aligns different research questions regarding the shift of emotional states, the interplay of image and text-based effect, and readers’ engagement and intention to pay. In Section 5, we will explore how AI could influence journalism and its implications. The article concludes in Section 6 with a summary of our findings and suggestions for further research. 2. Background The following sections provide an overview of related and relevant work in the field. First, we review the literature on news recommender systems (NRSs), emphasizing their role in mitigating information overload and aligning users’ preferences by tailoring content. Thereafter, we discuss work on affective framing in text and images; these factors affect news frames in shaping readers’ perceptions and behaviors. Finally, we work on how affective framing influences user engagement in news consumption. These subsections provide an understanding of the intersections between recommender systems, affective framing, and user engagement in digital journalism. 2.1. News Recommenders and User Preferences There is far more news content available online than one can reasonably consume or browse through. This leads to information overload [22, 23], which hinders users’ capacity to identify content relevant to their interests and benefits [1]. Given this scenario, NRSs are crucial as they filter content and present personalized recommendations, effectively enabling users to navigate the massive online news and information. By tailoring content that aligns with user profiles and preferences, these systems accelerate the information-retrieving process, raising the efficiency and relevance of online news consumption [24, 25, 26]. Existing studies indicate that personalized information systems enhance users’ perceived relevance, involvement, and engagement with content [27, 28]. In addition, NRSs not only facilitate access to content relevant to individual users, but also yield commercial advantages for platform providers [29]. NRSs are geared towards showing content that align with its users’ preferences. These preferences span a wide array of interconnected factors, including but not limited to specific subjects of interest, readers’ emotional states, the credibility of news, users’ attitude [30, 31, 24]. Recommender systems usually rely on ratings to indicate their preferences for items and also gather clickstream data to infer the interests or preferences of users [32, 33, 34]. These systems improve user engagement and satisfaction by finely tuning recommendations to align with individual user preferences [35, 36]. However, personalized recommender systems may cause the adverse polarization effect from offering customized content to users via users’ recorded behaviors, preferences, and tendencies [37, 38]. Considering users’ preferences are crucial for the efficacy of NRSs. It’s essential to balance personalization with diversity, ensuring users access to a wide array of information tailored to their interests and requirements [39, 40]. 2.2. Affective Framing in Text and Images 2.2.1. Framing. News media, central to democratic societies, significantly influences public opinion by framing issues and events to define and highlight particular aspects [11]. To frame means opting for aspects of a perceived reality and making them more prominent within a communicating text. In this way, it promotes a specific definition of the problem, causal interpretation, moral evaluation, and/or treatment recommendation for the issues being discussed [41]. Frames draw attention to specific information related to the subject being communicated. By making certain details more salient, they enhance the likelihood that the audience will notice and understand this information, subsequently processing and storing it in their memory [42, 41]. Frames are components of political debates, journalistic norms, and discourse in social movements. This is the alternative approach to interpreting and defining issues in the political and social world [11]. Journalists use news frames to provide interpretation of events and contentious topics. By reducing complex discussions and policy matters to the elements, these frames identify the responsibility for issues and offer guidance on possible directions for action [43]. The application of news framing varies depending on the nature of the media outlet and the topic. This variation is less pronounced between different media types, such as television versus print, but more significant between categories of news outlets, namely sensationalist versus serious [44]. For example, one news study on economic issues indicated that affective attributes of the news articles, particularly positive versus negative frames, have a certain influence on people’s evaluation of economics. Specifically, a negative frame can significantly affect readers’ expectations and performance in relation to the economy [45]. 2.2.2. Affective Framing in text and image. Numerous studies have demonstrated that affective factors in news production, play a significant role in shaping public perceptions. A key element in how the media influences readers’ interpretations of specific events is the use of affective frames, such as the representation of political candidates. The concept of affective framing relates to the emotional tone conveyed in news articles [15]. Journalism faces uncertainty amid economic, political, and social crises, creating a volatile environment where employing affective framing becomes a strategic tool to steer through and mitigate these challenges [46]. Affective framing is a spontaneous, non-inferential, and pre-reflective method of sorting and choosing information. This process simplifies complex information to first-personally manageable, giving it a specific cognitive significance. This form of framing emphasizes specific emotions further in the article content [47, 48]. Based on these, affective framing also affects the reader’s emotional states or attitudes. For instance, one affective framing study found that viewing the negatively framed tweets amplifies the unfavorable emotional state of readers and leads to an increased willingness for environmental protection [49]. Moreover, another study also indicates that the participants exposed to positively framed messages revealed a more optimistic attitude toward the water recycling issue than those subjected to negatively framed information [50]. In addition, regarding preference, one study on news frames suggests that individuals interested in political issues are more likely to select negative news stories. The results indicate a common preference among participants for negative news articles [51]. Therefore, affective framing shapes public perception and opinion. It influences the reader’s emotional states and thus impacts the decision-making process [52]. Another crucial element in news framing is imagery; which shapes readers’ interpretation of the text by triggering certain cognitive frameworks through associative reasoning [53, 54]. Images serve as an influential medium, offering a less cognitively demanding and intrusive experience compared to text. Their visual impact, which closely mirrors reality, has the capacity to evoke strong emotions. For instance, due to their compelling appeal, images often appear on pages and websites, setting the initial context for a story [55]. It is essential to combine images and text in news-press. According to [56], traditionally, images were used to illustrate the text, providing a visual representation that the text elaborated on. This study highlights a historical reversal: the text now relies on the image, adding layers of meaning, culture, and imagination to the primary visual message rather than the image just illustrating the words. This nuanced interaction between text and image underlines the complexity of how media construct and interpret meaning in news media. For example, research indicates that emotional responses to pleasing visual slides displayed high levels of positivity and minimal negativity [57]. One study on environmental protection suggests that environmental organizations can show visual elements that depict the negative impacts of human activity on the environment. This strategy is recommended to more effectively capture public attention [58]. Another study highlights that the presence of visual elements can influence readers’ willingness to share news, compared to articles that only contain text [59]. 2.3. Affective Framing and Engagement Affective framing, via both textual and visual elements, is essential in transforming users’ emotions, perceptions, and decision-making processes [60, 61, 62]; it highlights the importance of how news framing influences user engagement, potentially altering reader engagement through the manipulation of news articles, such as in political contexts [63]. For instance, one study investigates how positive and negative news framing affect people’s engagement in mobilizing to vote in referendum campaigns, especially for the group that is against the proposals. The result indicates that positive news mobilize skeptics to participate in a referendum vote [64]. Another study examining the effect between sharing news based on positive and negative news framing indicated that negative fake news increases readers’ willingness to share, while positive fake news does not reach a statistically significant level. Furthermore, negative emotions mediate in the viralization of news content, whereas positive emotions do not have the same effect [65]. Another study focusing on social media explores the readers’ engagement with news articles characterized by positive and negative emotions such as anger, fear, hope, and happiness. The finding shows a positive correlation between negative news and readers’ likes, shares, and comments on the articles [66]. Therefore, affective framing alters readers’ emotional state and impacts the decision- making process, potentially leading to varied levels of engagement. Consequently, affective framing significantly influences news engagement, shaping readers’ percep- tions and potentially impacting society. With ChatGPT’s swift rise to global, this technology offers a nuanced understanding of human language [67]. Understanding the complex relationship between affective framing and this advanced AI technology is crucial. 2.4. Contribution of Current Research We have discussed two key areas of research: NRSs and the role of affective framing in influencing user engagement. Much of the existing recommender research has focused on mitigating information overload in news consumption by personalizing content based on user preferences [1, 24, 25, 26]. Additionally, research on affective framing has shown its significant impact on readers’ emotions and decision-making processes [49, 15, 52]. However, there is limited research exploring the interaction between NRSs and affective framing, particularly in the context of news reframing by LLMs. Notably, previous studies have typically focused on either text or image framing independently [49, 50, 51, 53, 59]. In contrast, our research considers both elements together, investigating how AI-generated affective framing in both text and images influences user engagement and their willingness to pay for news services. Our contributions to the field are as follows: 1. Integration of Affective Framing and Recommender Systems: We explore how AI-driven affective reframing, using ChatGPT-4, impacts user emotions and perceptions in news recommenda- tions—an area that has not been extensively explored in existing research. 2. Congruency of Text and Image: We investigate the effects of congruency between affective text and images on user engagement and behavior, addressing a key gap in the research on how different media cues interact in news consumption. 3. User Engagement and Intention to Pay: Our findings provide empirical evidence on the influence of affective framing on user engagement and their willingness to subscribe to news services, which contributes not only to academia but also to the journalism industry. 3. Methods 3.1. System & Dataset We developed a research platform to address our research questions. It utilized news articles from Washingtonpost.com/opinions/, a popular commercial news website which features opinion articles. We chose this dataset primarily because it is a well documented and widely used dataset in news RecSys (see, for example, [68] and [69]). This makes the results more reproducible, as other users can also access the news articles and results. We selected three different news topics and sampled 18 articles from each: (1) Abortion: reproductive health and rights, (2) Economy: politics, and (3) Gun control: firearms. The news articles have been specifically selected to ensure that we would be able to perform a valid experiment, to control for different user attitudes regarding different controversial topics [70]. 3.1.1. Model selection For our study, we chose to summarize each of the 18 articles with OpenAI’s GPT-4 model (version gpt-4-0613) to generate two summaries, one with a positive valence and one with a negative valence. We utilized this model due to it’s advanced natural language processing capabilities, suitability for generating contextually relevant text, and tokens limitation. The temperature parameter was set to Figure 1: The procedure in this news research platform, from data collection via Prolific to users’ pre-post emotional states, news framing, engagement and intention to pay. 0.8 to ensure that the generated text was engaging, while maintaining a balance between creative and coherent headlines that comply with the input prompt. The Top_P parameter used in this experiment is the default parameter of 1.0. This parameter enhances the quality of the output by focusing on a broad range of probable next words while allowing variability and diversity in the output. All articles were written in English1 . Our study analyzed the shift in users’ emotional states, engagement and intention to pay in the news across six conditions. The affective framing for each condition in the study is illustrated in Figure 5, with each representing a distinct combination of text and image. Condition A, B, and C were positive framing, with no, incongruent, and congruent images. By contrast, conditions D, E, and F were negative news framing, with (i) no images, (ii) incongruent and (iii) congruent images. We used JavaScript, HTML, CSS for the frontend, and Django with Python for the backend, integrating Washington Post content. We also conducted a T-test and Structural Equation Modeling (SEM). 3.2. Procedure The procedure of the study is shown in Figure 1. Participants were invited to join a survey in which they could test out an online news service. After disclosing demographics, topic preference, and current emotional state, participants were presented three article previews in our news service interface based on their topic preferences. The topics included Abortion, Economics, and Gun Control. Please refer to the footnote to inspect the prompts of our study2 . For each preview, users were asked to indicate to which extent they would want to read more of the article preview, and would like to share with others. In addition, users had to assess how much they liked the article, the article’s content, and whether the article aligned with users’ general preferences and mood. Following the three previews, participants were then asked to self-report their emotional state when reading the previews, in addition to evaluating to which extent, based on the recommended previews, they would want to pay for access to a similar online news services (see Figure 1). 1 This research adhered to the ethical guidelines of the Research Council of Norway and the guidelines of University of Bergen for scientific research. The study was judged to pass without further extensive review, for it contained no misleading information, stress tasks, nor would it elicit extreme emotions 2 The prompts we used in ChatGPT-4 to reframe news articles are here: https://anonymous.4open.science/r/ RecSys2024AffectiveReframing-2505/README.md Figure 2: Emotional state pre-post differences (means and standard errors) for positive (blue) and negative (red) emotional states across the six conditions in this research. (A) - (C) highlight positive text examples and (D) - (F) negative ones, each with either no image, a negative image, or a positive image. 3.3. Measures We inquired on one three main dimension in our experiment: (i) users’ Emotional States, (ii) their experienced Engagement with our news platform, and (iii) their Intention to pay of news articles, all on 5-point Likert scales. 3.3.1. Emotional State Scale Users’ emotional states were measured both pre-test and post-test using the International Positive and Negative Affect Schedule Short-form (I-PANAS-SF) [71]. This included ten items split into two subscales: Positive Affect (‘alert’, ‘inspired’, ‘determined’, ‘attentive’, ‘active’) and Negative Affect (‘upset’, ‘hostile’, ‘ashamed’, ‘nervous’, ‘afraid’). 3.3.2. Engagement and Intention to pay Table 1 describes the items used for engagement and the results of our confirmatory factor analysis. All items were measured on 5-point Likert scales (1 = Strongly Disagree, 3 = Neutral, Disagree, 5 = Strongly Agree). Engagement included items related to willingness to read, liking, preference and mood alignment and sharing intent (adapted from studies cited in [1]). While we initially aimed to differentiate between these items and perceived trust (taken from [72]), the collapsed into a single factor. Additionally, we also inquired on a user’s intention to pay for a news service similar to the one used in the study, based on an item from [73]: “Based on the recommended articles, I would want to pay for access to online news services in the future similar to this one”. 4. Results A total of 300 participants from the USA (𝑀𝑎𝑔𝑒 = 40.31, 𝑆𝐷 = 12.04, 50.6% males) completed our user study. All 300 participants were recruited from the crowdsourcing platform Prolific. Our sample size is sufficiently large to be able to detect effects that were theoretically and practically meaningful [40]. We analyzed pre-post differences for positive and negative emotional states, across six affective reframing conditions. We also explored how different interaction of text, image and news topics affected readers’ engagement and intention to pay for a news subscription in a Structural Equation Model (SEM). 4.1. RQ1: Changes in Emotional States Overall, as shown in Figure 2, our results indicate that Negative news framed by GPT-4 triggers stronger emotional responses at significant statistical levels, especially when negative text is coupled with congruent imagery, compared with Positive news. Similarly, negative images alter more users’ emotional states [49]. Regarding positive conditions, emotional changes and responses were less pronounced. The emotions that reached a significant level of mean difference included active and nervous (condition A; no image), the inspired, upset and hostile (condition B; incongruent), and shame (condition C; congruent). This suggested that positive text was less likely to elicit clear changes in a user’s emotional state, regardless of the accompanying image shown. In contrast, negative conditions exhibited more pronounced emotional changes and responses. Condition D revealed inspired, upset, hostile, and afraid. Condition E displayed emotions such as inspired, upset, hostile, ashamed, and afraid. Finally, Condition F encompassed a range of emotions, including inspired, determined, upset, hostile, ashamed, and afraid. 4.2. RQ2-3: Engagement and Intention to Pay We submitted all measures and questionnaire items to a structural equation model (SEM) analysis. We first tested a model a fully saturated model, where all condition effects affected engagement and intention to pay and performed stepwise removal of non-significant relations afterwards. The resulting model is depicted in Figure 3, and had excellent fit statistics: 𝜒 2 (56) = 91.976, 𝑝 < 0.01, 𝐶𝐹 𝐼 = 0.996, 𝑇 𝐿𝐼 = 0.995, 𝑅𝑀𝑆𝐸𝐴 = 0.027, 90% − 𝐶𝐼: [0.016, 0.036]. Our path model met the guidelines for discriminant validity, as well as construct validity (cf. Table 1). Table 1 Questionnaire items used in the confirmatory factor analysis, as part of the SEM. The construct met the guidelines for construct validity, with average variance explained (AVE) exceeding 0.5 [74]. Aspect Item Loading I want to read more of this article. 0.86 Perceived I like this article. 0.93 Engagement The content of the article aligns with my preferences. 0.85 AVE = .75 The content aligns well with my mood. 0.84 𝛼 = .92 I would like to share this article. 0.85 I trust the article’s content. 0.81 We examined whether different affective frames and the alignment of affective frames affected engagement and intention to pay. As depicted in Figure 3, we found two effects of our affective frames. First, we found that users facing headlines that were comprised negative language were more likely to pay for a news service based on our system, as positive text negatively affected intention to pay (𝛽 = −.434). Although no effects were found on user engagement, it did suggest that users would be more likely to use a news service when observing negatively valenced news. Second, we observed an interaction effect between text and image emotions on intention to pay, which was complementary to the main effect of text. To better understand this interaction effect, please refer to Figure 4. Depicted is a user’s intention to pay accross the six reframing conditions. The graph clearly shows the main effect of average negative text leading to a higher intention to pay. For the interaction effect, there was a significantly higher intention to pay for negative headlines accompanied by a negative image (𝑀 ≈ 2.22), compared to positive headlines accompanied by a negative image (𝑀 ≈ 1.62). This suggested that negative emotions in a news article may have a positive effect on a user’s willingness to use that news service. The aforementioned interaction effect and Figure 4 also shows the role of alignment between text and image emotions. Whereas negative emotions seemed to support each other in eliciting user responses, the combination of a positive headline with a positive image actually led to a weaker user response, compared to, for example, a negative headline with a positive image. This suggested that alignment or congruency between image and text emotions was mostly beneficial for negative emotions. Regarding further mediation effects, Figure 3 shows no mediated effects from our reframing manipu- lations. Although we did observe a positive relation between user engagement and intention to pay (𝛽 = .633), it only acted as a mediator for a news category. As mentioned, all effects directly affected a user’s intention to pay. We did observe that users found news articles in the abortion topic more engaging, when compared to the economics category (𝛽 = .475). However, we did not observe any significant interaction effects between the news topic and the emotional manipulation, suggesting users found the news topic more engaging, but not in relation to our affective manipulations. Note that we observed no such effect between gun control and economics. 5. Discussion We have investigated the impact of affective reframing on news consumption by performing Large Language Models (LLMs). Our primary focus was to provide nuanced insights into news recommender systems (NRSs) regarding the emotional states, engagement, and subscription intentions of readers. This is achieved by exploring the intersectional effects of text framed by GPT-4 and imagery, and news topics aligning readers’ preferences. We have gathered valuable insights relevant to the application of Artificial Intelligence in journalism. The following insights are derived from our analysis of emotional states (RQ1), image and text-based affective reframing, and engagement and intention to pay for news articles (RQ2 and RQ3). Regarding [RQ1], we find that negative frames substantially trigger and alter readers’ emotions, compared to positive frames. This observation is consistent with prior studies outside of LLMs, which demonstrate that affective framing influences readers’ emotional states [49]. Figure 3: Structural Equation Model (SEM). The 𝛽-coefficients are indicated by the numbers on the arrows, while standard errors are shown within brackets. Objective system aspects are presented in purple, behavioral indicators in blue, experience aspects in orange. Not depicted are factors with only non-significant paths, used to form the interaction effects (e.g., negative image). [74]. ∗∗∗ 𝑝 < 0.001, ∗∗ 𝑝 < 0.01, ∗ 𝑝 < 0.05. Regarding RQ2 and RQ3, we found that congruence between negative news frames and images increased readers’ intentions to subscribe but did not significantly affect engagement. This aligns with research indicating that negative news affects user engagement more than positive news [65], and visual elements enhance engagement more than text-only content [59]. News topics matching user preferences also boost engagement, supporting findings that recommender systems can increase engagement by aligning content with individual preferences [35, 36]. Our path analysis shows a significant relationship where engagement leads to a greater willingness to pay. While this confirms that engagement predicts subscriptions [19], engagement does not mediate our reframing manipulations. These results contribute the essential consequences for both Newsroom and the development of NRSs. Our research initially depicts that affective frames by LLM model in news articles can significantly trigger emotional reactions. These changes in emotion may influence the audience’s decision making process [75, 76, 77, 78]. Our findings highlight that negative news frames, especially when accompanied by congruent imagery, enhanced user willingness to pay. These supports existing research suggesting that coordinating text and images in news articles influence users’ behavior and perception [79, 80], and readers’ preference, engagement and payment intentions are correlated to each other [81, 59, 19]. 6. Conclusions & Future Work The effects of LLM-driven affective reframing in news are nuanced. It seems that negative frames significantly amplify emotional reactions and increase the intention to pay for news content, especially when paired with congruent imagery. In contrast, incongruency regarding emotions in a news articles may inhibit changes in emotional states, as well as in perceived and experienced user engagement responses. Our study is subject to limitations. The lack of an actual news platform would have further increase the validity of our findings, as well as if our NRS would have been more extensive. Additionally, our focus on a knowledge-based personalized recommender system may overlook biases from content diversity Figure 4: Mean intention to pay across different text and image conditions. Error bars denote 1 S.E. Figure 5: (A) Presents the original text and image, while (B) depicts a sample of an affective framing style by GPT-4 with incongruent image and text. (C) Illustrates the prompt construction used to generate the framing style seen in (B). that could polarize perceptions. Our findings are also limited by only considering The Washington Post in our dataset, which may not represent broader media biases. Future research will examine the impact of affective framing on news consumption and assess subscription intentions from both personalized and varied-content approaches. We will also study complex multi-level interactions and include a wider range of news sources to expand the applicability of our results. 7. Acknowledgments This work was supported by the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the Centre for Research-based Innovation scheme, project number 309339. References [1] M. Karimi, D. Jannach, M. Jugovac, News recommender systems–survey and roads ahead, Infor- mation Processing & Management 54 (2018) 1203–1227. [2] A. L. Opdahl, B. Tessem, D.-T. Dang-Nguyen, E. Motta, V. Setty, E. Throndsen, A. Tverberg, C. Trattner, Trustworthy journalism through ai, Data & Knowledge Engineering 146 (2023) 102182. [3] J. V. Pavlik, Collaborating with chatgpt: Considering the implications of generative artificial intelligence for journalism and media education, Journalism & Mass Communication Educator 78 (2023) 84–93. [4] M. Hansen, M. Roca-Sales, J. M. Keegan, G. King, Artificial intelligence: Practice and implications for journalism (2017). [5] G. Romeo, E. Griglié, Ai ethics and policies: Why european journalism needs more of both, in: The 2021 Yearbook of the Digital Ethics Lab, Springer, 2022, pp. 229–245. [6] BBC LLM Use Cases, Llm use cases at bbc, 2024. URL: https://tech.ebu.ch/publications/ llm-use-cases-at-bbc, accessed: 2024-08-12. [7] Schibsted AI Strategy, Schibsted’s ai strategy with sven størmer thaulow, 2024. URL: https:// aiinside.show/episode/schibsteds-ai-strategy-with-sven-stormer-thaulow, accessed: 2024-08-12. [8] Reuters Institute for the Study of Journalism, Digital news report 2023: Executive summary, 2023. URL: https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2023/dnr-executive-summary, accessed: 2024-08-21. [9] N. Diakopoulos, Automating the news: How algorithms are rewriting the media, Harvard Univer- sity Press, 2019. [10] S. Petridis, N. Diakopoulos, K. Crowston, M. Hansen, K. Henderson, S. Jastrzebski, J. V. Nickerson, L. B. Chilton, Anglekindling: Supporting journalistic angle ideation with large language models, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1–16. [11] C. H. De Vreese, News framing: Theory and typology, Information design journal+ document design 13 (2005) 51–62. [12] T. Bolsen, M. A. Shapiro, The us news media, polarization on climate change, and pathways to effective communication, Environmental Communication 12 (2018) 149–163. [13] C. de Vreese, J. Peter, H. Semetko, et al., Framing politics at the launch of the euro: A cross-national comparative study of frames in the news., Political Communication 18 (2001). [14] P. López-Rabadán, Framing studies evolution in the social media era. digital advancement and reorientation of the research agenda, Social Sciences 11 (2021) 9. [15] R. Coleman, S. Banning, Network tv news’ affective framing of the presidential candidates: Evidence for a second-level agenda-setting effect through visual framing, Journalism & Mass Communication Quarterly 83 (2006) 313–328. [16] K. E. McIntyre, R. Gibson, Positive news makes readers feel good: A “silver-lining” approach to negative news can attract audiences, Southern Communication Journal 81 (2016) 304–315. [17] J. Berger, K. L. Milkman, What makes online content viral?, Journal of marketing research 49 (2012) 192–205. [18] H. S. Kim, Attracting views and going viral: How message features and news-sharing channels affect health news diffusion, Journal of communication 65 (2015) 512–534. [19] H. Davoudi, M. Zihayat, A. An, Time-aware subscription prediction model for user acquisition in digital news media, in: Proceedings of the 2017 SIAM International Conference on Data Mining, SIAM, 2017, pp. 135–143. [20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [21] C. Kolo, J. Mütterlein, S. A. Schmid, Believing journalists, ai, or fake news: The role of trust in media., in: HICSS, 2022, pp. 1–10. [22] B. Scheibehenne, R. Greifeneder, P. M. Todd, Can there ever be too many options? a meta-analytic review of choice overload, Journal of consumer research 37 (2010) 409–425. [23] A. D. Starke, J. Sedkowska, M. Chouhan, B. Ferwerda, Examining choice overload across single-list and multi-list user interfaces, in: 9th joint workshop on interfaces and human decision making for recommender systems (IntRS’22), 2022. [24] S. Raza, C. Ding, News recommender system: a review of recent progress, challenges, and opportunities, Artificial Intelligence Review (2022) 1–52. [25] P. Dandekar, A. Goel, D. T. Lee, Biased assimilation, homophily, and the dynamics of polarization, Proceedings of the National Academy of Sciences 110 (2013) 5791–5796. [26] D. Jannach, M. Zanker, A. Felfernig, G. Friedrich, Recommender systems: an introduction, Cam- bridge University Press, 2010. [27] M. A. Beam, Automating the news: How personalized news recommender system design choices impact news reception, Communication Research 41 (2014) 1019–1041. [28] E. Pariser, The filter bubble: What the Internet is hiding from you, penguin UK, 2011. [29] X. Wang, C. Liu, et al., Design of personalized news recommendation system based on an improved user collaborative filtering algorithm, Mobile Information Systems 2023 (2023). [30] P. Pu, L. Chen, R. Hu, A user-centric evaluation framework for recommender systems, in: Proceedings of the fifth ACM conference on Recommender systems, 2011, pp. 157–164. [31] J. Liu, P. Dolan, E. R. Pedersen, Personalized news recommendation based on click behavior, in: Proceedings of the 15th international conference on Intelligent user interfaces, 2010, pp. 31–40. [32] G. Jawaheer, P. Weller, P. Kostkova, Modeling user preferences in recommender systems: A classification framework for explicit and implicit user feedback, ACM Transactions on Interactive Intelligent Systems (TiiS) 4 (2014) 1–26. [33] C. Trattner, D. Elsweiler, An evaluation of recommendation algorithms for online recipe portals, CEUR Workshop Proceedings, 2019. [34] M. Elahi, A. Starke, N. El Ioini, A. A. Lambrix, C. Trattner, Developing and evaluating a university recommender system, Frontiers in Artificial Intelligence 4 (2022) 796268. [35] D. Kalimeris, S. Bhagat, S. Kalyanaraman, U. Weinsberg, Preference amplification in recommender systems, in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 805–815. [36] S. M. McNee, S. K. Lam, J. A. Konstan, J. Riedl, Interfaces for eliciting new user preferences in recommender systems, in: User Modeling 2003: 9th International Conference, UM 2003 Johnstown, PA, USA, June 22–26, 2003 Proceedings 9, Springer, 2003, pp. 178–187. [37] C. C. Aggarwal, et al., Recommender systems, volume 1, Springer, 2016. [38] J. Stray, Designing recommender systems to depolarize, arXiv preprint arXiv:2107.04953 (2021). [39] M. Elahi, D. Jannach, L. Skjærven, E. Knudsen, H. Sjøvaag, K. Tolonen, Ø. Holmstad, I. Pipkin, E. Throndsen, A. Stenbom, et al., Towards responsible media recommendation, AI and Ethics (2022) 1–12. [40] A. Starke, E. Asotic, C. Trattner, “serving each user”: Supporting different eating goals through a multi-list recommender interface, in: Proceedings of the 15th ACM Conference on Recommender Systems, 2021, pp. 124–132. [41] R. M. Entman, Framing: Toward clarification of a fractured paradigm, Journal of communication 43 (1993) 51–58. [42] S. T. Fiske, S. E. Taylor, Social cognition, Mcgraw-Hill Book Company, 1991. [43] D. Tewksbury, D. A. Scheufele, News framing theory and research, in: Media effects, Routledge, 2009, pp. 33–49. [44] H. A. Semetko, P. M. Valkenburg, Framing european politics: A content analysis of press and television news, Journal of communication 50 (2000) 93–109. [45] J. B. Hester, R. Gibson, The economy and second-level agenda setting: A time-series analysis of economic news and public opinion about the economy, Journalism & Mass Communication Quarterly 80 (2003) 73–90. [46] E. Siapera, L. Papadopoulou, Hate as a ‘hook’: The political and affective economy of ‘hate journalism’, Journalism 22 (2021) 1256–1272. [47] M. Maiese, R. Hanna, The mind-body politic, Springer, 2019. [48] J. Protevi, Covid-19 in the united states as affective frame, Frontiers in Psychology 13 (2022) 897215. [49] M. Sanford, M. Witkowska, R. Gifford, M. Formanowicz, Emotional framing in online environ- mental activism: Pairing a twitter study with an offline experiment, Frontiers in Psychology 13 (2023) 1099331. [50] T. Greenaway, K. S. Fielding, Positive affective framing of information reduces risk perceptions and increases acceptance of recycled water, Environmental Communication 14 (2020) 391–402. [51] M. Trussler, S. Soroka, Consumer demand for cynical and negative news frames, The International Journal of Press/Politics 19 (2014) 360–379. [52] K. Gross, L. D’ambrosio, Framing emotional response, Political psychology 25 (2004) 1–29. [53] A. D. Starke, K. Emami, A. Makarová, B. Ferwerda, Using visual and linguistic framing to support sustainable decisions in an online store (2023). [54] C. Brantner, S. Geise, K. Lobinger, Fractured paradigm? theories, concepts and methodology of visual framing research: a systematic review, in: Annual Conference of the International Communication Association (ICA)–visual communication studies division, 2012, pp. 1–40. [55] L. Rodriguez, D. V. Dimitrova, The levels of visual framing, Journal of visual literacy 30 (2011) 48–65. [56] R. Barthes, Image-music-text, volume 6135, Macmillan, 1977. [57] C. J. Patrick, S. A. Lavoro, Ratings of emotional response to pictorial stimuli: Positive and negative affect dimensions, Motivation and Emotion 21 (1997) 297–321. [58] G. Salazar, M. C. Monroe, M. Ennes, J. A. Jones, D. Veríssimo, Testing the influence of visual framing on engagement and pro-environmental action, Conservation Science and Practice 4 (2022) e12812. [59] R. R. Mourão, D. K. Brown, Black lives matter coverage: How protest news frames and attitudinal change affect social media engagement, Digital Journalism 10 (2022) 626–646. [60] P. D’Angelo, Doing news framing analysis II: Empirical and theoretical perspectives, Routledge, 2018. [61] J. Bingaman, P. R. Brewer, A. Paintsil, D. C. Wilson, “siri, show me scary images of ai”: Effects of text-based frames and visuals on support for artificial intelligence, Science Communication 43 (2021) 388–401. [62] M. E. Grabe, E. P. Bucy, Image bite politics: News and the visual framing of elections, Political Psychology, 2009. [63] M. C. Nisbet, Framing science: A new paradigm in public engagement, in: Communicating science, Routledge, 2009, pp. 54–81. [64] A. R. Schuck, C. H. de Vreese, Reversed mobilization in referendum campaigns: How positive news framing can mobilize the skeptics, The International Journal of Press/Politics 14 (2009) 40–66. [65] N. Corbu, A. Bârgăoanu, F. Durach, G. Udrea, Fake news going viral: The mediating effect of negative emotions, Media Literacy and Academic Research 4 (2021) 58–87. [66] S. Salgado, G. Bobba, News on events and social media: A comparative analysis of facebook users’ reactions, Journalism studies 20 (2019) 2258–2276. [67] A. Haleem, M. Javaid, R. P. Singh, An era of chatgpt as a significant futuristic support tool: A study on features, abilities, and challenges, BenchCouncil transactions on benchmarks, standards and evaluations 2 (2022) 100089. [68] N. Muralidhar, H. Rangwala, E.-H. S. Han, Recommending temporally relevant news content from implicit feedback data, in: 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), IEEE, 2015, pp. 689–696. [69] A. D. Starke, S. Ø. Larsen, C. Trattner, Predicting feature-based similarity in the news domain using human judgments, in: Proceedings of the 9th international workshop on news recommendation and analytics (INRA 2021) co-located with 15th ACM conference on recommender systems (RecSys 2021), 2021. [70] M. Cinelli, G. De Francisci Morales, A. Galeazzi, W. Quattrociocchi, M. Starnini, The echo chamber effect on social media, Proceedings of the National Academy of Sciences 118 (2021) e2023301118. [71] E. R. Thompson, Development and validation of an internationally reliable short-form of the positive and negative affect schedule (panas), Journal of Cross-Cultural Psychology 38 (2007) 227–242. URL: https://doi.org/10.1177/0022022106297301. doi:10.1177/0022022106297301 . arXiv:https://doi.org/10.1177/0022022106297301 . [72] R. Fletcher, S. Park, The impact of trust in the news media on online news consumption and participation, Digital journalism 5 (2017) 1281–1299. [73] H. I. Chyi, Willingness to pay for online news: An empirical study on the viability of the subscription model, Journal of Media Economics 18 (2005) 131–142. URL: https://doi.org/10.1207/ s15327736me1802_4. doi:10.1207/s15327736me1802\_4 . [74] B. P. Knijnenburg, M. C. Willemsen, Evaluating recommender systems with user experiments, in: Recommender systems handbook, Springer, 2015, pp. 309–352. [75] D. DeSteno, R. E. Petty, D. D. Rucker, D. T. Wegener, J. Braverman, Discrete emotions and persuasion: the role of emotion-induced expectancies., Journal of personality and social psychology 86 (2004) 43. [76] R. Kühne, C. Schemer, The emotional effects of news frames on information processing and opinion formation, Communication Research 42 (2015) 387–407. [77] A. H. Eagly, S. Chaiken, The psychology of attitudes., Harcourt brace Jovanovich college publishers, 1993. [78] I. J. Roseman, C. Wiest, T. S. Swartz, Phenomenology, behaviors, and goals differentiate discrete emotions., Journal of personality and social psychology 67 (1994) 206. [79] J. M. Sontag, Visual framing effects on emotion and mental health message effectiveness, Journal of Communication in Healthcare 11 (2018) 30–47. [80] S. Lecheler, L. Bos, R. Vliegenthart, The mediating role of emotions: News framing effects on opinions about immigration, Journalism & Mass Communication Quarterly 92 (2015) 812–838. [81] R. L. Nabi, C. M. Dobmeier, C. L. Robbins, D. Pérez Torres, N. Walter, Effects of scanning health news headlines on trust in science: an emotional framing perspective, Health Communication (2024) 1–13.