Personalized recommender systems facilitate decision-making in various domains by presenting content closely aligned with users' preferences. However, personalization can lead to unintended consequences. In news, selective information exposure and consumption might amplify polarization, as users are empowered to seek out information that is in line with their own attitudes and viewpoints. However, personalization in terms of algorithmic content and persuasive technology could also help to narrow the gap between polarized user attitudes and news consumption patterns. This paper presents a pilot study on climate change news. We examined the relation between users' level of environmental concern, their preferences for news articles, and news article content. We aimed to capture a news article's viewpoint through sentiment analysis. Users (N = 180) were asked to read and evaluate 10 news articles from the Washington Post. We found a positive correlation between users' level of environmental concern and whether they liked the article. In contrast, no significant correlation was found between sentiment and environmental concern. We argue why a different type of news article analysis than sentiment is needed. Finally, we present our research agenda on how persuasive technology might help to support more exploration of news article viewpoints in the future.
It is believed that people tend to seek out information that support their own viewpoints. Users
of online news media are more likely to interact with opinion reinforcing information [19]. For a
variety of topics, this may stem from a person’s attitudinal disposition towards a certain issue [
This issue has been amplified by the introduction of personalized news recommender
systems. These involve algorithms and interfaces that present content based on what users liked
in the past [25]. Consequently, this has led to the notion of people being in a ‘filter bubble’, which
has gained popularity since 2011 [42]. Nonetheless, recent studies have sowed doubt about the
existence of such an effect. While users might seek out opinion-reinforcing content [
Digital news media face a tradeoff between presenting content that either challenges or
confirms a user’s viewpoints. The latter is the rationale used by most personalized news
technologies [25], while persuasive technology has the potential to help with the former. People’s
attitudes and beliefs are important determinants of what news is consumed, but also what news
they are presented [28,52]. Persuading users to also consume news that is at odds with their
viewpoints, might mitigate selective exposure. This might help to reduce polarization among
individuals, for selective exposure tends to reinforce existing attitudes [
Traditional news recommender systems are ill-equipped to mitigate this problem, for they
only focus on short-term preferences [25]. According to [
We argue that different approaches are needed. Both algorithmic and interface driven
approaches that aim to persuade people to engage with more diverse news. While in this paper
we examine a short-term scenario, we aim to eventually change attitudes over a longer time
period. According to recent studies [
As a first step, we examine the relation between user attitudes, news article content, and user
news consumption and preferences. We consider the complex domain of climate change, where
people have varying opinions about measures that should be taken and the impact it will have.
This can be operationalized through their environmental attitudes and levels of environmental
concern, which have both been associated with specific news avoidance [
A method to operationalize opinions voiced in a news article is sentiment analysis. It has been used in previous studies to capture how an author thinks about a topic at hand or how an opinion is being put forward [44]. This method quantifies the valence of the text in a news article, by differentiating between negative and positive text We expect that the valence is related to the main topic at hand discussed. For the domain of climate change, we expect that authors of news articles will discuss the negative impacts of climate change if they consider this to be a serious threat, which is expected to be related to a user’s level of environmental concern.
This paper takes an algorithmic approach. We examine whether we can predict user preferences based on a user’s level of environmental concern and news article characteristics. Regarding the latter, we operationalize the opinion voiced in a news article through sentiment analysis, as well as consider basic characteristics, such as word length. We formulate the following research question: – RQ: To what extent can preferences for climate change news articles be predicted from a user’s environmental concern and a news article’s sentiment?
We set up a research platform for presenting climate change news articles to users. US-based participants (N = 180) were invited from the crowdsourcing platform Prolific and asked about their level of environmental concern. Subsequently, each participant was presented 10 news articles on climate change, for which they indicated whether they liked and trusted it. Upon completion, we comprehensively analyzed the correlations between relevant variables and constructed regression models utilizing the observed data.
We employed a single news source for our study. News articles were obtained from the TREC Washington Post Corpus [39], which was also used in previous news recommender research [48]. The collection was published between January 2012 and December 2020. Each news article consisted of a title, main body text including section headers, author, a short author biography, and date of publication. This is depicted in Figure 1. Among the larger corpus, we only retrieved news articles from the Climate & Environment category, which consisted of 2,368 articles in total. After data cleaning, deleting missing values and duplicate data, a total of 1,276 news articles remained.
For our study, we sampled 100 news articles from the larger set of 1,276 articles in the Climate & Environment category. See Table 1 for an overview of the filtering process. A sentiment analysis was performed on each news article. It is important to point out that each news article represented a journalistic discussion of climate change, for which it could be assumed that they were either neutral or pro-climate.
We invited 180 participants (54% male) through the crowdsourcing platform Prolific. Each participant received compensation of 1.25 GBP for their participation in the study, which required approximately 15 minutes per user. With respect to demographics, 32% of participants fell within the 25 to 35 age range.
Our experiment was hosted on a platform that consisted of both front-end and back-end systems. The front-end interface used JavaScript, HTML, and CSS for visual layout. Meanwhile, the back-end system was built using Django and Python, incorporating the TREC dataset, SQLite, and machine learning scripts for both regression models and sentiment detection. Specifically, we implemented the regression models using R, while Python was used for sentiment detection.
The procedure of the online experiment is depicted in Figure 2. After briefing participants on the purpose of the study, we inquired on demographic information, such as age, gender, education level, and country of origin.
Thereafter, participants completed a questionnaire to measure their level of Environmental Concern. Subsequently, we randomly sampled ten news articles from our dataset, which were presented to the users in a randomized order, and at the same time avoiding duplicate news article. Each user was asked to read the presented news article, after which they were asked to respond to four questionnaire items related to their evaluation: the extent to which they read the news article, trusted it, agreed with it, and whether they would recommend it to others (all on 5point scales). Three attention checks were implemented underneath three news articles [43].
Sentiment Analysis. To examine whether a news article’s sentiment was related to a user’s
level of environmental concern, we analyzed the sentiment of each article’s title and body text. To
this end, we utilized SentiStrength, taking a maximum of 100 words for each news article. The
primary reason for selecting SentiStrength among other tools was based on two factors. First, it
has been extensively used in scientific research through peer review [45]. Second, we compared
SentiStrength with other widely used sentiment detection tools, including Vader and TextBlob
(see Figure 3). Our analysis revealed that, when the text limitation is 100 words, Vader and
SentiStrength reported the strongest correlation between a news article’s title and body text,
ahead of TextBlob, with 0.685 and 0.623 respectively. In addition, following the comparison of 21
sentiment detection tools, Sentistrength had the highest degree of accuracy in English [
Environmental Concern. To assess a user’s viewpoint on climate change, we considered their
level of environmental concern. We adopted items from [
User Evaluation and Preferences. We further examined how users evaluated each news article. We focused on four different aspects: (a) reading behavior, (b) trust, (c) agreement, and (d) recommending the news article to others. For each aspect, we formulated a proposition: (a) I have read the entire news article, (b) I trust the article’s content, (c) I agree with the article’s content and (d) I would recommend the chosen article to others. Each proposition was measured on a 5-point Likert scale.
A descriptive overview of user responses is depicted in Figure 5. The majority of participants agreed with the propositions related to reading, trusting, agreeing, and recommending news articles to others, where reading had the highest level of agreement.
Table 2: Results from the confirmatory factor analysis, performed in MPlus. It describes a
twostep model, with items loading onto sub-aspects (e.g., ‘Balance of Nature’), which in turn load onto
the environmental concern aspect. Loadings of the aspects are for environmental concern. All
items were based on [
Comparative Fit Index (CFI): 0.980 R2
To address our research question, we examined the relationships between critical variables involving sentiment in the news articles, user preferences, total words in articles, and users’ level of environmental concern. Afterwards, we performed two regression analyses involving four different models, predicting to what extent users liked and read our news articles.
First, we performed a correlation analysis between the different measures in our study. The results are visualized in Figure 6, only depicting correlational strengths for significant relations. Our study revealed three primary findings regarding user preferences, their level of environmental concern, and news sentiment.
User Preferences. Figure 6 indicates that the user evaluation items were strongly correlated with each other. As this raised doubts about the validity of the observed variables and investigate the potential presence of an latent variable as the causal factor, we performed an exploratory factor analysis. This initial finding revealed that user preferences for trust, agreement, and recommendation could be considered to be a latent evaluative aspect, which was labelled as “Like”. This factor was used for further analysis (cf. Figure 6. The results of the factor analysis are described in Table 3.
Relation between User Evaluation and Environmental Concern. The correlational analysis showed a significant correlation between our ‘Like’ latent factor and environmental concern: r = 0.42, p < 0.001. This indicated that for our collection of climate change articles, people who had a higher level of environmental concern were also more likely to ‘like’ the news article. Given the underlying items used, this suggested that users who indicated to trust and agree with a climate change news article were more likely to be concerned, and vice versa.
News Article Sentiment. A striking finding was that a news article’s sentiment was not correlated to other characteristics. Not with environmental concern (p > 0.05), nor with ‘Like’ (p > 0.05). Moreover, the correlation between title and body text was only found to be weak (r = 0.3, p < 0.001). This was in spite of observing variation in the sentiment across different news articles. These findings suggested that, despite all news articles discussing climate change in a serious and cautious manner, the sentiment in a news article could not be related to a user’s attitudinal disposition. On the one hand, this indicated that sentiment analysis was not appropriate to detect opinions for the climate change domain. On the other hand, it was possible that sentiment analysis was not suitable for journalistic articles.
To further examine the relation between user evaluation, concern, and text characteristics, we finally performed linear regression analyses. The results are reported in Table 4. We performed the multivariate regression analysis to predict user preferences using their level of environmental concern and the other measures. Two models were developed: a full and a reduced one.
Liking Prediction. Initially, we constructed a model to predict the extent to which users evaluate a news article positively. Table 4 reveals that a user’s level of environmental concern positively predicted whether a positive user evaluation (i.e., ‘Like’), while the total number of words did so negatively. This applied to both the full model and the reduced model.
In contrast, both title and body text sentiment did not significantly affect liking. This suggested that the text’s sentiment did not affect user evaluations, having no predictive value.
Reading Prediction. Subsequently, we developed an additional model to predict the extent to which users read a news article. Although we instructed each user to read each news article carefully, the task involved might have led to users skipping content.
Similar to the model for ‘Like’, we found that environmental concern and total words significantly predicted whether a user read a news article, with a positive and a negative effect, respectively. This suggested that people with a higher level of environmental concern seemed to be more interested in engaging with news articles on climate change, while they were discouraged from reading longer news articles.
Model Comparison. Finally, we compared the two adjusted models in Table 4. In summary, the Like had a higher R2, yielding 0.174, compared to the 0.011 of the Read model. This suggested that the findings from the ‘Like’ were more relevant than those obtained from the ‘Read’ model.
This study has utilized a news research platform to explore the relationships between a user’s environmental concern, their preferences, and news article sentiment for news on climate change. We have done so by inquiring on user evaluations based on reading specific news articles in an experimental setting. Our findings have revealed that users’ environmental concern correlates positively with evaluations of climate change news articles, in which climate change is discussed as a real and serious phenomenon or threat. The evaluation has been operationalized as liking a news article, based on questionnaire items related to trusting a news article, agreeing with it, and recommending it to others.
A similar finding is shown in our regression analyses. Users’ level of environmental concern
positively predicted the extent to which users would like and read news articles. This finding is
not only in line with previous work in environmental psychology [
In addition, we have observed that the total number of words in each article significantly influenced the users’ preference for liking and reading the news. This reduced the extent to which users engaged with or liked the news article. This could indicate that users have been less motivated to read longer news articles, which might have affected their judgment. This is potential confounding factor to our results.
Our study’s evaluative findings are consistent with previous research on political news.
According to [
Arguably surprisingly, we have not discovered a significant correlation between a news article’s sentiment and other variables. While we have expected that the sentiment in a news article would be useful insights into its relationship with other variables (in this case: climate change), it does not seem to apply in this study. Since all news articles concern climate change, it would seem reasonable that a correlation between sentiment and environmental concern could be established.
We propose a number of arguments why this is not the case. First, climate change can be
discussed ambiguously in terms of negativity or positivity. One could write very negatively about
the consequences of climate change while being pro-environment, and vice versa. Moreover,
having a positive attitude towards climate change does not equal being pro-environment, for
climate change is a negative phenomenon. Second, another possibility is that because our news
articles are written in a rather neutral manner, sentiment analysis is not an appropriate method
[
Future research could also consider other factors that influence the variation in sentiment
distributions. According to [29], various factors such as the topic, time of day, and user
characteristics can affect how produced news is expressed in terms of sentiment. For instance,
users tend to express more positive sentiment on weekends. Moreover, different demographics
might lead to different sentiments[
Furthermore, the extent to which sentiment varies per news source may differ strongly across news sources. The main opinion voiced by such an outlet may then be rather homogeneous, leading to too little variation when sampling a news article dataset from a single source. Two recent studies have shown that there are differences in polarization across news sources; one study examined climate change coverage and found that The Wall Street Journal is less likely to discuss the impacts and potential threats of climate change, and more likely to present negative efficacy information than other news sources such as the Washington Post [17]. Another study found that Fox News portrayed immigration policy and undocumented immigrants more negatively than other news sources such as The New York Times, The Washington Post, CNN, and MSNBC [24]. Therefore, our study’s correlation between sentiment and other variables may vary depending on the news source used, as well as the topic domain at hand.
Based on our discussion, we argue to explore demographic factors as well as other news
domains. It seems reasonable that a more straightforward topic in terms being ‘pro’ or ‘against’
[
The main limitation of our study is the absence of a functional recommender system within our news research platform. This could have offered participants personalized news recommendations. Instead, we provided a randomly selected list of news articles and built regression models using the observed data from participants.
Moreover, we could not control how well people have read the different news articles, except for the self-report question. Hence, in a follow-up study, we intend to construct a recommender system that takes into account the data we obtained in this research.
It has also been a limiting factor that we have not been able to measure trust and agreeing
separately. Nonetheless, the strong correlation between these factors is in line with previous
research. For example,the decision to share social media news is heavily influenced by liking and
agreeing (known as selective exposure). Notably, they also depend on whether people trust the
news source [
This paper outlines a pilot study. With this research line we aim to support a more diverse news consumption in the context of personalized news media. Whereas we seek to take the algorithm of the news recommender ‘off the shelf’, our main focus will be on the interface perspective as a means of altering people’s attitudes. We seek to use persuasive technology to explain to users whether news articles are aligned with their level of attitude or not, and whether they could diversify their reading taste.
One approach would be to use informational nudges. In the context of environmental behavior and decision-making, informational nudges have been identified as an effective approach for reducing environmental damage at the individual level [37], In addition, social nudges can be used to alter negative behaviors by leveraging the tendency of large groups to conform [22,36]. In our forthcoming study, we intend to employ different types of nudges to alter people’s attitudes.
Since sentiment analysis might not be effective in some journalistic outlets, our subsequent
research could incorporate an examination of the author’s stance to ensure a more
comprehensive understanding. This way, we aim to increase the diversity of news articles
recommended based on how the author’s and the user’s stance are aligned, with the goal of
expanding their exposure to different perspectives [
Changing user choices with nudges and persuasion is one of the possible aspects in recommender system research that has been explored. Nudges are said to be one of the means to facilitate attitudinal and behavioral change [50, 33]. Various types of nudges have been shown to have an impact on people’s behaviors in a wide range of areas, including politics, commerce, and health [51, 37,56,41].
In future studies, the factor related to user preferences observed in this research could be used to assess user experience. Additionally, the user preference patterns identified in this study, such as sentiment and news appeal, may be employed to investigate potential intriguing differences in subsequent research involving diverse news sources. With respect to the regression models, it would be valuable to not only incorporate novel predictors but also to compare the predictors employed in this study during future investigations, to verify their contribution to users’ news consumption patterns. This evidence may be instrumental in persuading or altering users’ attitude in future research.
Subsequent studies should take into account a range of factors to comprehensively investigate the relationships between users’ stances and the impact of polarized news sources on people’s preferences. Firstly, it is recommended to incorporate a wider selection of polarized news sources to expand upon the existing findings. Secondly, our proposed approach is to utilize persuasive technologies, specifically news recommender systems and nudges, to provide diverse news content and implement manipulations through nudges for readers with differing stances. This methodology has the potential to enhance the diversity of perspectives available to readers, while also enabling the evaluation of the extent to which they may influence users’ preferences or stances. By considering these factors, future research can provide a more nuanced understanding of individuals’ underlying preferences for news sources based on differing stances. These varying perspectives can be presented through both content (algorithmic persuasion) and explanations (persuasive communication).
We also propose to examine attitudinal changes over time. Such work would fall in line with Garreton et al. [18], who have examined attitudinal changes over a few weeks time, due to news articles either being presented with illustrations or visualizations.
This work was in part supported by the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the Centre for Research-based Innovation scheme, project number 309339. [17] Feldman, L., Hart, P.S., Milosevic, T.: Polarizing news? representations of threat and efficacy in leading us newspapers’ coverage of climate change. Public Understanding of Science 26(4), 481–497 (2017) [18] Garret´on, M., Morini, F., Celhay, P., Do¨rk, M., Parra, D.: Attitudinal effects of data visualizations and illustrations in data stories. IEEE Transactions on Visualization and Computer Graphics (2023) [19] Garrett, R.K.: Echo chambers online?: Politically motivated selective exposure among internet news users. Journal of computer-mediated communication 14(2), 265–285 (2009) [20] Gomez-Uribe, C.A., Hunt, N.: The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS) 6(4), 1–19 (2015) [21] Haim, M., Graefe, A., Brosius, H.B.: Burst of the filter bubble? effects of personalization on the diversity of google news. Digital journalism 6(3), 330–343 (2018) [22] Jesse, M., Jannach, D.: Digital nudging with recommender systems: Survey and future directions. Computers in Human Behavior Reports 3, 100052 (2021) [23] Joris, G., Vercoutere, S., De Clercq, O., Van Damme, K., Mechant, P., De Marez, L.: Nudging towards exposure diversity: Examining the effects of news recommender design on audiences’ news exposure behaviours and perceptions. Digital Journalism pp. 1–22 (2022) [24] Joseph, T.D.: Framing immigrants: News coverage, public opinion, and policy (2018) [25] Karimi, M., Jannach, D., Jugovac, M.: News recommender systems–survey and roads ahead. Information Processing & Management 54(6), 1203–1227 (2018) [26] Kellstedt, P.M., Zahran, S., Vedlitz, A.: Personal efficacy, the information environment, and attitudes toward global warming and climate change in the united states. Risk Analysis: An International Journal 28(1), 113–126 (2008) [27] Knijnenburg, B.P., Willemsen, M.C.: Evaluating recommender systems with user experiments. Recommender systems handbook pp. 309–352 (2015) [28] Knobloch-Westerwick, S., Meng, J.: Looking the other way: Selective exposure to attitudeconsistent and counterattitudinal political information. Communication Research 36(3), 426–448 (2009) [29] Kucuktunc, O., Cambazoglu, B.B., Weber, I., Ferhatosmanoglu, H.: A large-scale sentiment analysis for yahoo! answers. In: Proceedings of the fifth ACM international conference on Web search and data mining. pp. 633–642 (2012) [30] Ling, R.: Confirmation bias in the era of mobile news consumption: the social and psychological dimensions. Digital Journalism 8(5), 596–604 (2020) [31] Ludwig, K., Grote, A., Iana, A., Alam, M., Paulheim, H., Sack, H., Weinhardt, C., Mu¨ller, P.: Divided by the algorithm? the (limited) effects of content-and sentiment-based news recommendation on affective, ideological, and perceived polarization. Social Science Computer Review p. 08944393221149290 (2023) [32] Matthew, J.R.: Netflix and the design of the audience: The homogenous constraints of data-driven personalization. MedieKultur: Journal of media and communication research 36(69), 052–070 (2020) [33] Mattis, N., Masur, P., M¨oller, J., van Atteveldt, W.: Nudging towards news diversity: A theoretical framework for facilitating diverse news consumption through recommender design. new media & society p. 14614448221104413 (2022) [34] McMillan, M., Hoban, T.J., Clifford, W.B., Brant, M.R.: Social and demographic influences on environmental attitudes. Journal of Rural Social Sciences 13(1), 5 (1997) [35] Mohammed, D.J., Aleqabie, H.J.: The enrichment of mvsa twitter data via captiongenerated label using sentiment analysis. In: 2022 Iraqi International Conference on Communication and Information Technologies (IICCIT). pp. 322–327. IEEE (2022) [36] Nagatsu, M.: Social nudges: their mechanisms and justification. Review of Philosophy and
Psychology 6, 481–494 (2015) [37] Nelson, K.M., Bauer, M.K., Partelow, S.: Informational nudges to encourage proenvironmental behavior: Examining differences in message framing and human interaction. Frontiers in Communication 5, 610186 (2021) [38] Neumann, T., Kelm, O., Dohle, M.: Polarisation and silencing others during the covid-19 pandemic in germany: An experimental study using algorithmically curated online environments. Javnost-The Public 28(3), 323–339 (2021) [39] NIST: Trec washington post corpus (2019),data retrieved from: https://trec.nist.gov/data/wapost/ [40] O’Connor, R.E., Bord, R.J., Yarnal, B., Wiefek, N.: Who wants to reduce greenhouse gas emissions? Social Science Quarterly 83(1), 1–17 (2002) [41] Osei-Frimpong, K., Donkor, G., Owusu-Frimpong, N.: The impact of celebrity endorsement on consumer purchase intention: An emerging market perspective. [42] Journal of marketing theory and practice 27(1), 103–121 (2019) [43] Pariser, E.: The filter bubble: How the new personalized web is changing what we read and how we think. Penguin (2011) [44] Prolific: Prolific’s attention and comprehension check policy (2023), data retrieved from: https://researcher-help.prolific.co/hc/en-gb/articles/360009223553-ProlificsAttention-and-Comprehension-Check-Policy [45] Robertson, C.E., Pro¨llochs, N., Schwarzenegger, K., Pa¨rnamets, P., Van Bavel, J.J., Feuerriegel, S.: Negativity drives online news consumption. Nature Human Behaviour pp. 1–11 (2023) [46] Sentistrength: Evaluation in the peer-reviewed academic articles (2023), data retrieved from: http://sentistrength.wlv.ac.uk/ [47] Singh, S.: Impact of color on marketing. Management decision 44(6), 783–789 (2006) [48] Starke, A., Asotic, E., Trattner, C.: “serving each user”: Supporting different eating goals through a multi-list recommender interface. In: Proceedings of the 15th ACM Conference on Recommender Systems. pp. 124–132 (2021) [49] Starke, A.D., Larsen, S.Ø., Trattner, C.: Predicting feature-based similarity in the news domain using human judgments. In: Proceedings of the 9th International Workshop on News Recommendation and Analytics (INRA 2021) co-located with 15th ACM Conference on Recommender Systems (RecSys 2021) (2021) [50] Starke, A.D., Trattner, C.: Promoting healthy food choices online: a case for multilist recommender systems. In: Proceedings of the ACM IUI 2021 Workshops (2021) 50. Starke, A.D., Willemsen, M.C., Snijders, C.: With a little help from my peers: Depicting social norms in a recommender interface to promote energy conservation. In: Proceedings of the 25th international conference on intelligent user interfaces. pp. 568– 578 (2020) [51] Starke, A.D., Willemsen, M.C., Trattner, C.: Nudging healthy choices in food search through visual attractiveness. Frontiers in Artificial Intelligence 4, 621743 (2021) [52] Stroud, N.J.: Media use and political predispositions: Revisiting the concept of selective exposure. Political Behavior 30, 341–366 (2008) [53] Sunstein, C.: Republic. com 2.0. princeton university press (2007) [54] Trattner, C., Elsweiler, D.: Investigating the healthiness of internet-sourced recipes: implications for meal planning and recommender systems. In: Proceedings of the 26th international conference on world wide web. pp. 489–498 (2017) [55] Trattner, C., Jannach, D., Motta, E., Costera Meijer, I., Diakopoulos, N., Elahi, M., Opdahl, A.L., Tessem, B., Borch, N., Fjeld, M., et al.: Responsible media technology and ai: challenges and research directions. AI and Ethics 2(4), 585–594 (2022) [56] Wei, P.S., Lu, H.P.: An examination of the celebrity endorsements and online customer reviews influence female consumers’ shopping behavior. Computers in Human Behavior 29(1), 193–201 (2013) [57] Gil de Zu´n˜iga, H., Correa, T., Valenzuela, S.: Selective exposure to cable news and immigration in the us: The relationship between fox news, cnn, and attitudes toward mexican immigrants. Journal of Broadcasting & Electronic Media 56(4), 597–615 (2012)