TrustSearch: Analyzing the Stance of Media for Promoting Critical Thinking

Abstract

In the last decade, we have experienced a growing distrust in all sources of information. According to the Edelman Trust Barometer 2024, trust in information is at a record low. This lack of trust could be addressed by a new generation of search engines that promote critical thinking. A way to promote critical thinking is to ofer users information about the stance of articles towards important topics and to encourage them to consume more pluralistic information. In this paper, we present the stance detection tool, which is the first result of the TrustSearch project. The goal of the project is to promote critical thinking by providing users with a new experience of browsing news articles by emphasizing the pluralistic nature of information. The tool implements two diferent use-cases to detect an article's stance on some controversial topics such as climate change, immigration, and vaccination against COVID-19.

Keywords

• eol>Critical thinking
• stance detection
• large language models
• prompt engineering
• entailment

1. Introduction

3Horizon Europe Project funded by Next Generation Internet (NGI) Search: https://www.ngi.eu/ngi-projects/ngi-search/ spectives: 1) the point of view of the general population; polarization in the media sector. The core of NewsCube and 2) personalized user view. These two approaches was aspect level browsing, a method to provide readers were implemented in the interactive graphic tool that we with a classified view of a set of articles with diferent developed as part of the project, and which are presented aspects. Authors also explore the efect of a depiction in the following sections of the paper. of information in clusters conducting three user studies. One encouraging result in this research is that by presenting the news in classified clusters according to their 2. Related works content (NewsCube depiction), users read significantly more articles with diferent points of view compared to Stance detection is an emerging opinion mining paradigm the GoogleNews interface (list view). for various social and political applications [ 1 ]. Recently A similar efort was made at UC Berkeley, which crethere has been a growing research interest in detecting ated Opinion Space [ 10 ], a self-organizing interactive automatically the stance from opinionated texts, espe- visualization of an information space to encourage the cially if about controversial topics (e.g. vaccines, climate reading of a greater diversity of opinions in microblogchange, etc.). Stance detection is the text classification ging. The need to provide a multiple perspective about problem where the stance of the author of the text to- events, especially if controversial, to promote more critwards the target comes from this set of labels: {Favor, ical thinking has also captured the interest of the techAgainst, Neither} [ 2 ]. According to Küçük and Can [ 2 ], nology industry. In December of 2017, Bing4 launched stance detection is closely related to, but distinct from, several new Intelligent Answers that go beyond the travarious NLP tasks, such as: (1) sentiment analysis, (2) ditional Q&A style of search and ofer answers to more emotion recognition, (3) perspective identification, (4) complicated questions. Microsoft search engine was insarcasm/irony detection, (5) controversy detection, (6) terested in providing multiple perspectives to an answer argument mining, and (7) biased language detection. The via a tool that ofers the opposite points of views about a recent survey of ALDayel and Magdy (2021) [ 3 ] presents query. Unfortunately, this functionality is currently not an exhaustive review of stance detection techniques, the available in the search engine. diferent types of targets, the features set used, and the To the best of our knowledge, there is not an AI tool best performing machine learning / deep learning ap- that from one user query ofers a plurality of news in proaches. To solve stance detection problems, traditional diferent European news outlets to contrast the stance of machine learning approaches, such as support vector the news. This is the goal of the first part of the Trustmachine (SVM), logistic regression (LR), random forest Search project, and the result is the demo tool presented (RF) and k-nearest neighbors (kNN), were mainly used in this paper. until 2019 [ 2 ]. However, in recent years, deep learning approaches, such as long short-term memory (LSTM), gated recurrent unit (GRU), convolutional neural net- 3. Detecting stance of news works (CNN), graph neural networks (GNN) and transformers, have become increasingly popular [ 4 ]. The latest In order to face the stance detection problem of news trend is to use large language models (LLMs) as few-shot media articles regarding the topics of interest, we ex[ 5 ] or zero-shot [ 6 ] reasoners to solve this task. LLMs are periment with two diferent approaches. The first one, quite capable of handling the stance detection task, with named as Definition-based approach, is based on a fixed proper prompt engineering they achieve the state of the definition of the topic. The latter, named as User-driven art performance in zero-shot mode [ 7 ]. Furthermore, re- approach, is based on the user’s previous ideas about a formulating a potential NLP task into an entailment one topic. In order to reduce trial costs, in both approaches can lead to performance improvements [ 8 ]. Addressing we make predictions based on short summaries of arstance detection from an entailment perspective holds ticles rather than the full text of articles. We obtained promise in capturing the underlying logical structure of these summaries from full-text articles using GPT-3.5arguments and opinions. By considering the entailment Turbo5 model. The quality of the generated summaries relationships between statements, models can potentially was checked by three independent evaluators on 30 randiscern the stance expressed towards a target more accu- domly selected samples of the test corpus and all evaluarately. tors agreed that the quality of the selected samples was

The idea of changing the depiction of information to satisfactory. encourage users to be exposed to a plurality of sources and opinions has been directly approached in some research, for instance in Korea and the USA. The NewsCube project [ 9 ] was developed in Korea Advanced Institute of Science and Technology with the aim of combating

4https://www.bing.com 5https://platform.openai.com/docs/models/gpt-3-5

expanding to the general population. Additionally, strategies include ensuring vaccine supply, establishing vaccination centers, implementing public education campaigns, and monitoring vaccine safety and eficacy.” Immigration: “Immigration refers to the process by which individuals enter and settle in a foreign country with the intention of residing there permanently or for an extended period. It involves crossing international borders and often requires complying with legal requirements, such as obtaining visas or work permits. Immigration can be motivated by various factors, including economic opportunities, seeking asylum or refuge, family reunification, or pursuing education and better living conditions.”

3.1. Definition-based approach

The Definition-based stance detection approach is designed to provide an assessment of an article’s attitude towards a selected topic relative to the general population’s view. There are three possible responses to the article’s stance: 1. "The article is in favor w.r.t. the topic", if the article emphasizes the arguments in favor of the topic; 2. "The article is against w.r.t. the topic", if the article emphasizes the arguments against the topic; 3. "The article is neutral w.r.t. the topic", if the article does not emphasize ether of these arguments.

The definitions produced by the LLMs are consistent

with the man-made ones. We also used the LLMs to generate arguments in favor and against these topics and both LLMs generated arguments similar to those generated by humans. Finally, we searched for appropriate This approach relies on predefined topic definitions and prompts and confirmed that the both LLMs provide the predefined arguments in favor and against the topic. It well-formed responses needed to successfully extract the is based on prompt engineering and utilizes large lan- models’ predictions as zero-shot classifiers. guage models (LLMs) as zero-shot classifiers. According In performance experiments, we asked LLMs to evaluto Zhang et al. [ 7 ] and our experiments, LLMs are quite ate an article’s stance w.r.t. the selected topic. Together capable in solving this task, and their performance is with question we provide the LLMs with the summary of currently the state of the art. However, there are advan- the article body. In addition, to explore the reasoning betages and disadvantages of using LLMs. The upside is hind model predictions, we expand the query and make that LLMs can produce good performance with expla- these models provide explanations to elaborate their prenations without needing a specific dataset and training dictions. The prediction of models were verified by 2 for each topic. The downside is that using LLMs is more individuals and in most cases the estimation of LLMs expensive than using smaller models. There are also were consistent with humans. Next, we explored the issues specific to commercial LLMs. These models are possibility of obtaining more comprehensive information opaque, i.e. the models’ parameters are unknown. There about the stance of an article. The idea is to change the is also a risk in terms of service continuity, the company prediction type form classification into regression by promay stop providing access to the model in the future. viding an intensity estimate of an article stance, i.e. how In the experiments presented in this paper, we use two much the article is against or in favor of the topic. LLMs, GPT-3.5-Turbo and GPT-46, and three diferent For convenience, we unified the stance intensity into a topics: 1) climate change, 2) vaccination strategy against single score presented as continuous variable with values COVID-19 and 3) immigration. First, to ensure that the within interval [ -1, 1 ], where: 1) value -1 means that an selected LLMs had the correct conception about these article is strongly against the topic; 2) value 0 means that topics, we assessed the quality of topic definitions gener- an article is neutral to the topic; and value 1 means that ated by these LLMs. These are the definitions generated an article is strongly in favor of the topic. For this reaby GPT-4. son, we extended the zero-shot classification approach with LLMs using two diferent strategies: 1) the sentenceClimate change: “Climate change refers to long-term shifts in based strategy; and 2) the frequency-based strategy. The temperature patterns and weather conditions on Earth. It is primarily caused by human activities such as the burning of sentence-based strategy, is to detect stance on the senfossil fuels, deforestation, and industrial processes, which release tence level, i.e. estimating the stance class for each sengreenhouse gases into the atmosphere. These gases, like carbon dioxide, trap heat from the sun and result in a warming efect tence in article’s text separately. Afterwards, we calculate on the planet. Climate change leads to various efects, including the final score as a weighted sum of all individual senrising sea levels, extreme weather events, changes in ecosystems, and threats to human health and livelihoods. Taking action to tence stance scores, where the sentence length is used to reduce greenhouse gas emissions and adapt to these changes is calculate the weight/importance of the sentence and the crucial in addressing climate change.”

individual sentence score is produced by transforming iCnovvoildvevsagclcoinbaatlliyond:is“tTrhibeuvtainccginaantdioandsmtriantiesgtyerainggaianpstprCoOveVdIDv-a1c9- the sentence stance class: 1) ’in favor’ to 1; 2) ’neutral’ cines to prioritize high-risk groups such as healthcare workers, to 0; and 3) ’against’ to -1. The frequency-based strategy tshtreateeldgyeralyim,asntdo preedoupclee wtraitnhsmunisdseirolny,isnegvehreeailltlhnecsosnesd,iatinodnsd.eTahthiss is to detect stance on each article’s summary multiple caused by the virus. Vaccination campaigns usually follow a times and use the frequency of classes to calculate the phased approach, starting with priority groups and gradually ifnal intensity score. In the experiment we use 10 repetitions and calculate the mean value of all individual scores, where each individual score is given by transform6https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo ing the stance class: 1) ’in favor’ to 1; 2) ’neutral’ to 0; for bart-large-mnli9 after being trained on the MultiNLI and 3) ’against’ to -1. In our experiments, the sentence- (MNLI)10 dataset, then we have applied a NLI-based Zero based stance intensity strategy gives weak performance7 Shot Text Classification between two classes: the user for both LLMs, however the frequency-based strategy text and the opposite user text. The input consists basiachieves better results that are significantly correlated cally in the user’s question and the user’s stance. Indeed, with human annotations8. This is the reason we utilize the topic is not mandatory for this approach and the the frequency-based strategy to produce stance intensity opposite user text can be automatically generated. The scores for Definition-based approach within the tool. output consists of a binary classification between the aforementioned two classes. In this way we are able to 3.2. User-driven approach obtain a clear view of the relation between the user’s stance and the news article’s stance promoting the critiThe user-driven stance detection approach, rooted in en- cal thinking of the user. In contrast to traditional methtailment task, diverges from the Definition-based stance ods, this approach empowers users to actively guide the detection methodology by focusing on contextual nu- model’s understanding, making it particularly adaptable ances in user-generated content. Unlike the latter, which to diverse contexts and individual perspectives. The emaims to gauge the general population’s stance on a cho- phasis on user-driven input allows for a more nuanced sen topic, the user-driven approach centers around an and tailored stance detection process. It should be noted actual user query and their previous personal perspective that in this way we obtain a topic-independent approach (i.e., the stance) on the matter. The process begins with a which is totally generalized and applicable to real-world user query and with a newspaper article relevant to the scenarios. query. The user expresses their personal stance, indicating their initial perspective on the query. Subsequently, a Large Language Model (i.e., GPT-3.5-Turbo), is employed 4. Demo to generate a sentence that combines the query and the user’s stance leveraging on prompt engineering. To demonstrate these two stance detection approaches

The obtained sentence, hereinafter referred to as the and their utility in searching for relevant news, we imuser text encapsulates the context and user’s perspective, plemented them as interactive graphical tool. The demo forming a cohesive statement. To explore the opposing can be accessed directly by visiting the dedicated hugviewpoint, we utilize the LLM to generate another sen- gingface space11. The tool is composed of two interactive tence that represents the negation of the user text here- tabs. inafter referred to as the opposite user text. This sentence The first tab is based on the Definition-based stance encapsulates the opposite stance, providing a balanced detection approach. When the user selects a topic, the perspective for further analysis. Finally, we employ a definition of the topic will be displayed on the right side Transformer model designed for entailment tasks. This of the screen, and a list of news relevant to that topic model takes as input the user text, the opposite user text, will be available via the drop-down menu. Users can and the text of the retrieved newspaper article, in order select any of these articles by clicking on the article title. to assign scores indicating the likelihood of entailment Subsequently, a summary of the article will be displayed for both perspectives. In this context, the goal is to de- along with the estimated value of the article’s stance w.r.t. termine the entailment relation between two texts. In the selected topic, as shown in Figure 1. particular, given two paragraphs, named premise and The second tab is based on the User-driven stance dehypothesis respectively, the goal of the entailment task tection approach. In this case, the user selects a question is to identify whether the hypothesis is true based on related to the topic. Then, a real user’s stance and the the information given in the premise [ 11 ]. If we call the opposite stance will be displayed, together with a list of newspaper article text as premise and the user text (or news relevant to the topic as mentioned previously. Eventhe opposite user text) as hypothesis, we will be able to tually, the classification scores between the two classes obtain in output an answer to the following question: will be displayed w.r.t. the selected topic, as shown in “Is the user’s personal stance true based on the informa- Figure 2. tion contained in the article?”. In this regard, we have used a encoder-decoder Transformer-based model name 5. Conclusions BART [ 12 ] since the promising performance on the entailment tasks. In particular, we have used a checkpoint In this paper we present a tool for discovering news articles with the respect to the stance of the article’s author.

7Pearson correlation between predictions and human annotations is for: 1) English≈ 0.12; and 2) Spanish≈ 0.13.

8Pearson correlation between predictions and human annotations is for: 1) English≈ 0.63; and 2) Spanish≈ 0.48.

9https://huggingface.co/facebook/bart-large-mnli 10https://github.com/nyu-mll/multiNLI 11https://huggingface.co/spaces/UPV-PRHLT-NLP/TrustSearch

The tool implements two diferent approaches. The first views. one provides the article’s stance towards selected top- The tool is deployed as a web application which makes ics from a general population’s view. This approach is it easily accessible by users. All computation is done intended to provide users with means to explore informa- on the server side and no installation is required on the tion on topics about which they have no prior knowledge user side. The only requirement for users is Internet or personal views. It is also useful for exploring the views access and a standard web browser installed on a comof the general population on a chosen topic. The second puter. The tool provides a natural way of interaction one is a user-centric approach which provides informa- and presentation of results, which makes it suitable for tion on how well an article aligns with the user’s personal use by inexperienced users without additional training. Both approaches are powered by state-of-the-art LLMs utilized in zero-shot mode to detect the stance of articles.

The first approach is based on GPT-3.5-Turbo model and prompt engineering, while the second one is based on entailment and utilize BART [ 12 ] model. To the best of our knowledge, this is the first time entailment prediction has been used to detect stance of articles.

However, this pilot tool has some limitations. First, the tool currently only provides selection from a limited number of samples for each input field. But, the ultimate goal is to allow all inputs to be in free text format, which will allow for a wider application of the tool. Second, the tool displays the stance detection result only after selecting one of the articles. While this method is useful for obtaining stance information, a more efective way to explore the diferent views that exist in the media landscape will require ofering diferent news together.

5.1. Future work

To address the limitations of the current version of the tool, in the the second part of TrustSearch project, we plan to apply two major updates to it. First, we plan to change the way of browsing articles from a list view to a customized graphical view, where the physical position of the article in the graph shows the stance of the article according to: 1) the selected topic or 2) personal stance. Second, we plan to combine stance detection with information on the ideological orientation of the media outlet, as well as information on its audience and its age. Finally, we plan to test the hypothesis that combining these two pieces of information, on content and on source characteristics, can help to approach search results with greater confidence and promote more critical thinking.

6. Acknowledgements

This work was carried out in the framework of TrustSearch, a project funded by NGI Search (Horizon Europe Project) Support Programme.

References

[1] M. Lai , M. Tambuscio , V. Patti , G. Rufo, P. Rosso , Stance polarity in political debates: A diachronic perspective of network homophily and conversations on twitter , Data Knowledge Engineering 124 ( 2019 ) 101738 .

[2] D. Küçük , F. Can , Stance detection: A survey, ACM Computing Surveys (CSUR) 53 ( 2020 ) 1 - 37 . URL: https://doi.org/10.1145/3369026. doi: 10 . 1145/3369026.

[3] A. ALDayel , W. Magdy, Stance detection on social media: State of the art and trends , Information Processing amp; Management 58 ( 2021 ) 102597 . URL: http://dx.doi.org/10.1016/j.ipm. 2021 . 102597 . doi: 10 .1016/j.ipm. 2021 . 102597 .

[4] N. Alturayeif , H. Luqman , M. Ahmed , A systematic review of machine learning techniques for stance detection and its applications , Neural Computing and Applications 35 ( 2023 ) 5113 - 5144 . URL: https://doi.org/10.1007/s00521-023-08285-7. doi: 10 .1007/s00521-023-08285-7.

[5] T. Brown , et al., Language models are fewshot learners , in: H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , H. Lin (Eds.), Advances in Neural Information Processing Systems , volume 33 , Curran Associates , Inc., 2020 , pp. 1877 - 1901 . URL: https://proceedings. neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[6] T. Kojima , S. S. Gu , M. Reid , Y. Matsuo , Y. Iwasawa , Large language models are zero-shot reasoners , in: S. Koyejo, S. Mohamed , A. Agarwal , D. Belgrave , K. Cho , A . Oh (Eds.), Advances in Neural Information Processing Systems , volume 35 , Curran Associates , Inc., 2022 , pp. 22199 - 22213 .

[7] B. Zhang , D. Ding , L. Jing, How would stance detection techniques evolve after the launch of chatgpt ?, 2023 . arXiv: 2212 . 14548 .

[8] S. Wang , H. Fang , M. Khabsa , H. Mao , H. Ma, Entailment as few-shot learner , CoRR abs/2104 .14690 ( 2021 ). arXiv: 2104 . 14690 .

[9] S. Park , S. Kang , S. Chung , J. Song , Newscube: Delivering multiple aspects of news to mitigate media bias , in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09 , Association for Computing Machinery, New York, NY, USA, 2009 , p. 443 - 452 . doi: 10 .1145/1518701. 1518772.

[10] S. Faridani , E. Bitton , K. Ryokai , K. Goldberg , Opinion space: A scalable tool for browsing online comments , volume 2 , 2010 , pp. 1175 - 1184 .

[11] I. Dagan , O. Glickman , B. Magnini , The pascal recognising textual entailment challenge , 2005 , pp. 177 - 190 . doi:https://doi.org/10.1007/ 11736790_9.

[12] M. Lewis , Y. Liu , N. Goyal , M. Ghazvininejad , A. Mohamed , O. Levy , V. Stoyanov , L. Zettlemoyer, BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension , in: D. Jurafsky , J. Chai , N. Schluter , J. Tetreault (Eds.), Proc. of the 58th Annual Meeting of the Association for Computational Linguistics , ACL , Online, 2020 , pp. 7871 - 7880 .