Slanted news coverage strongly afects public opinion. This is especially true for coverage on politics and related issues, where studies have shown that bias in the news may influence elections and other collective decisions. Due to its viable importance, news coverage has long been studied in the social sciences, resulting in comprehensive models to describe it and efective yet costly methods to analyze it, such as content analysis. We present an in-progress system for news recommendation that is the first to automate the manual procedure of content analysis to reveal person-targeting biases in news articles reporting on policy issues. In a large-scale user study, we find very promising results regarding this interdisciplinary research direction. Our recommender detects and reveals substantial frames that are actually present in individual news articles. In contrast, prior work rather only facilitates the visibility of biases, e.g., by distinguishing left- and right-wing outlets. Further, our study shows that recommending news articles that diferently frame an event significantly improves respondents' awareness of bias.
How topics are covered in the news frames public debates and profoundly impacts collective decision-making, such as during elections [1, 2]. News may be subtly biased through various forms, such as word choice, framing, intentional omission or misrepresentation of specific details [3]. In extreme cases, “fake news” may present entirely fabricated facts to intentionally manipulate public opinion toward a given topic. A rich diversity of opinions is desirable, but systematically biased information can be problematic as a basis for decision-making if not recognized as such. Therefore, it is crucial to empower newsreaders in recognizing relative biases in coverage.
In this paper, we thus seek to answer the following research question. “How can we efectively
communicate instances of media bias in a set of news articles reporting on the same political
event?” Instead of identifying and then communicating each of the various forms of bias
individually, we focus on person-oriented polarity, which is a fundamental efect resulting from
various bias forms [4]. While the problem statement misses bias not related to persons, coverage
on policy issues is largely person-oriented, e.g., because decisions are made by politicians
or afect individuals in society. Our key contributions—especially when comparing to prior
work—are: (
We publish the survey materials, including questionnaires and anonymized answers, articles, and visualizations freely at: https://doi.org/10.5281/zenodo.5517401
Media bias has been long studied in the social sciences, resulting in a comprehensive set of models to describe it, such as political framing [5] and the news production process defining causes, forms, and efects of bias [ 3], and efective methods to analyze it [ 3]. Established methods, such as content analysis and frame analysis [6], typically include systematic reading and labeling of texts. Despite their high efectiveness and reliability, they are largely conducted manually and do not scale with the vast amount of news. In contrast, many methods in computer science concerned with media bias employ automated and thus more eficient approaches but yield non-optimal results [3], e.g., because they treat bias as only vaguely defined “topic diversity” [7] or “diferences in [news] coverage” [ 8]. Though, methods for the identification of biased words exist for other domains, such as Wikipedia articles [9].
Helping news consumers to become aware of media bias is an efective means to mitigate the negative efects of slanted news coverage, such as polarization [ 8, 10]. Moreover, most studies ifnd that automated approaches concerned with the communication of biases in the news can successfully increase bias-awareness in news consumers [11, 12, 13, 14]. However, previous approaches sufer from at least one of the following shortcomings. First, researchers cannot quantitatively pinpoint which individual components facilitate bias-awareness [11, 12, 13, 14]. Instead, studies measure overall efectiveness of analysis or visualizations. Second, approaches are concerned with the identification or communication of biases only in titles [ 11], on the article-level [8, 14], or outlet-level [15]. Considering only the overall article or even properties of its publisher, may lead to incorrect classification or missing instances of bias, e.g., that afect readers’ perception on the sentence-level.
In sum, many approaches efectively communicate biases to users and most studies indicate how society benefits from doing so. However, many approaches identify only vaguely defined or superficial biases, e.g., because they do not use the established, efective models and analyses. Further, none of the reviewed approaches narrows down efectiveness regarding change in bias-awareness to individual analysis and visualization components. Lastly, to our knowledge, no approach identifies biases on the sentence level in news articles.
Given a set of news articles reporting on the same political event, our system’s analysis aims to ifnd groups of articles that frame the event similarly using four analysis tasks. These groups are later visualized (Section 4) to enable non-expert news consumers to quickly get a bias-sensitive synopsis of a given news event.
For (
(
The last step of frame identification is to determine groups of articles that similarly frame
the event, i.e., the persons involved in the event. We currently use a simple, polarity-based
method that first determines the person that occurs most frequently across all articles, named
most frequent actor (MFA). Then, the method assigns each article to one of three groups,
depending on whether the article’s MFA mentions are mostly positive, ambivalent, or negative.
We also calculate each article’s relevance to the (
The overall workflow follows typical online news consumption, i.e., users see first an overview of news events and then view individual news articles. To measure efectiveness not only of our visualizations but also their constituents, we design them so that their components can be altered. To more precisely measure the change in bias-awareness concerning only the textual
Tags shown near a headline denote the political orientation (as self-identi!ed by the publisher) of its article ( left center right ) and how the article possibly portrays (determined automatically) the topic's main person President Donald Trump ( possibly contra possibly ambivalent possibly pro ).
Trump, Congress Reach Agreement On 2-Year Budget Deal center possibly pro President Trump announced an agreement on a two-year budget deal and debt-ceiling increase. The deal would raise the debt ceiling past the 2020 elections and set $1.3 trillion for defense and domestic spending over the next two years. […]
Each article is assigned to either of the following groups depending on how the article portrays the main person. Below, you see for each group its most representative article. To determine how an article reports on the main person, we automatically classify the sentiment of all mentions of that person. In this topic, the main person is:
President Donald Trump
The ovFeurvrtihewer airmticsletos enable users to quickly get a synopsis of a news event. We devise three
visualizations. (
The layoriguhtt opfosPsiobllySaimdbeivsalaenntd MFAP is vertically divided in three parts, two of which are shown in Figu▶reT1ru.mTphAenenvoeunntc’essmDeaailnOanrtDicelbet L(pimaritt, SAp)enshdionwg sCatphse leefvtepnosts’isblymamobsivtalrenetpresentative article. The comparative bias-groups part (C) shows up to three frames present in event coverage, by showcasing each frame’s most representative article. PolSides yields these frames by grouping
Continue: Click here when you !nished reading. articles depending on their political orientation (left, center, and right) [15]. For MFAP, we use our polarity-based grouping (Section 3) so that the resulting groups represent frames that are
Progress: 57% primarily in favor, against, or ambivalent regarding the event’s MFA. Conceptually, PolSides employs the left-right dichotomy, which is a simple yet often efective means to partition the media into distinctive slants. However, this dichotomy is determined only on the outlet-level and thus may incorrectly classify event-specific framing, e.g., articles with diferent perspectives having supposedly identical perspectives (and vice versa). Finally, a list shows the headlines of further articles reporting on the event (bottom, not shown in Figure 1).
In each overview, further components can be enabled depending on the conjoint profile (cf. message and its perspective(s) on the topic.
Once !nished, click the button on the bottom of the page to continue the survey.
Information about the article
How articles that report on the topic portray (determined automatically) the
topic's main person Gov. Bill Lee:
Tennessee lawmakers pass fetal heartbeat abortion bill backed Section 5). PolSides tags (shown close to D in Figure 1) and/or MFAP tags are shown next to by governor each article headline, and indicate the political orientation of the article’s outlet and the article’s overall polaritWyarsehginargdtoinngTtehneneMssFeAe, rlaewspmeackteivreslyh.ave passed a bill backed by the state's Republican governor Bill Lee that would ban abortions after a 4.2. ArticlefeVtiaelwheartbeat is detected. Early Friday morning, the Tennessee Senate approved the bill, after the House had passed the legislation earlier.
syndrome. as gray), and disabled (no highlights are shown).
The article vieRwepshubolwicsanasncaorntitcrloel’sb otethxtcahnadmobpertsio.nally the following visual clues to communicate or altered depending on the conjoint profile.
exceptions to protect the life of the woman, but not for instances of rape or
bias information: (
The legislation e"ectively bans abortion after a fetal heartbeat is detected, as MFAP tags (with identical function to those in the overview). These clues are enabled, disabled, early as six weeks, through 24 weeks into a pregnancy. The bill would make
In-text polairnicteysth.iAgbholritgiohntssaaftiemr vtioabeilnitya,bwlehicuhs eisrsartoounidde2n4tiwfyeepkse,rasroena-ltraeragdeytiilnleggasleinntiment on the sentence-levelT.eWnneestseeset tehxceepetfecwtihveennethses owfotmhaen'fsolllifoewiisngin mdoandgeesr:. TshiengTleen-nceoslsoere(vbiilslually marking a person menptuioninsheussianbgortaionnepurotvriadelrcsowliothr,uip.et.o, 1g5rayyea,risfinthjaeil raensdpae$c1ti0v,0e00semnatxeimncuem mentions the person positiv!enlye. oItranlseogpartoihvieblitys),antwaob-ocrotiloonrw(huesriengthegrdeoecntorankndorwesdthceolwoormsafnorispsoeseiktiinvge and negative mentions, respaenctaibvoerltyio)n,three-coloofr t(hsaemcheilads'stwracoe-,coselox,r oarnda addiadgintoiosinsailnldyicsahtionwgiDnogwnneutral polarity because depending on!tlhinegirsuoitv. eTrhaellTpeonnlaersisteye rbeilglacrodmiensgatsheseMverFaAl.states have passed restrictive The polarity context bar aims to enable users to quickly contrast how the current article
The American Civil Liberties Union, the Center for Reproductive Rights, Planned and others portray the MFA. The 1D scatter plot depicted in Figure 2 places articles as circles Parenthood and several abortion providers challenged the bill in federal court, abortion laws with the hopes of forcing a broad court challenge to the 1973 landmark Supreme Court decision that legalized abortion nationwide. Judges have
blocked all of the laws.
To evaluate the efectiveness of our system in supporting non-expert users to become aware of
Senate Democratic Leader Je" Yarbro decried Republicans' move to pass the bill biases in newsacroouvnedramgiden,iwghet,ccoalnlidngucitttehde a"muossetrapstpuadlliyngcdoenpsairsttuirnegfroofmtwdeomcoocrnajtoicinnotremxsperiments. The study seeks toI'avenssewenerwshtiwle oserrveisnegairncthheqluegeissltaitounres.. "WAcchoartdianrgetoefeYcatribvreo, mtheeaCnaspittoolcboumildmingunicate biases to non-expert nweawssclocosends utomtherespwubhleicnavndietwheinSgenaanteohvaedrvsiueswpeonfdaednietswrsulteosp.iMco(rReQov1e)r,atnhed when reading time to examine the bill, propose amendments, or engage in questioning. respondents’ information is available freely (Section 1). a single news abriltliwcales n(RevQe2r)p?rinTtehdeofirnstaneyxppuebrliicmneontitce(Eno1r) afnoyccuasleensdoanrfoimrtpheroSveinnagte.tYhaerbgroeneral design of the overvieswaid(RthQa1t)t,hweSheinleattehheadseocnolyndplaenxnpeedrtiomaedndtre(sEs2l)egfioslcautiosensthoant wanass wreelartiendgtoboth RQ1 with improved visucoarloiznaatviirouns,soarnndecResQsa2ry. Atolplassusrtvheeybuddagteat.iDnecmluodcriantgs wqeureens'ttioginvennasiure#sciaenntd anonymized 5.1. Metho devoerlyoghuyman life is precious, and we have a responsibility to protect it." Lee Republican Gov. Bill Lee had proposed this new legislation, saying, "I believe that celebrated the bill's passage, calling it the "strongest pro-life law in our state's In both experiments, we used a conjoint design to “separately identify [. . .] component-specific history.” causal efects by randomly manipulating multiple attributes of alternatives simultaneously” Further articles
Tennessee advances 6-week abortion ban, lawsuit <led Tennessee passes abortion restriction bill Tennessee Legislature Passes Fetal Heartbeat Bill
Continue: Click here when you !nished reading. [20]. Respondents are asked to rate so-called “profiles,” which consist of multiple “attributes,” which are for example the overview, which topic it shows (or which article is shown in the article view), and if or which tags or in-text color highlights are shown. In conjoint design, these attributes are chosen randomly and independently of another for each respondent, which allows an estimation of the relative influence of each component on the bias-awareness (called Average Marginal Component Efects (AMCE)) [20].
We selected three news topics to ensure varying degrees of expected polarization as an indicator for biased coverage: gun control (high polarization), debt ceiling (high-mid), and Australian bushfires (low). We selected a single event for each topic. To ensure heterogeneity in content and writing styles, we manually retrieved a balanced selection of ten articles from left-, center, and right-wing US online outlets as self-identified by them.
We conducted both experiments on Amazon Mechanical Turk. Respondents had to be located in the US, have a history of successfully completed, high quality work, and were compensated 1-2$ depending on the study duration. In E1, we used data of 260 (of 308) respondents, which satisfied our quality measures, i.e., we discarded 48 respondents that, e.g., were unrealistically fast or answered test questions incorrectly. To keep cognitive load low, respondents were shown only a single topic in the overview, which was randomly drawn from the aforementioned selection. In E2, we used data of 98 (of 110) respondents. To increase cost eficiency, we only showed a selection of overview variants that exhibited positive trends in E1 (instead of fully randomly varying all attributes as in E1). Further, we showed respondents three tasks (resulting in 294 tasks in total), where each task consisted of a single overview and article view.
Our study consists of seven steps. A (
We found positive, significant efects on the change in bias-awareness when using the overview variants PolSides or MFAP (RQ1) in E2. Tags also increased bias-awareness significantly. Regarding the article view (RQ2), E2 yielded insignificant results.
Prior to E2, we focused our evaluation of E1 on qualitatively identifying flaws in the design of visualizations and the study, due to the lack of significant trends in E1. For example, 35% users experienced a lack of clarity and transparency, e.g., how the visualized information was derived. This weakness was exaggerated if respondents felt the shown information was incorrect (6% PolSides, 11% MFAP), e.g., an article that seemed negative from its headlines was labeled as ambivalent. Prior to E2, we addressed all the major lines of criticism, e.g., by adding brief explanations about all visual clues and in particular about the bias grouping (see B in Figure 1).
The goal of E2 was to test the set of overviews that had positive trends in E1 (to answer RQ1) and to test components in the article view (RQ2). E2 showed positive, significant efects of both bias-sensitive overviews, where the best overviews were PolSides (with PolSides tags) and MFAP (without tags). The AMCEs in Table 1 show that both overviews have very high and strongly significant efectiveness (PolSides = 7.83 and MFAP = 6.13, which are not significantly diferent to another albeit one being slightly higher [ 22]). Quantitative (by analyzing the efects on the individual questions composing the overall post-overview score) and qualitative analysis of both overviews suggests that MFAP reveals biases that are actually present in the news articles, whereas PolSides only facilitates the visibility of biases—a typical issue of prior approaches for automated bias detection [14]. By imitating the content analysis, our system yielded substantial frames as shown in Figure 1 whereas PolSides showed, e.g., the following headlines of rather “artificial” frames: “Trump Announces Deal On Debt Limit, Spending Caps” and “Trump, Congress Clinch Debt-Limit Deal After Tense Negotiations.” MFAP (random), an overview where articles were randomly assigned to one bias-group, yielded = 5.76, indicating that only pointing out possible biases already increased bias-awareness.
In MFAP, showing no tags yielded the highest efectiveness ( 6.13 compared to 5.87 when both tags were shown). This indicates that the MFAP design reveals “enough” bias information and further visual clues may yield too complex visualizations. None of the tags alone have significant efects when combined with the Plain version, indicating that a bias-group layout is necessary for bias-awareness.
In the article view, only showing the PolSides tags had a significant positive efect on bias awareness (2.45). There were no significant efects for the MFAP tags and polarity context bar. Analyzing respondents’ criticism in the post-study questions, we attribute this to two shortcomings. First, too few in-text highlights to have a consistent efect (21% of the article views had ≤ 5 highlights, 7% had none). When controlling for the number of highlights, they had a significant, positive efect on bias-awareness. Second, there was a strong influence of individual topics on the efectiveness, e.g., respondents reported the debt ceiling topic was “too complicated” or “boring” to follow. We plan to address this by conducting a study with more respondents and a wider range of topics, to ensure a better representation of the public discourses. Doing so will also strengthen the generalizability of the results and allow to investigate the efects of users’ demographic data on their bias-awareness and change thereof [23]. Due to the small sample size in E2, our current analysis was inconclusive regarding demographic efects.
Although we did not filter for a representative sample of the US population, the distributions of our samples are approximately similar to the distributions of the US population in key dimensions such as age and political education.1 However, we propose to verify the generalizability of the study’s findings to the entire US population or other countries using a larger respondent sample. Further, in E1 and E2 we assumed that MTurk workers are mostly non-expert news consumers. To verify this, we propose to explicitly ask for participants’ degree of media literacy.
We present the first system to automatically identify and then communicate person-targeting forms of bias in news articles reporting on policy events. Earlier, these biases could only be identified using content analyses, which–despite their efectiveness in capturing also subtle yet powerful biases–could only be conducted for few topics in the past due to their high cost, manual efort, and required expertise. In a large-scale user study, we employ a conjoint design to measure the efectiveness of visualizations and individual components. We find that our overviews significantly increase bias-awareness in respondents. In particular and in contrast to prior work, our bias-identification method seems to reveal biases that emerge from the content of news coverage and individual articles. In practical terms, our results suggest that the biases found and communicated by our method are actually present in the news articles, whereas the reviewed prior work only facilitates detection of biases, e.g., by distinguishing between left- and right-wing outlets. In sum, our exploratory work indicates the efectiveness of bias-sensitive news recommendation as a promising line of research for future work.
This work is funded by the WIN program of the Heidelberg Academy of Sciences and Humanities, ifnanced by the Ministry of Science, Research and the Arts of the State of Baden-Württemberg, Germany. The authors thank the anonymous reviewers for their valuable comments that helped to improve this paper.
1For statistics of the sample please refer to the online resources, see Section 1.