English. Different events and their reception in different reader communities may give rise to controversy. We propose a distant supervised entropy-based model that uses Facebook reactions as proxies for predicting news controversy. We prove the validity of this approach by running within- and across-source experiments, where different news sources are conceived to approximately correspond to different reader communities. Contextually, we also present and share an automatically generated corpus for controversy prediction in Italian.
Italiano. Diversi tipi di eventi e la loro percezione in diverse comunita` di utenti/lettori possono dare vita a controversie. In questo lavoro proponiamo un modello basato su entropia e sviluppato secondo il paradigma della “distant supervision” per predire controversie sulle notizie usando le reazioni di Facebook come “proxy”. La validita` dell’approccio e` dimostrata attraverso una serie di esperimenti usando dati provenienti dalla stessa fonte o da fonti diverse. Contestualmente, presentiamo anche un corpus generato automaticamente per la previsione delle controversie in italiano.
The explosion of social media (e.g. Facebook, Twitter, Disqus, Reddit, Wikipedia, among others) and the increased interactions with readers-users that traditional newspapers embraced, have transformed the Web in a huge agora, where news are shared, opinions are exchanged, and debates arise. On many topics, such as climate change, abortion, vaccination, among others, people strongly disagree. Following the work by Timmermans et al. (2017), we call controversies situations where, even after lengthy interactions, opinions of the involved participants tend to remain unchanged and become more and more polarized towards extreme values.
Modeling and understanding controversies may
be useful in many situations. Journalists and news
agencies may pay additional attention in the
framing of a certain news, government officials and
policy makers may be more aware of the issues
involved in specific laws, social media managers
might be more careful, i.e. monitor
controversial content, in order to avoid the spreading of
hate speech, and the general public may benefit
as well thanks to a reduction of the “filter bubble”
effect
Recently, computational approaches on
controversy detection have been developed with
varying degrees of success
The main contribution of this work is two-fold: i.) we propose a distant supervised entropy-based 1.) In volo sul Piemonte con biplano anni ’30 2.) Medico anti vaccini radiato 3.) Piacenza, abbattuto il cinghiale Agostino model to predict controversial news; and ii.) we present and share an automatically created corpus to train and test models for controversy detection. At this stage of development, we focused only on Italian, although the methods are completely language independent and can be reproduced for any language for which news are available on Facebook. The remainder of the paper is structured as follows: Section 2 illustrates the methods used to collect the data and develop the entropy-based model. Section 3 reports on the experiments and results both in a within- and acrosssource setting. Finally, Section 4 draws conclusions and outlines future research. Data and code are made available at https://anbasile. github.io/predictingcontroversy/. 2
We used the Facebook Graph API1 to download news headlines (including the description and body fields) from four major Italian newspapers. Of these, two are slightly politically biased (Corriere della Sera and La Repubblica, both centre/centre-left), two openly biased ones (Il Manifesto, left-wing, and il Giornale, right-wing), and one news agency (ANSA).
Together with each news, we also downloaded
all users’ reactions.2 Facebook reactions can be
used as a proxy for annotations
2Since February 2016, Facebook users can react to a post not only with a like but by choosing from a set of 5 different emotions: ANGRY, LIKE, HAHA, WOW, SAD, LOVE. sider the data in Table 1. Each sample is the text of a Facebook post, for which we report the reaction breakdown (including LIKE), and its overall entropy based on reaction counts. Users expressing different reactions suggest that a text is likely to be controversial as it is shown by the high values of the entropy, as illustrated in examples 2.) and 3.) vs. example 1.).
For each source, namely the newspaper pages mentioned at the beginning of this section, we downloaded a collection of posts which appeared between mid-April and early July 2017. Posts with less than 30 reactions in total were discarded. For each post, we collected: i.) the link to the full article on the source’s website (a large majority of the posts include this); ii.) an excerpt of the article (the variable text); iii.) additional texts commenting the article, when available (the variable descriptor); iv.) the full list of users’ reactions. Finally, for a portion of the posts (1024 out of 3595, i.e. 28,48%; column “# body” in Table 2) we downloaded the entire text of the article (the variable body).3. Table 2 provides an overview of the data collected, including, for each source, the number of Facebook posts, the number of tokens, the number of posts for which the full article was retrieved, the token-post ratio, i.e. the number of tokens per post, and, finally, the average entropy.
To further verify the soundness of using entropy as an indicator of controversy, we inspected the top-10 and bottom-10 news in the full dataset 3The full text of the article is not always available or accessible. Furthermore, there is a monthly limit to the data that can be downloaded. We made sure that the final dataset we used contained, for each source, the same number of posts for which the full body could be downloaded. This constraint did not apply to ANSA sorted by entropy (high values on top, high controversy) and manually assigned them to a topic. Table 3 illustrates the results for the top 5 and bottom 5 posts, in terms of entropy score. In addition to identifying a different distribution of topics according to degrees of controversy, we also observed that in some cases, the entities and the specific event mentions interact to generate controversy. For instance, in the case of the “25th April” topic4, the controversial news involves a political actor (i.e. ANPI, the National Association of Italian Partisans), and divisions on the celebration of this day, while the non-controversial news reports on museums being open on that day. The entropy score appears to capture this distinction. 3
We use the ANSA dataset to develop our model. The rationale behind this is that, being ANSA a news agency, the texts should be more objective and the controversy should depend on the event itself rather than by its framing in a specific, potentially biased, community. We treat this task as a regression problem, and use mean squared error (MSE) to measure the performance of our system. As baseline, we use a dummy regressor which always predicts the mean entropy of the train dataset: considering that the values range between 0 and 2.9, with a standard deviation of 0.4, a system that always predicts the mean entropy is already performing reasonably well. Furthermore, this is in line with the average entropy values of each dataset, ranging from 0.6195 (Table 2, Il Manifesto) up to 1.1266 (Table 2, Il Giornale).
4April 25th is a national holiday in Italy to celebrate the end of World War II.
Settings We use two main settings. Firstly, the
data for training and testing the model originates
from the same Facebook page, and we use
crossvalidation. Secondly, we train and test across
pages, so as to investigate the model’s
portability across potentially different communities. This
second setting can shed light on the issue of
perspective bias, as controversy around a specific
topic or entity could exist in one domain (or, in this
case, in one community as proxied by Facebook
pages) and not in another one. In both settings, we
run our best model, developed as described below.
Features For predicting the entropy of the
reactions to a given text, we built a system using a
sparse feature representation and an SVM
regressor, with the scikit-learn LinearSVR
implementation
As sentiment might contribute to controversy
prediction
The best model shows an improvement of 0.094 MSE with respect to the baseline when extending the variable text with descriptor and body. The use of the variable text alone still beats the baseline, but obtains a lower score than the models which include both the descriptor and the body variables. The extensions with the polarity scores from Sentix decrease the model performances (though still outperforming the baselines). We believe that this behaviour is mainly due to noise in the resource itself and calls for better and more context-oriented sentiment lexicons in Italian. Table 5 summarises the features of the best model, which is based on a combination of the three text variables only: text, descriptor, and body (whenever available), represented as word and character n-grams, ignoring the polarity vectors. This model was used on the reminder of the datasets.
Results on the test set Table 6 illustrates crossvalidated results for the newspaper datasets. For comparison and completeness, we report also the results of the cross-validation on the full test set, with and without the extension of the data with ANSA.
With the exception of Il Giornale, our model always beats the baseline, confirming the validity of the designed approach. Extending the newspaper dataset with the data from ANSA, we can observe a reinforcement of the predicting power of the model, with a range between 0.04 to 0.1 points with respect to the corresponding baselines. The positive effect on Il Giornale dataset can be due to an extension of the number of tokens, since Il Giornale is the dataset with the lowest token-post ration (8,47 tokens per post), which clearly affects our model.
Cross-source results in Table 7 are less clearcut. In these experiments, it clearly emerges that our model works in the large majority of cases, although with no big gains over the baselines. All datasets fail to beat the baseline when predicting controversy on Il Giornale and, on the contrary, training on Il Giornale only fails to beat the baseline when testing on La Repubblica. This suggests that either there must be a difference in the wording used by Il Giornale with respect to the other datasets, or that the controversy is affected by perspective bias associated to different communities.
On the other hand, slightly politically oriented newspapers (La Repubblica and Il Corriere della Sera) and the ANSA news agency tend to have a homogeneous behavior, being able to correctly predict controversy in highly politically oriented news (see results for Il Manifesto in Table 7). As a matter of fact, the more the post/token ratio is similar between different sources, the better the model works in predicting controversy. For instance, Il Corriere della Sera and Il Manifesto have a very similar post/token ratio (40,08 and 48,5, respectively) and not surprisingly both cross-source experiments beat the baseline. This paper presents a simple regression model to predict the entropy of a post’s reactions based on the Facebook reaction feature. We take this measure as a proxy to predict the controversy of news, where the higher the entropy (indicated by highly mixed reactions), the bigger the controversy. We run experiments both within and across communities, exemplified by the Facebook pages of specific newspapers. As a by-product, we have also automatically generated a first reference corpus for controversy prediction in Italian.
The results are promising, given that our model beats the baseline in almost all cases in crossvalidation of same source data (see Table 6), and in the large majority of cases when applied crosssources (see Table 7). At this stage of development, we observed that coarse-grained sentiment values are not useful, although this may depend on the quality of the lexicon employed. Test and training on openly biased datasets (e.g. Il GiornaleT RAI N - Il ManifestoT E ST , and vice-versa) results in the lowest entropy, suggesting perspective bias in the different community.
The approach we have developed is based on discrete linguistically motivated features. This has an impact in the learned model as it is not able to generalise enough when dealing with lowfrequency features and unseen data in the test set. To alleviate this issue, we are planning to model the post representations by using word embeddings.
We are planning to expand the model to account for perspective bias in different communities. News from different sources may be aggregated per event type, for example via the EventRegistry API5, allowing to explore entropy (and polarisation of reactions) on exactly the same event instance. A first step in this direction would be to detect and match Named Entities to approximately identify similar events. At the reactionlevel, the obvious next step is to explore and experiment with clusters of reactions (for instance, positive (LIKE, LOVE, AHAH), negative (ANGRY, SAD), or ambiguous (WOW)), instead of treating them all as single and distinct indicators.
Another follow-up is to extend this work to
other social media data, such as Twitter. Twitter
does not allow for nuances in reactions in the same
way that Facebook does, as only one kind of “like”
is provided. However, the substantial use of
hashtags and emojis might offer alternative proxies to
capture a variety of reactions. There is plenty of
work on the usefulness of leveraging hashtags as
reaction proxies both at a coarse and finer level
One of the authors wants to thank the SpinozaNWO Project “Understanding Language by Machines” subtrack 3 for making this work possible. 5http://eventregistry.org