Research has repeatedly demonstrated that fake news tend to appeal to the emotions, but it is less common to consider the presence of emotions in relation to the inverted pyramid, a key aspect of journalistic writing. In this work, we adopt an existing neural model for fake news detection, FakeFlow, one of the few approaches to consider the salience of emotions with respect to the structure of news articles, and adapt it for Spanish. We conduct our experiments on the Spanish Fake News Corpus, introduced at the Fake News Detection in Spanish shared task (FakeDeS), with the goal of gaining a better understanding of the characteristics underlying such texts. In our analyses, we show that both the distribution of afective features and the attention mechanism of the model validate the importance of considering the inverted pyramid structure for detecting fake news.
performing approach in the competition achieved an
In an increasingly connected world, research on detect- 1 of 0.76, leaving plenty of room (and need) for
ing fake news is more necessary than ever [
SLEaPnLgNua-2g0e2P4r:o4c0etshsiCnogn.fVearellnacdeoolifdt,hSepaSpina.n2i4sh-2S7oSceiepttye mfobreNra2t0u2r4a. l In text classification tasks, the information structure
$ btogni1@upvnet.upv.es (B. Togni); mcoll@prhlt.upv.es of news articles is not often accounted for, even though
(M. Coll Ardanuy); berta.chulvi@symanto.com (B. Chulvi); some research exists in this direction. In fact, the
bestprosso@dsic.upv.es (P. Rosso) performing approach at the FakeDeS 2021 competition
0000-0001-8455-7196 (M. Coll Ardanuy); 0000-0003-1169-0978 [
Research has repeatedly demonstrated that fake news
tend to appeal to the emotions [
One such approach is FakeFlow [15], a neural model which was developed with the specific goal of learning the flow of semantic and afective information in news articles, and using this information for detecting fake news.
In this paper, we adapt FakeFlow for Spanish and use its inherently interpretable capabilities for analysing the distinction between fake and true news, focusing on the SFNC dataset. Unlike in English, the FakeFlow-based classifier for Spanish underperforms when compared with a RoBERTa-based classifier, but nevertheless ofers useful insights on the data. We extensively analyse our results both in relation to the afective information and the structure of the news articles, and show why a generalisable solution to fake news detection is hard to achieve.
The rest of the paper is structured as follows: we describe the FakeFlow approach and how we have adapted it for Spanish in Section 2; we describe the classification experiments in Section 3; we analyse and discuss our findings in Section 4; and provide a conclusion and directions for further research in Section 5.2 over the two classes (fake and true) is returned.3
the openly available word2vec [16] embeddings trained on the Spanish Billion Words Corpus4 [17]. The afective 2. Approach information branch requires that each segment is represented as a vector of lexical features. These vectors are FakeFlow [15] was developed with the specific goal built from term frequency representations (based on lexof learning the ‘flow’—i.e. the sequence of information ical resources), which are then normalised by the length salience—of semantic and afective features in news ar- of the article. In order to adapt FakeFlow for Spanish, we ticles. As shown in Figure 1, FakeFlow consists of two looked for lexical resources in Spanish to capture the afcommunicating branches: on the one hand, the topic fective dimension of the news articles. We have extracted information branch is built upon word embedding rep- the following features for each news segment: resentations of the news segments, and is responsible for capturing the semantics of the texts; on the other hand, • Emotion features: As in the original paper, we the afective information branch is built upon vectors have used the NRC Emotion Lexicon [18, 19],5 of lexical features which represent the afective aspect which associates words with eight emotions: of the texts. In order to model the flow of information, anger, fear, anticipation, trust, surprise, sadness, each news article is first split in N segments. The neural joy, and disgust. After processing the lexicon, the architecture of the model is designed to capture the joint Spanish lexicon consists of 11,326 words.6 Each interaction between both the topic and afective information in each segment, including a self-attention layer which was added with the purpose of highlighting the importance of a segment in relation to the neighbouring segments in the news article. Finally, the two branches are combined and, for a given text as input, a probability emotion corresponds to a feature in our model (8 the association between words and features, and only features). kept nouns, verbs, adjectives, and adverbs. • Sentiment features: The same NRC Emotion
Lexicon also contains associations between words 3. Experiments (10,898 after processing) and sentiment polarity: positive and negative. Each is assigned a feature We describe the dataset we have used in our investigation in the model (2 features). in Section 3.1 and describe the classification experiments • Hurtful language feature: The original Fake- and results in Section 3.2.
Flow paper used the Moral Foundations
Dictionary [
An examination of the content showed that this 3.1. Dataset
resource is strongly rooted in American culture. We have conducted our experiments on the Spanish Fake
Given that some of the categories in the dictio- News Corpus (SFNC)8 [
7https://github.com/valeriobasile/hurtlex. We have used only words categorised as: negative stereotypes and ethnic slurs, physical disabilities and diversity, cognitive disabilities and diversity, moral and behavioural defects, words related to social and economic disadvantage, words related to prostitution, words related to homosexuality, words with potential negative connotations, derogatory words, felonies and words related to crime and immoral behaviour, and words related to the seven deadly sins of the Christian tradition.
Flow was originally developed, we eventually used the
same hyperparameters than for English:9 while we experimented with diferent choices of hyperparameters, we did not consistently obtain significant improvements. We show the results of adapting FakeFlow for Spanish in Table 2. For comparison, we provide the most common class baseline (in this case, ‘true’) and a RoBERTa-based baseline.10 While the original FakeFlow in English surpassed transformer-based approaches [15], in our case we see that the RoBERTa-based classifier is by far a better performing approach, with a diference of 8 points in macro F1-score with respect to the FakeFlow approach. We also compare the FakeFlow model with two simpliifed versions of the approach, using only a subset of the features: (1) only emotions uses only the eight emotions as features; (2) only negative considers only negative features: ‘anger’, ‘disgust’, ‘fear’, ‘sadness’, ‘negative’ and ‘hurtful’. Neither of the simplified versions of FakeFlow yield a better performance, in both cases providing a lower 1 for the true class and a higher score for the fake class. In the following sections, we show how a careful inspection of the dataset, using the features and the outputs of the FakeFlow model, can shed some new light to help us interpret these results.
10We have used an existing model available on the
HuggingFace hub: Narrativaai/fake-news-detection-spanish,
which was fine-tuned for text classification on the training and
development sets of the SFNC dataset. It is based on the
PlanTL-GOB-ES/roberta-large-bne model, pre-trained on the
largest existing corpus in Spanish [
In this section, we analyse the diferences observed between the fake and true subsets of the SFNC dataset in terms of their distributions of the features used by the FakeFlow classifier (in Section 4.1) and the attention weights produced by the classifier (in Section 4.2), and discuss our findings.
An analysis of the distribution of the features on the full
dataset reveals that fake news have on average higher
emotional content than true news, as is also reported
in previous works [11, 15]. In Figure 2, we show the
changing salience of the sixteen features over the article
segments. First, we see that, on average (i.e., the
dotted lines), fake news have higher values than true news,
and that, both in true and fake news, emotion values are
higher at the beginning of the article. This pattern is
particularly noticeable on some features, such as ‘anger’,
‘anticipation’, and ‘fear’. We observe that ‘negative’ is
more prominent in fake news, whereas the signal is less
clear for ‘positive’, in the same vein as previous work
indicates [11]. The ‘hurtful’ feature is consistently higher
in fake news, and so is ‘hyperbolic’, even though less
prominently. Finally, the four afective-semantic features
are higher in true news, except in the rfist segment. In
the case of ‘concreteness’ and ‘imageability’, this
information could be used to corroborate findings from previous
research, according to which fake news tend to require a
smaller cognitive efort from the reader in order to
process the content [11]. Further research using additional
lexical resources, such as [
A breakdown of the dataset into topics reveals difer- that the model attended more at the beginning of the arent distributions of features. In Figure 3, we show the ticles, concluding that in English most of the information distribution of the eight emotions (plus ‘hurtful’) on three useful for discriminating between fake and true news topics: politics, entertainment, and Covid-19.11 Here, we is presented at the beginning of the news articles. As distinguish between the training set and the test set, to described in Section 2, the FakeFlow architecture has a account both for the topic imbalance and also the difer- self-attention mechanism which highlights the imporent temporal coverage of both subsets. First, focusing tance of a segment in relation to the other segments in on the training set (i.e., the figures on the left, built from the news article. We therefore extracted the matrices of news articles collected during the first half of 2018), we attention weights from the test set and averaged them. observe that ‘hurtful’ stands out both on true and fake In Figure 4, we show the matrices of averaged attention news on entertainment, whereas the other emotions are, weights of the texts that have been correctly classified with the exception of ‘surprise’ and ‘joy’ on the fake as true and those correctly classified as fake. We can see subset, less prominent than on politics. In general, the that, both in fake and true news, the model attends more language of political news is more emotional, the main at the beginning of the article—as in English, according diference between true and fake being higher ‘hurtful’, to [15]—whereas the middle part is less attended, and the ‘anger’, and ‘joy’ values in the latter. last part is the least attended. This suggests that the first
On the test set (i.e., the figures on the right, built part of the articles is on average more useful for discrimfrom news articles collected between November 2020 inating fake from true news, but further investigation on and March 2021), we see that articles on Covid-19 have a the role of inter-segment attention with respect to the very diferent distribution of features, especially patent classification decision is needed. Finally, our version of in the case of ‘fear’ and ‘sadness’, which stand out both FakeFlow allows the user to inspect the diferent articles for true and fake news. Fake news on Covid-19 have con- in the dataset, and visualise how much the classifier atsistently higher values than true news for all emotions tended to each of the segments in the article, as illustrated and for hurtful language. Finally, it is interesting to note in Figure 5. the evolution of the emotional content in political news In order to validate whether these findings are specific from 2018 (i.e., the training set) to 2020–2021 (i.e., the of the FakeFlow approach or more generic, we have done test set): we see true news becoming more similar to fake a text-based ablation study using the RoBERTa classifier. news, particularly with respect to ‘anger’ and ‘hurtful’. In Table 3, we observe how the performance of the clasGiven that almost three convulsed years span between sifier changes depending on the input that is provided: the articles in both datasets, it remains to be investigated ‘first part’ contains only segments 1 to 3 (i.e., the first part whether this diference is specific of this dataset or it is of the articles), ‘middle part’ includes segments 4 to 7, the product of the larger question of whether, in recent and ‘last part’ includes segments 8 to 10. For comparison, years, the style of mainstream news has become more we provide the full text (excluding the headline, hence similar to that of fake news, perhaps as a strategy to the diference with respect to Table 2) and the full text compete with the appeal of fake news. removing either the middle or the last part. It is important to note the drop of 1 of the true class for individual parts, whereas the 1 of the fake class is less (or not) afected. The results show that the middle and last parts are clearly less informative, and we show 11Topics were selected based on their frequency in the dataset: that removing the last part does not afect the classifica‘mpoolsittipcso’piuslathretompiocstinptohpeultararitnoipnigc aonvderdaellv,e‘leonptmeretnaitnsmetesn(tb’eissidthees tion (and even improves the 1 of the fake class). ‘politics’), and ‘Covid-19’ is the most popular topic in the test set. This is an interesting finding, not only from a scientific perspective (as it aligns with the well-known inverted pyramid approach to writing news articles), but also for practical application.12
12Language models usually truncate long input sequences to a maximum length. A better understanding of which parts of the article are more likely to be informative may be a safe strategy to reduce time and computational costs, especially when processing large amounts of text, since the quadratic complexity of transformers means that, the longer the text, the longer the time to process it. (a) True news.
(b) Fake news.
While transformer-based models are the state-of-the-art
approaches for text classification, it is notorious that they
sufer from interpretability issues [
constructive reviews. We would also like to thank Damir Korenc˘ić and Ivan Grubis˘ić for their feedback.
This work was funded by the research project