This paper describes the framework developed for the Debunker-Assistant, an application that allows users and newspapers to assess the trustworthiness of a news item starting from its headline, body of text and URL. The Debunker-Assistant adapts ideas from Natural Language Processing and Network Science to counter the spread of online misinformation. Its centerpiece is a set of four News Misinformation Indicators based on linguistically engineered features, models, network analysis metrics (Echo Efect, Alarm Bell, Sensationalism, and Reliability). In this short contribution, we describe the back-end structure on which the indicators are implemented.
news outlets’ credibility. For this reason, tackling misinformation is an open challenge faced by governments, private companies, and the scientific communities [1].
There are many proposed approaches, some based on AI methods, others on fact-checking by human experts, still others on a combination of the two [2, 3]. However, the fact that fake news detection algorithms are often owned by private social media companies, and additionally the adoption of “black-boxed” algorithms contribute to the lack of transparency in the fake news identification and filtering process.
Debunker-Assistant (D-A) is a back-end AI tool which supports the analysis and detection of online misinformation. The D-A tool takes as an input the link of a news article and returns its misinformation profile based on Natural Language Processing (NLP) and Network Analysis (NA) features. Such an approach is inspired by the survey provided by [4].
D-A works with Italian and is not bounded on a specific topic: it is designed as a general purpose tool that can extract relevant features for assessing the quality of information1.
Its main purposes are:
mation; 2. de-biasing the mechanisms to make trustworthy the internet; 3. showing insights about a certain context to aid the search and discovery of information.
and NA embedded in the D-A tool and how these features have been designed and developed. As represented in Figure 1, D-A allows users to search for news and compare them against a given set of 4 macro-indicators of misinformation: Echo Efect, Alarm Bell, Sensationalism, and Reliability. These indicators are designed on the basis of specific linguistic and network features, such as the absence of sources, non-authority of references, the presence of specific figures of speech or flames, and other stylistic characteristics.
available version can be found at https://github.com/AequaTech/
DebunkerAssistant
To operationalize our objectives, we focus on three political and immigration issues, annotated for irony; the main actions: Inform D-A can be freely used by citizens extended version of HaSpeeDe2 contains Italian tweets who want to check if a content they have read includes and news headlines about the integration of minority potential misinformation signals; Support D-A can also communities, annotated for hate speech, stereotypes, and be adopted by journalists who work in the field of debunk- irony. ing, empowering their activity; Teach teachers are able To create the models, we fine-tuned the base version of to use D-A for designing information literacy activities BERT for Italian3, optimizing basic hyperparameters like in their classrooms. learning rate and batch size for each phenomenon. To make the models smalls and easy to call at each request of the API, we reduced the number of trainable parameters, 2. News Misinformation Indicators adopting the LoRA technique implemented by HuggingFace [11]. If the aim of the training is to minimize the 2.1. Echo Efect loss function4 on the validation set, the evaluation of Echo Efect encodes the coupled aspects of origin and the models is focused on the analysis of f1-macro on the dissemination. There are news sources that occupy a test sets of news headlines of HaSpeeDe2 (0.722 for hate key role in the spread of misinformation [5]. This class speech and 0.749 for stereotype) and of tweets of IronITA aims to identify the most important domains in the news (0.769 for irony)5. ecosystem, and measure the impact and range of these The tool returns values in a range from 0 to 1, represources. We employ the network science framework, senting the probabilities that these phenomena (irony, modeling each domain as a node of a network model, hate speech and stereotypes) are present in the news. = (, , ), where is a set of nodes of domains, The presence of them could be an alarm bell on the seri it the set of link = (, ) that encodes the pairwise ousness and trustworthiness of the news [12], as well as interactions between two domains, and : × → Y of their malicious intentions [4]. is a function, defined for each pair of nodes , ∈ , that maps each link into a weight , that stands for the 2.3. Sensationalism total number of interactions from the domain to .
A well established method to assess the importance of Sensationalism includes three groups of features that are a node in a network is to measure the value of “centrality” indicative of the difusion of clickbait, especially while that a node has with respect to all the other nodes. There observing online headlines: informal style, syntactic comare many possible definitions of importance and so many plexity, emotion profile. Inspired by [ 13], who studied the centrality measures. In the context of news, we adopted linguistic and typological features commonly associated the hyperlink-induced topic search [6] algorithm to infer with clickbait in online news headlines, we designed the the “origin” and “destination” of the echo efect, i.e., the following features. set of nodes responsible for the amplification of signals traversing the network and the set of nodes in which the signal resonates the most. While the betweenness centrality [7] measures the extent to which a node can reach other locations in the network2. 2.2. Alarm Bell
pragmatic implications like flaming and ironic language in the news. Specifically, it analyzes whether the headline and news content contain these elements, and outputs an average score of the various probabilities.
To obtain these values, we designed specific classifiers
to account for the presence in the text of irony, hate
speech and stereotypes. For training our models, we
used two well-known benchmark datasets: IronITA [8]
and the extended version of the HaSpeeDe2 [
4The function used in our experiments is the Binary Cross Entropy with diferent weight for each class, considering the constant imbalance of the positive class in all the datasets.
5The models are available in https://github.com/AequaTech/ DebunkerAssistant 2. syntactic_complexity: avg length - A metric that computes the textual length of the news headline and compares it with the average textual length of a news headline stored in a proprietary database. The value is normalized between 0 and 1. shortest/longest - A metric that shows the comparison of the length of the headline and collocates it in one of the four quartiles of the database with all the other news headlines previously analyzed. be linked than the nodes in diferent communities. We employed the stochastic block model [22], a generative model that regards in a formal context the actual presence of a specific not random map into which the network can be partitioned. Once having the diferent communities, we characterized them by assigning to each a label, white or black, counting the majority of their nodes (web pages) belonging to either the white or the black list.
tage of the “whois” metadata attached to each
URL domain, which is the country of registration,
creation date, and expiration date. We calculated
the amount of time (days) that a specific domain
maintains the same characteristics.
3. affective_profile:
overall emotion - Inspired by the study
Vosoughi et al. [14] about the presence of specific
emotions in false stories, this feature returns the
averaged score about the presence of emotions
in the news. This value is based on the
identification of the eight primary emotions—anger, fear,
sadness, disgust, surprise, anticipation, trust, and 3. Conclusions
joy— following the psycho-evolutionary
taxonomy proposed by [15] through the multilingual D-A is a tool developed in support of detecting online
NRC lexicon of [
1. neighborhood: We employ the network analysis framework to evaluate the neighborhood of a web page. We retain the same model described in Section 2.1 to detect special agglomerations of nodes, or communities (locally dense connected sub-graphs), such that the nodes that belong to the same community have a higher probability to
set of characteristics about the article and a set of information about the domain hosting the article. Thus, the output generated does not consist of a binary classification of the truthfulness of an online newspaper article.
Nonetheless, we are aware of the ethical issues surrounding the characterization and evaluation of online news.
First of all, the fallibility of NLP models must be taken into evalita 2020 hate speech detection task, in:
Proceedaccount, secondly some aspects concerning the world ings of Seventh Evaluation Campaign of Natural
of information can have shades of subjectivity and be Language Processing and Speech Tools for Italian.
sensitive especially for some users. Final Workshop (EVALITA 2020), Online. CEUR.
org, 2020.
[
true and false news online, Science 359 (2018) [12] V. L. Rubin, N. Conroy, Y. Chen, S. Cornwell, Fake 1146–1151. URL: https://www.science.org/doi/abs/ news or truth? using satirical cues to detect po10.1126/science.aap9559. doi:10.1126/science. tentially misleading news, in: Proceedings of the aap9559. second workshop on computational approaches to [2] P. Nakov, D. P. A. Corney, M. Hasanain, F. Alam, deception detection, 2016, pp. 7–17.
T. Elsayed, A. Barr’on-Cedeno, P. Papotti, S. Shaar, [13] R. Kemm, The linguistic and typological features G. D. S. Martino, Automated fact-checking for as- of clickbait in youtube video titles, Social Commusisting human fact-checkers, in: International Joint nication 8 (2022) 66–80.
Conference on Artificial Intelligence, 2021. [14] S. Vosoughi, D. Roy, S. Aral, The spread of true and [3] S. I. Manzoor, J. Singla, Nikita, Fake news detec- false news online, Science 359 (2018) 1146–1151. tion using machine learning approaches: A sys- doi:10.1126/science.aap9559. tematic review, in: 2019 3rd International Con- [15] R. Plutchik, A general psychoevolutionary theory ference on Trends in Electronics and Informatics of emotion, in: Theories of emotion, Elsevier, 1980, (ICOEI), 2019, pp. 230–234. doi:10.1109/ICOEI. pp. 3–33.
2019.8862770. [
ACM computing surveys (CSUR) 31 (1999) 5. 3331285. doi:10.1145/3331184.3331285. [7] L. C. Freeman, D. Roeder, R. R. Mulholland, Cen- [18] V. Basile, M. Nissim, Sentiment analysis on Italian trality in social networks: Ii. experimental results, tweets, in: Proceedings of the 4th Workshop on Social networks 2 (1979) 119–141. Computational Approaches to Subjectivity, Senti[8] A. T. Cignarella, S. Frenda, V. Basile, C. Bosco, ment and Social Media Analysis, 2013, pp. 100–107.
V. Patti, P. Rosso, Overview of the EVALITA 2018 [19] R. Baly, G. Karadzhov, D. Alexandrov, J. Glass, task on irony detection in Italian tweets (IronITA), P. Nakov, Predicting factuality of reporting and bias in: Proceedings of the Sixth Evaluation Campaign of news media sources, in: Proceedings of the 2018 of Natural Language Processing and Speech Tools Conference on Empirical Methods in Natural Lanfor Italian (EVALITA 2018) co-located with the Fifth guage Processing, Association for Computational CLiC-it, volume 2263, 2018, pp. 1–6. Linguistics, Brussels, Belgium, 2018, pp. 3528–3539. [9] M. Sanguinetti, G. Comandini, E. Di Nuovo, URL: https://aclanthology.org/D18-1389. doi:10.
S. Frenda, M. Stranisci, C. Bosco, T. Caselli, V. Patti, 18653/v1/D18-1389.
I. Russo, Haspeede 2@ evalita2020: Overview of the [20] B. Ghanem, S. P. Ponzetto, P. Rosso, F. Rangel,
Here are the details of Section 2.1. The hyperlink-induced topic search algorithm (also known as HITS or hubs and authorities) is defined for directed networks, and computes the authority centrality and the hub centrality, which quantify vertices’ prominence in the two roles, as a “receiver” or as a “provider” of information. The hub vector ⃗ℎ = (ℎ1,, . . . , ℎ| |,)⊺ and the authority vector ⃗ = (1,, . . . , | |,)⊺ in ∈ of = (, , ) are defined by the limit of the following set of iterations: (1) (2) (3) and ⃗ℎ( + 1) = ()⃗( + 1) ⃗( + 1) = ()⊺⃗ℎ(), where () and () are normalization factors to make the sums of all elements become unity, i.e., ∑︀|= |1 ℎ,(+ 1) = 1 and ∑︀|= |1 ,( + 1) = 1. The initial values of the scores are ℎ,(0) = |1 | and ,(0) = |1 | for all ∈ .
The betweenness centrality measures the extent to which a node lies on paths between other vertices. We define the betweenness centrality of a node ∈ at time ∈ as (, ) = ,∈ ̸≠= ∑︁ ,() , , where , is the total number of shortest paths from node to node at time , and ,() is the number of such paths passing through node .