English. Propaganda is a rhetorical technique to present opinions with the deliberate goal of influencing the opinions and the actions of other (groups of) individuals for predetermined misleading ends. The employment of such manipulation techniques in politics and news articles, as well as its subsequent spread on social networks, may lead to threatening consequences for the society and its more vulnerable members. In this paper, we present PROTECT (PROpaganda Text dEteCTion), a new system to automatically detect propagandist messages and classify them along with the propaganda techniques employed. PROTECT is designed as a full pipeline to firstly detect propaganda text snippets from the input text, and then classify the technique of propaganda, taking advantage of semantic and argumentation features. A video demo of the PROTECT system is also provided to show its main functionalities.
Italiano. La propaganda e` una tecnica retorica per presentare determinate opinioni con l’obiettivo deliberato di influenzare le opinioni e le azioni di altri (gruppi di) individui per fini predeterminati e tendenzialmente fuorvianti. L’impiego di tale tecnica di manipolazione in politica e nella stampa, cos`ı come la sua diffusione sulle reti sociali, puo` portare a conseguenze disastrose per la societa` e per i suoi membri piu` vulnerabili. In questo articolo presentiamo PROTECT (PROpaganda Text Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). dEteCTion), un nuovo sistema per identiifcare automaticamente i messaggi propagandistici e classificarli rispetto alle tecniche di propaganda utilizzate. PROTECT e` un sistema progettato come una pipeline completa per rilevare in primo luogo i frammenti di testo propagandistici dato il testo proposto, e successivamente classiifcare tali frammenti secondo la tecnica di propaganda usata, sfruttando le caratteristiche semantiche e argomentative del testo. Questo articolo presenta anche un video dimostrativo del sistema PROTECT per mostrare le principali funzionalita` fornite all’utente.
Propaganda represents an effective but often
misleading communication strategy which is
employed to promote a certain viewpoint, for
instance in the political context
To achieve this ambitious goal, we present in this paper a new tool called PROTECT (PROpaganda Text dEteCTion) to automatically identify and classify propaganda in texts. In the current version, only English text is processed. This tool has been designed with an easy-to-access user interface and a web-service API to ensure a wide public use of PROTECT online. To the best of our knowledge, PROTECT is the first online tool for propagandist text identification and classification with an interface allowing the user to submit his/her own text to be analysed.1
PROTECT presents two main functionalities: i)
the automatic propaganda detection and
classification service, which allows the user to paste or
upload a text and returns the text where the
propagandist text snippets are highlighted in different colors
depending on the propaganda technique which is
employed, and ii) the propaganda word clouds, to
show in a easy to catch visualisation the identified
propagandist text snippets. PROTECT is deployed
as a web-service API, allowing users to download
the output (the text annotated with the identified
propaganda technique) as a json file. The
PROTECT tool relies on a pipeline architecture to first
detect the propaganda text snippets, and second to
classify the propaganda text snippets with respect
to a specific propaganda technique. We cast this
task as a sentence-span classification problem and
we address it relying on a transformer
architecture. Results reach SoTA systems performances
on the tasks of propaganda detection and
classiifcation (for a comparison with SoTA algorithms,
we refer to
The paper is structured as follows: first, Section 2 discusses the state of the art in propaganda detection and classification and compares our contribution to the literature. Then Section 3 describes the pipeline for the detection and classification of propaganda text snippets as well as the data sets used for the evaluation and the obtained results. Section 4 describes the functionalities of the web interface, followed by the Conclusions. 2
In the last years, there has been an increasing
interest in investigating methods for textual
propaganda detection and classification. Among
them,
The most recent approaches for propaganda
detection are based on language models that mostly
involve transformer-based architectures. The
approach that performed best on the NLP4IF’19
sentence-level classification task relies on the
BERT architecture with hyperparameters
tuning without activation function
As most of the above mentioned systems, also
PROTECT relies on language model architectures
for the detection and classification of propaganda
messages, empowering them with a rich set of
features we identified as pivotal in propagandist
text from computational social science literature
PROTECT addresses the task of propaganda technique detection and classification at fragmentlevel, meaning that both the spans and the type of propaganda technique are identified and highlighted in the input sentences. In the following, we describe the datasets used to train and test PROTECT, and the approach implemented in the system to address the task. 3.1
To evaluate the approach on which PROTECT
relies, we use two standard benchmarks for
Propaganda Detection and Classification, namely
the NLP4IF’19
Those classes are not uniformly distributed in the data sets. Loaded-Language and Name-Calling Labeling are the classes with the 2https://propaganda.qcri.org/nlp4if-s hared-task/
3https://propaganda.qcri.org/semeval2 020-task11/ higher number of instances (representing respectively 32% and 15% of the propagandist messages on all above-mentioned datasets). The classes with the lower number of instances are Whataboutism, Red-Herring, Bandwagon, Straw-Men, respectively occurring in 1%, 0.87%, 0.29%, 0.23% in NLP4IF’19 datasets. In SemEval’20T11 such labels where merged, and the classes Whataboutism StrawMen Red-Herring, Bandwagon respectively represent 1.33% and 1.29% of the propagandist messages. 3.2
Given a textual document or a paragraph as input, the system performs two steps. First, it performs a binary classification at token level, to label a token as propagandist or not. Then, it classifies propagandist tokens according to the 14 propaganda categories from SemEval task (T11).
For instance, given the following example “Manchin says Democrats acted like babies at the SOTU (video) Personal Liberty Poll Exercise your right to vote.” the snippets “babies” is first classiifed as propaganda (step 1), and then more specifically as an instance of the Name-Calling Labeling propaganda technique (step 2).
Step 1: Propaganda Snippet Detection. To train PROTECT, we merge the training, development and test sets from NLP4IF, and the training set from Semeval’20 T11. The development set from Semeval’20 T11 is instead used to evaluate the system performances.4 In the preprocessing phase, each sentence is tokenized and tagged with a label per token according to the IOB format.
For the binary classification, we adopt
Pretrained Language Model (PLM) based on BERT
(bert-base-uncased model)
5https://scikit-learn.org/stable/modu les/generated/sklearn.metrics.precisio n recall fscore support.html
Appeal to Authority Appeal to fear-prejudice Bandwagon,Reductio ad hit. Black-White-Fallacy Casual-Oversimplification Doubt Exaggeration,Minimisation Flag-Waving Loaded Language Name Calling,Labeling Repetition Slogans Thought-terminating Cliches Whatab.,Straw Men,Red Her.
0.67 ing Softmax as activation function6).
We then perform a post-processing step to automatically join tokens labelled with the same propaganda technique into the same textual span.
Given that PLM is applied at token-level, each token is processed into sub-words (e.g., “running” is tokenized and cut into two tokens: “run” and “##ing”). Such sub-words can mislead the classiifer. For instance, in the following sentence: “The next day, Biden said, he was informed by Indian press that there were at least a few Bidens in India.”, our system detects least a few Bidens in as a propagandist snippet, but it misclassifies one sub-word (“at” was not considered as part of “at least”, and therefore excluded from the propagandist snippet).
We cast this task as a sentence-span multi-class
classification problem. More specifically, both the
tokenized sentence and the span are used to feed
the transformer-based model RoBERTa
(robertabase pre-trained model)7
7https://huggingface.co/transformers/ model doc/roberta.html form both a sentence classification and a span classification. More precisely: i) we input a sentence to the tokenizer where max length is set to 128 with padding; ii) we input the span provided by the propaganda span-template from SemEval T11 dataset, and we set max length value of 20 with padding. RoBERTa tokenizer is applied in both cases. If a sentence does not contain propaganda spans, it is labeled as “none-propaganda”.
To take into account context features at
sentence-level, a BiLSTM is introduced. For each
sentence, semantic and argumentation features are
extracted following the methodology proposed in
To combine the results from sentence-span
based RoBERTa with the feature-based BiLSTM
we apply the joint loss strategy proposed in
Nloss where each loss value is produced from CrossEntropy function of its classifier (e.g., losssentence and lossspan from RoBERTa models of sentence and span, losssemantic argumentation features from the BiLSTM model.)
To train the above mentioned methods for
the propaganda technique classification task, we
merged the data sets of NLP4IF’19 and
SemEval’20 T11 (same setting as in Step 1). Then
we tested the full pipeline of PROTECT on the
development set from Semeval’20 T11. The output
of the snippet detection task (Step 1) are provided
as a span-pattern to the models performing Step 2.
Table 1 reports on the obtained results of the full
pipeline (Step 1+Step 2) averaged over 5 runs (we
cannot provide a fair comparison of those results
with SoTA systems, given that in SemEval the two
tasks are separately evaluated and no pipeline
results are provided). We can notice however, that
our results in a pipeline are comparable with the
ones obtained in
Given the high complexity of the propaganda technique classification task and the classes’ unbalance, some examples are miss-classified by the system. For instance, in the following sentence “The Mueller probe saw several within Trump’s orbit indicted, but not Trump’ as family or Trump himself”, the system annotated the snippet in italics as “Name Calling,Labeling”, while the correct labels would have been “Repetition”. 4
As previously introduced, PROTECT allows a user to input plain text and retrieve the propagandist spans in the message as output by the system. In the current version of the system, two services are provided through the web interface (and the API), described in the following. 4.1
The system accepts an input plain text in English, and then the architecture described in Section 3.2 is run over such text. The output consists of an annotated version of the input text, where the different propagandist techniques detected by the system are highlighted in different colours. The colour of the highlighted snippet is distinctive of a certain propaganda technique: the darker the color, the higher the confidence score of the system in assigning the label to a textual snippet. Figure 1 shows an example of PROTECT web interface. Checkboxes on the right side of the page provide the key to interpret the colors, and allow the user to check or un-check (i.e. highlight or not) the different propagandist snippets in the text, filtering the results. Faded to dark colours represent the confidence level of the prediction (the darker the colour, the higher the system confidence). The snippets in bold contain multiple propaganda techniques in the same text spans, that can be unveiled hovering with the mouse over the snippets.
As said before, PROTECT can be used through the provided API, and annotated text can be downloaded as a JSON file with the detected propagandist snippet(s) at character indices (start to end indices of a snippet) based on individual sentence, propaganda technique(s) used, and the confidence score(s)). 4.2
The propagandist snippets output by the system can also be displayed as word clouds, where the size of the words represents the system confidence score in assigning the labels (see Figure 2). The different sizes represent the confidence score of the prediction, and the colors the propaganda technique (as in Service 1). If multiple techniques are found in the same snippet, it is duplicated in the word cloud. As for the first service, a checkbox on the right side of the word clouds allows the user to select the propagandist techniques to be visualized. Also in this case, a json file can be downloaded with the system prediction.
The word cloud service has been added to PROTECT in addition to the standard visualization, to provide a different and informative way to summarise propaganda techniques on a topic, and to facilitate their identification. 5
In this paper, we presented PROTECT, a propaganda detection and classification tool. PROTECT relies on a pipeline to detect propaganda snippets from plain text. We evaluated the proposed pipeline on standard benchmarks achieving stateof-the-art results. PROTECT is deployed as a web-service API that accepts a plain text input, returning downloadable annotated text for further usage. In addition, a propaganda word clouds service allows to gain further insights from such text.
This work is partially supported by the ANSWER project PIA FSN2 n. P1595642661789/DOS0060094 between Inria and Qwant. This work has also been supported by the French government, through the 3IA Coˆ te d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002.