Political competitions are complex settings where candidates use campaigns to promote their chances to be elected. As we can recently observe, some candidates choose to focus on a negative campaign that emphasizes the negative aspects of the competing person and is aimed at offending opponents or the opponent's supporters. The big challenge in this area is the lack of annotated datasets for training efficient classifiers. Therefore, transfer learning from other relevant domains and other languages could be very useful for this task. Considering the recent success of meta-learning in domain adaptation, we apply it to our task of utilizing available datasets from different domains and languages. This work explores the negative campaign detection task from multiple perspectives: the efficiency of different text representations and classification models, and the effect of transfer learning from offensive language detection in different languages for negative campaign detection in Hebrew. We demonstrate that the lack of training data for negative campaign detection in a low-resourced language such as Hebrew can be compensated to some extent by available datasets for offensive language detection in the same and other languages. We report an empirical case study for political campaigns in Israeli municipal elections. 1
Political competitions aim at promoting the candidates' chances to be elected. The main decision in such competitions regards the nature of the campaign -that is, whether a candidate should apply a positive campaign that highlights the candidate's achievements, leadership skills, and future programs, or focus on a negative campaign that emphasizes the negative sides of the competitors [1,2,3,4].
In recent years, we witness the intensive use of negative campaigns by political candidates which target the weaknesses and failures of the opponents promising to do the opposite [2,3,4].
The application of language technologies in the political sciences is recently in high demand [5]. However, despite some works dedicated to the analysis of elections-related materials [6,7,8], we were unable to find any work on automated negative campaign analysis and detection.
Our work reports the results of extensive experiments, aimed at answering multiple research questions: (1) Which supervised model and representation are more effective at automatically detecting negative campaigns in Hebrew? (2) Can we effectively detect negative campaigns with a model trained to identify offensive language? (3) Can meta-learning with different domains and languages boost negative campaign detection in Hebrew?
We adopt and extend the representation models applied in [9,10,11], where the gain of semantic vectors and sentiment knowledge for offensive language and negative campaign detection was empirically shown. In order to increase classification accuracy in a mono-domain setting, we use knowledge about cities, country districts (regions), and politicians. We use this information in a meta-learning setting as well. In [10], we have also shown the efficiency of transfer learning for cross-lingual training of offensive language classifiers with Semitic languages. We adopt and explore this idea for this study. The lack of Hebrew datasets is addressed in this study by using cross-domain and cross-lingual transfer learning, in contrast to [11].
Our contribution is multi-fold: (1) we experimented with different representations and classifiers for efficient encoding and classification of texts in Hebrew for negative campaign detection;
(2) we explored the efficiency of meta-learning in mono-domain experiments; (3) we explored an efficiency of a transfer learning from offensive language detection in different languages to negative campaign detection; (4) we explored a gain of meta-learning vs. conventional fine-tuning of language models in transfer learning for cross-domain experiments.
The data was collected from the Facebook accounts of local politicians from several big Israeli cities running for mayor's offices. There was a total of 12 cities and 27 mayoral candidates whose number for elections that took place in 2018. Data statistics appear in Table 1. The data is freely available for download from GitHub at https://github.com/NataliaVanetik1/TONIC. Collected posts were annotated as either negative or not by two independent annotators; in case of a disagreement between them, the third annotator decided on a final label. The annotators were instructed to label a post as a "negative campaign" only if it contained negative (but not necessarily offensive) content about the opponent of the post's owner or her supporter. Kappa agreement between the annotators was 0.862. The majority rule, i.e., the portion of the bigger class in our data, is 0.78 (the distribution between two classes is 78% β 22%, with the majority class being benign texts, and the minority class containing negative campaign texts).
Our approach follows a standard flow of supervised learning, including text representation, model training, and its application on a test set for the model's evaluation. The following techniques were employed for the post representation:
β’ Term frequency-inverse document frequency (tf-idf), where every post is treated as a separate document and the whole dataset as a corpus. β’ N-grams of π consecutive words seen in the text, with π = 1, 2, 3.
β’ BERT sentence embeddings using one of the pre-trained BERT models-a multilingual model [12] and a Hebrew model [13]. We use BERT embeddings to represent post text, region, and city. β’ Sentiment weights generated by the HeBERT model [14], producing a probability distribution for positive, negative, and neutral sentiments, for every post.
For classification, we experimented with three different types of classifiers:
β’ Traditional classsifiers, including Random Forest (RF) [15], Logistic Regression (LR) [16], and Extreme Gradient Boosting (XGB) [17]. β’ Fine-tuned BERT, including a multilingual model called bert-base-multilingual-cased (denoted as mBERT) [18] and AlephBERT [13], a large pre-trained language model for Modern Hebrew. Both models were fine-tuned on the train portion of our data. β’ Meta-learning, where create a meta-model for detecting unfavorable campaigns when training data for this particular task and language is missing (or not sufficient). To quickly adapt to new target cases, ModelAgnostic Meta-Learning (MAML) [19], a general optimization framework, uses the gradient descent process to create a strong initial model. Therefore, in this study, we used MAML for meta-learning. As the foundation for our meta-learning, we use a pre-trained BERT language model as a base model. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. We use three different criteria to split our data into training tasks: (1) an account of politician, where one training task aims at the identification of posts with negative campaigns published by the same politician; (2) a city, where a training task focuses on the data generated by politicians from the same particular city; and (3) a region of the country, where we train our model on the annotated posts generated by politicians from the same region of the country.
A full pipeline of our approach is depicted in Figure 1.
Our experiments aim to evaluate (1) different models and representations of Hebrew data in the negative campaign domain; (2) transfer learning from the hate speech domain, in Hebrew and other languages; and (3) meta-learning approach in mono-domain and cross-domain learning.
For the monolingual experiments on the TONIC dataset, RF, LR, and XGB are trained on 80% of the dataset and evaluated on the remaining 20%. For the cross-domain monolingual experiments, the models are trained on 100% of the other domain data and tested on 20% of the TONIC dataset. For the cross-domain cross-lingual experiments, we train our models on 100% of the data in another language, and test on the 20% of the TONIC dataset. In all cases, the test portion of the TONIC dataset is the same which allows us to conduct proper statistical significance analysis. Fine-tuned BERT was trained a 75% of the data with the validation set containing 5% of the data, and it was tested on the remaining 20%. Fine-tuning was run for 10 epochs with batch size 16. For the cross-domain experiments, we used the Hebrew offensive language dataset [20] called OLaH. Traditional models were implemented in sklearn [21] and neural models were implemented in Keras [22] with the TensorFlow backend [23]. Experiments were performed on google colab [24] with Pro settings.
Here we report the results-precision, recall, f1-measure, and accuracy scores-of the evaluation of and comparison of various models and text representations to detect negative campaigns in political posts written in Hebrew. In particular, we explore whether or not BERT sentence embeddings perform better than traditional text representations such as tf-idf and n-grams. We also compare two pre-trained BERT models to determine whether a model specifically trained in Hebrew is preferable.
Table 2 (left) summarizes the results for the conventional models and representations without sentence embeddings. All models were trained and tested on the TONIC training and test sets, respectively. The text representations use either tf-idf or n-grams (ngX denotes n-grams for π = 1, 2, 3), or their combinations (tfidf-ngX denotes a concatenation of tf-idf vectors with n-grams of sizeπ = 1, 2, 3). All the systems are significantly better than the majority rule. Also, the XGB classifier with tf-idf, unigrams, and sentiment labels outperforms the other classifiers. Confusion matrix of the top-performing model (XGB ππππ‘+ππππππ+πππ ) contains TP = 75, TN = 391, FP = 22, and FN = 39, with ππππππ πππ = 0.77 and ππππππ = 0.66. These results show that the model does a good job of identifying and eliminating negative samples (non-negative campaigns), but it misses positive samples (negative campaigns). As a result, TN is the most important accuracy compound, while FN represents the biggest amount of errors. In a 10 misclassified case sample that we manually examined, more than half of the errors (6), including four samples incorrectly identified as negative campaigns when we actually found them to be neutral and two samples incorrectly labeled as neutral, were the result of incorrect labeling by our annotators.
Table 3 shows the scores for the same models over sentence embeddings, produced by two different BERT models-multilingual BERT [25] and Hebrew-language AlephBERT [13]. We can see that enriched sentence embeddings of cities and regions' names boost the classification performance. XGB outperforms the other classifiers as in the previous experiment. We cannot recommend one particular BERT model, because both models seem to provide sentence embeddings with similar quality. However, when we compare these BERT models fine-tuned on the classification task on TONIC (see Table 4), AlephBERT, which is trained solely in Hebrew, significantly outperforms multilingual BERT producing accuracy which falls below the majority rule. Nonetheless, both models are outperformed by the best traditional models, probably due to less information encoded in the text representation. While both BERT classifiers use only self-produced embeddings, traditional models also utilize sentiment labels, and embeddings representing the cities and regions of the candidates.
Table 4 contains the results of meta-learning where tasks are specified by three different criteria.
on Hebrew; and (4) meta-learning outperforms language models traditionally fine-tuned in crossdomain and cross-lingual scenarios, but not in a mono-lingual setting. We also observe that in a monolingual setting that employs either a fine-tuned BERT or BERT sentence embedding, the AlephBERT model trained on Hebrew is preferable to a multilingual BERT model. In the future, we plan to apply our analysis to elections for the Israeli government, to explore the common characteristics and differences between political campaigns in different countries, and to study possible relations between the candidate's gender, perceived strength, initial support, etc. and their engagement in a negative campaign.
We can see that multilingual BERT achieves the best accuracy score; however, for all the options for task division, meta-learning scores are very close to the majority rule, that evidence that there is not much information that can be efficiently learned and transferred between tasks. We can also see that for a fine-tuned BERT, AlephBERT has a clear advantage over the multilingual BERT model in all parameters.
According to the scores in Tables 2 and 3 (we omitted meta-learning models because of their low performance), the top performing model is XGB, applied on bert embeddings enriched by region and location embeddings. In general, the XGB classifier outperforms other classifiers in most cases.
Cross-domain mono-lingual (all models were trained and tested on Hebrew data) experiments in Table 2 (right) show that using an offensive language dataset as a training set decreases classification accuracy for all the models, indicating that the task of detecting negative campaigns is different from the task of offensive language detection. Only a few models trained on offensive language data achieved accuracy that is slightly higher than or equal to the majority rule. Additionally, we can see that F1 scores are really low, meaning that models simply 'guess' the majority rule.
Table 5 shows the results of the traditional models with BERT embeddings as a text representation for transfer learning from the offensive language detection in Hebrew. From Table 2 (right) and Table 5, we can conclude that (1) the XGB classifier mostly performs better than other classifiers and (2) its performance is slightly higher with BERT embeddings than with tf-idf vectors and n-grams. Table 6 shows the results of meta-learning trained on hate speech data and tested on the TONIC dataset. Two BERT models are initiated with the weights generated by meta-learning. The table also contains the scores of fine-tuned BERT without meta-learning.
We can see that (1) best traditional models perform better than both fine-tuned language models and meta-models when trained in foreign languages, the only exception is the recall and F1 scores of meta-learning which is evidence of its better ability to recognize the positive samples-negative political campaign-but fail at filtering out neutral posts (also confirmed by lower Precision); (2) AlephBERT performs better with meta-learning than multilingual BERT;
(3) meta-learning outperforms fine-tuned language models in terms of both precision and recall.
Table 7 shows the evaluation of traditional models for the cross-domain cross-lingual scenario.
In this setting, we train our models on hate speech datasets in other languages -English and Arabic. The only text representation that we can use here is multilingual BERT sentence embeddings generated by the pre-trained BERT model bert-base-multilingual-cased [18]. Table 8 shows the results of meta-learning trained on hate speech data in other languages (Arabic and English) and tested on the TONIC dataset. An English-language dataset is the Offensive Language Identification Dataset (OLID) [26], which is a collection of 14,100 tweets (we used 13,240 annotated tweets from its training set). We used the OLaA dataset in Arabic, which we collected and introduced in [9] previously. OLaA is a collection of 9,000 comments from Twitter annotated for hate speech. We used a multilingual BERT model [18] for these experiments. For comparison, we also show the scores of this BERT model fine-tuned on Arabic and English hate-speech data and tested on TONIC.
Both experiments evidence that meta-learning adapts pre-trained models much better to the new domains than traditional fine-tuning and it can be efficiently applied for transfer learning from other domains and even languages. In particular, we can observe the following:
(1) fine-tuned language models and meta-learning perform better than best traditional models when trained on foreign languages; (2) meta-learning outperforms fine-tuned language models.
Based on the results of extensive experiments aimed to answer various research questions (see Section 1), we can conclude that (1) the best combination of text representation and classification model for negative campaign detection in Hebrew texts is XGB with sentence embeddings enriched with region and location information; (2) transfer learning with models trained to detect offensive content is inefficient for the detection of a negative campaign; meaning that there is no strong relation between offensive language and negative campaigns; (3) transfer learning from different languages can be applied to Hebrew in the negative campaign detection task, while training on a large set in a foreign language can be even more efficient than training