Political competitions are complex settings where candidates use campaigns to promote their chances to be elected. As we can recently observe, some candidates choose to focus on a negative campaign that emphasizes the negative aspects of the competing person and is aimed at ofending opponents or the opponent's supporters. The big challenge in this area is the lack of annotated datasets for training eficient classifiers. Therefore, transfer learning from other relevant domains and other languages could be very useful for this task. Considering the recent success of meta-learning in domain adaptation, we apply it to our task of utilizing available datasets from diferent domains and languages. This work explores the negative campaign detection task from multiple perspectives: the eficiency of diferent text representations and classification models, and the efect of transfer learning from ofensive language detection in diferent languages for negative campaign detection in Hebrew. We demonstrate that the lack of training data for negative campaign detection in a low-resourced language such as Hebrew can be compensated to some extent by available datasets for ofensive language detection in the same and other languages. We report an empirical case study for political campaigns in Israeli municipal elections.1 Political competitions aim at promoting the candidates' chances to be elected. The main decision in such competitions regards the nature of the campaign - that is, whether a candidate should apply a positive campaign that highlights the candidate's achievements, leadership skills, and future programs, or focus on a negative campaign that emphasizes the negative sides of the In recent years, we witness the intensive use of negative campaigns by political candidates which target the weaknesses and failures of the opponents promising to do the opposite [2, 3, 4]. 1Our dataset is freely available for researchers at https://github.com/NataliaVanetik1/TONIC. ∗Corresponding author. †These authors contributed equally.
The application of language technologies in the political sciences is recently in high demand [5]. However, despite some works dedicated to the analysis of elections-related materials [6, 7, 8], we were unable to find any work on automated negative campaign analysis and detection.
Our work reports the results of extensive experiments, aimed at answering multiple research
questions: (
We adopt and extend the representation models applied in [9, 10, 11], where the gain of semantic vectors and sentiment knowledge for ofensive language and negative campaign detection was empirically shown. In order to increase classification accuracy in a mono-domain setting, we use knowledge about cities, country districts (regions), and politicians. We use this information in a meta-learning setting as well. In [10], we have also shown the eficiency of transfer learning for cross-lingual training of ofensive language classifiers with Semitic languages. We adopt and explore this idea for this study. The lack of Hebrew datasets is addressed in this study by using cross-domain and cross-lingual transfer learning, in contrast to [11].
Our contribution is multi-fold: (
The data was collected from the Facebook accounts of local politicians from several big Israeli cities running for mayor’s ofices. There was a total of 12 cities and 27 mayoral candidates whose number for elections that took place in 2018. Data statistics appear in Table 1. The data is freely available for download from GitHub at https://github.com/NataliaVanetik1/TONIC. Collected posts were annotated as either negative or not by two independent annotators; in case of a disagreement between them, the third annotator decided on a final label. The annotators were instructed to label a post as a “negative campaign” only if it contained negative (but not necessarily ofensive) content about the opponent of the post’s owner or her supporter. Kappa agreement between the annotators was 0.862. The majority rule, i.e., the portion of the bigger class in our data, is 0.78 (the distribution between two classes is 78% − 22%, with the majority class being benign texts, and the minority class containing negative campaign texts).
Our approach follows a standard flow of supervised learning, including text representation, model training, and its application on a test set for the model’s evaluation. The following techniques were employed for the post representation: • Term frequency-inverse document frequency (tf-idf), where every post is treated as a separate document and the whole dataset as a corpus. • N-grams of consecutive words seen in the text, with = 1, 2, 3 . • BERT sentence embeddings using one of the pre-trained BERT models—a multilingual model [12] and a Hebrew model [13]. We use BERT embeddings to represent post text, region, and city. • Sentiment weights generated by the HeBERT model [14], producing a probability distribution for positive, negative, and neutral sentiments, for every post.
classifiers :
• Traditional classsifiers , including Random Forest (RF) [15], Logistic Regression
(LR) [16], and Extreme Gradient Boosting (XGB) [17].
• Fine-tuned BERT, including a multilingual model called bert-base-multilingual-cased
(denoted as mBERT) [18] and AlephBERT [13], a large pre-trained language model for
Modern Hebrew. Both models were fine-tuned on the train portion of our data.
• Meta-learning, where create a meta-model for detecting unfavorable campaigns when
training data for this particular task and language is missing (or not suficient). To
quickly adapt to new target cases, ModelAgnostic Meta-Learning (MAML) [19], a general
optimization framework, uses the gradient descent process to create a strong initial model.
Therefore, in this study, we used MAML for meta-learning. As the foundation for our
meta-learning, we use a pre-trained BERT language model as a base model. The goal of
meta-learning is to train a model on a variety of learning tasks, such that it can solve
new learning tasks using only a small number of training samples. We use three diferent
criteria to split our data into training tasks: (
BERT sentence vectors n-gram vectors posts tokenization tf ∗ idf vectors sentiment analysis prediction model
Our experiments aim to evaluate (
Mono-domain Evaluation Results Here we report the results–precision, recall, f1-measure, and accuracy scores–of the evaluation of and comparison of various models and text representations to detect negative campaigns in political posts written in Hebrew. In particular, we explore whether or not BERT sentence embeddings perform better than traditional text representations such as tf-idf and n-grams. We also compare two pre-trained BERT models to determine whether a model specifically trained in Hebrew is preferable.
Table 2 (left) summarizes the results for the conventional models and representations without sentence embeddings. All models were trained and tested on the TONIC training and test sets, respectively. The text representations use either tf-idf or n-grams (ngX denotes n-grams for = 1, 2, 3 ), or their combinations (tfidf-ngX denotes a concatenation of tf-idf vectors with model RF + LR + XGB + RF1+ LR1+ XGB1+ RF2+ LR2+ XGB2+ RF3+ LR3+ XGB3+ RF +1+ LR +1+ XGB +1+ RF +2+ LR +2+ XGB +2+ RF +3+ LR +3+ XGB +3+ model n-grams of size = 1, 2, 3 ). All the systems are significantly better than the majority rule. Also, the XGB classifier with tf-idf, unigrams, and sentiment labels outperforms the other classifiers.
Confusion matrix of the top-performing model (XGB++ ) contains TP = 75, TN =
391, FP = 22, and FN = 39, with = 0.77 and = 0.66 . These results show that
the model does a good job of identifying and eliminating negative samples (non-negative
campaigns), but it misses positive samples (negative campaigns). As a result, TN is the most
important accuracy compound, while FN represents the biggest amount of errors. In a 10
misclassified case sample that we manually examined, more than half of the errors (
Table 3 shows the scores for the same models over sentence embeddings, produced by two diferent BERT models–multilingual BERT [ 25] and Hebrew-language AlephBERT [13]. We can see that enriched sentence embeddings of cities and regions’ names boost the classification performance. XGB outperforms the other classifiers as in the previous experiment. We cannot recommend one particular BERT model, because both models seem to provide sentence embeddings with similar quality. However, when we compare these BERT models fine-tuned on the classification task on TONIC (see Table 4), AlephBERT, which is trained solely in Hebrew, significantly outperforms multilingual BERT producing accuracy which falls below the majority rule. Nonetheless, both models are outperformed by the best traditional models, probably due to less information encoded in the text representation. While both BERT classifiers use only self-produced embeddings, traditional models also utilize sentiment labels, and embeddings representing the cities and regions of the candidates.
Table 4 contains the results of meta-learning where tasks are specified by three diferent criteria.
We can see that multilingual BERT achieves the best accuracy score; however, for all the options for task division, meta-learning scores are very close to the majority rule, that evidence that there is not much information that can be eficiently learned and transferred between tasks. We can also see that for a fine-tuned BERT, AlephBERT has a clear advantage over the multilingual BERT model in all parameters.
According to the scores in Tables 2 and 3 (we omitted meta-learning models because of their low performance), the top performing model is XGB, applied on bert embeddings enriched by region and location embeddings. In general, the XGB classifier outperforms other classifiers in most cases.
Cross-domain mono-lingual (all models were trained and tested on Hebrew data) experiments
in Table 2 (right) show that using an ofensive language dataset as a training set decreases
classification accuracy for all the models, indicating that the task of detecting negative campaigns
is diferent from the task of ofensive language detection. Only a few models trained on ofensive
language data achieved accuracy that is slightly higher than or equal to the majority rule.
Additionally, we can see that F1 scores are really low, meaning that these models simply ’guess’
the majority rule.
sentation for transfer learning from the ofensive language detection in Hebrew. From Table 2
(right) and Table 5, we can conclude that (
mBERT TONIC dataset. Two BERT models are initiated with the weights generated by meta-learning. The table also contains the scores of fine-tuned BERT without meta-learning.
We can see that (
Table 7 shows the evaluation of traditional models for the cross-domain cross-lingual scenario. In this setting, we train our models on hate speech datasets in other languages - English and Arabic. The only text representation that we can use here is multilingual BERT sentence embeddings generated by the pre-trained BERT model bert-base-multilingual-cased [18].
Table 8 shows the results of meta-learning trained on hate speech data in other languages (Arabic and English) and tested on the TONIC dataset. An English-language dataset is the Ofensive Language Identification Dataset (OLID) [ 26], which is a collection of 14,100 tweets (we used 13,240 annotated tweets from its training set). We used the OLaA dataset in Arabic, which we collected and introduced in [9] previously. OLaA is a collection of 9,000 comments from Twitter annotated for hate speech. We used a multilingual BERT model [18] for these experiments. For comparison, we also show the scores of this BERT model fine-tuned on Arabic and English hate-speech data and tested on TONIC.
Both experiments evidence that meta-learning adapts pre-trained models much better to
the new domains than traditional fine-tuning and it can be eficiently applied for transfer
learning from other domains and even languages. In particular, we can observe the following:
(
Based on the results of extensive experiments aimed to answer various research questions (see
Section 1), we can conclude that (
URL: https://www.tensorflow.org/, software available from tensorflow.org. [24] E. Bisong, Building machine learning and deep learning models on Google cloud platform:
A comprehensive guide for beginners, Apress, 2019. [25] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [26] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, Predicting the Type and Target of Ofensive Posts in Social Media, in: Proceedings of NAACL, 2019, p. 1415–1420.