The present paper addresses the PAN at CLEF 2022 challenge "Profiling Irony and Stereotype Spreaders on Twitter" (IROSTEREO). The challenge strives to identify whether an author spreads sarcasm through their tweets. In general, author profiling tasks, whether mechanical or manual, are based on extensive author text. Many machine learning-based author profiling studies indicate that author-by-author classification could benefit the performance compared with a text-by-text way.
We address the challenge through fine-tuning BERT model. BERT has shown satisfactory results on many natural language processing tasks. However, BERT model cannot exert its advantages in some specific tasks, like handling long documents, due to its limitation on the maximum input token length. Our author profiling task is one of these specific tasks. The present work addresses this dilemma through a re-segmentation approach: We first concatenate all tweets of an author into one document representation. We then split the document in such a way that the split text lengths do not exceed the maximum input token length of the BERT model and that we still retain the advantages of continuous text through an appropriate choice of overlap. Ultimately, the BERT model uses the hard voting method made the final decision.
Our work first compares the performance of two pre-trained BERT models, i.e., the RoBERTa and BERTweet models, trained with external datasets. Then, we fine-tune BERT models with three different loss functions. In addition, we also demonstrate and evaluate a BERT feature-based CNN model. The winning models of the PAN author profiling task in recent years are re-implemented as baselines. Finally, the BERTweet model trained with the cross-entropy weighted focal loss function achieves an accuracy of 98.89% on the official test set. Adding a further soft voting ensemble method, which integrates BERTweet models with different loss functions as well as the BERT feature-based CNN model, we placed first in the challenge and improved our model performance to 99.44%. 1
Despite great effort being exerted by researchers and developers to detect and filter toxic language, the amount and impact of hate speech on social media are still posing serious threats to the mental health and well-being of users as well as to the possibility of democratic discourse on such platforms. According to the latest statistics [1], Twitter receives more than 5 million tweets per day. Hate speech is not in the minority in these tweets. Although Twitter has enacted a hateful conduct policy 1 , hate speech is still rampant. As an implicit way to express hatred, irony makes detection more challenging. In this work, we address the PAN at CLEF 2022 challenge: profiling the irony spreaders on Twitter. The challenge provides a dataset containing authors with a set of their tweets to identify whether the author spreads irony within their tweets.
The previous PAN author profiling tasks reveal that combining all documents from an author into one document representation and using it for author profiling has better performance than tweet-by-tweet classification. It is easy to figure out that not every tweet from an irony spreader may be ironic. Also, tweet-wise author profiling may cause much noise for classification. In accordance with this reasoning, in the PAN 2021 challenge for the detection of hate speech spreaders, the best performance was achieved by training a convolutional neural network (CNN) on the combined author document representations [2].
The Bidirectional Encoder Representations from Transformers (BERT) model [3] was proposed in 2018 and is widely used in many natural language processing (NLP) tasks due to its outstanding performance. Training a BERT model from scratch requires large amounts of training data, and the number of parameters of the BERT model is also considerable. Therefore, it requires specific hardware, and the training process is also time-consuming. Transfer learning [4] overcomes these disadvantages and makes the Transformer model more attractive compared with other approaches. However, the BERT model has a maximum token length limitation. For our task, the combined document representation could exceed this limitation. Thus, this work improves the BERT model performance with long documents by adopting a re-segmentation strategy, i.e., the combined document representation is split with overlap to fit BERT's length limit.
The remainder of the paper is structured as follows: To establish the context of this work, Section 2 looks back at the author profiling tasks of PAN and their winning models in the last decade. Section 3 describes the dataset of this year's PAN challenge and the employed text preprocessing strategies. All models are detailed in Section 4, with their training setup introduced in Section 5. Section 6 shows and analyzes the training results of each model. Finally, Section 7 summarizes the strategies, conclusion and experience of this work.
The PAN organizing committee launched a series of author profiling tasks during the past decade. Until 2021, all challenges were multilingual tasks. Table 1 gives an overview of all PAN author profiling challenges.
The objective of the Author Profiling Tasks from PAN 2013 to 2016 was to identify authors' gender and age group through their documents [5,6,7,8]. In 2015, in addition to identifying gender and age, participants also needed to score five personality traits [7]. The 2017 PAN challenge focused not only on authors' gender detection but also on the language variety identification [9]. The task in 2018 was still to detect gender, but in a multi-modal way. The organizer provided text and image data for the model training [10]. With the rapid rise in social network users, people have also become aware of the problems that arise in such virtual spaces. Hence, the PAN challenge author profiling task has a new focus on Twitter from 2019. The goal in 2019 was bot detection, meanwhile identifying the gender of the human authors [11]. The tasks for 2020 and 2021 were Twitter fake news and hate speech spreader identification, respectively [12,13].
The best result papers in the past show the development of author profiling over the last few years. Most contestants achieved the best results using conventional classifiers like support vector machines (SVMs), decision trees, Expectation Maximization Clustering (EMC), and LibLINEAR [14,15,16,17,18,19,20,21]. Among these, SVM combined with n-gram features is the most frequently used. Before 2018, researchers mainly discussed text preprocessing and feature extraction with the conventional classifiers. Since 2018, some new deep learning algorithms have gained advantages-the winning group in 2018 adopted representation fusion: text and image features are extracted by RNN and CNN, respectively [22]. Features are fused by using direct-product, column-and row-wise pooling. The fused representation of the texts and images is fed to the fully connected layers for classification. In 2021, deep learning methods still outperformed conventional classifiers. The optimal model utilizes CNN for classification; a self-trained embedding layer extracts the features [2] (detailed in Section 4.1).
Since the transformer model was proposed in 2017 [23], attention mechanisms have attracted much attention and discussion. In 2018, the BERT model, proposed by the Google team, has achieved remarkable results in many natural language processing (NLP) tasks. Due to the attractiveness of the BERT model, more and more teams choose the BERT model to handle author profiling tasks. Only one team used the BERT model in 2019 [24], in 2020, there were three teams [25,26,27], and most recently, the BERT model was widely used in the 2021 PAN author profiling task.
The author profiling tasks are usually based on many documents of that author. It is a sensible and effective strategy to profile by combining all of the author's manuscripts, as can be confirmed in some previous winning models [2]. Although the BERT model is attractive, it has a bound on the maximum input token sequence length, thus limiting the ability of the BERT model to handle long texts. To overcome this drawback, some researchers have proposed a document re-segmentation strategy, dividing long documents into sub-documents that match the maximum sequence length of BERT [28,29]. Last year, one team achieved the best accuracy for the English author profiling task by fine-tuning the BERT model with a similar strategy [30], concatenating 20 tweets of each author into one sample.
This year's PAN challenge [31] author profiling subtask is a monolingual task that aims to identify English-language sarcasm spreaders on Twitter [32]. The balanced training set contains 420 author samples, each of which has 200 tweets by this author. The official test set has 180 author samples, again with 200 tweets each. Tagged users, hashtags, and URLs are already normalized as "#USER#", "#HASHTAG#" and "#URL#", respectively. To train the model and test its effectiveness, a 4-fold cross-validation is adopted during the training stage. Thus, the official training set is split into an inner training set and a test set in each fold, with 315 and 105 author samples, respectively.
This work experiments with two BERT models. The first is the twitter-roberta-base -irony [33]2 , and the other is bertweet-large [34] 3 . Two distinct text preprocessing schemes are adopted for these two BERT models. In addition, there is one scheme for the CNN model and the TF-IDF (term frequency-inverse document frequency) features. The TF-IDF features are used to train conventional classification models like SVMs.
β’ Scheme 1: for the twitter-roberta-base-irony model remove "#USER#", "#HASHTAG#" and "#URL#" -replace multiple spaces by one single space convert emojis to text with the Python emoji replace "#USER#" with "@USER" -replace "#HASHTAG#" with "#HASHTA" -replace "#URL#" with "HTTPURL" -text normalization with the embedded normalization processes
β’ Scheme 3: for the CNN model and the conventional classification approaches like SVMs, linear regression (LR), and random forest (RF) classifiers.
remove "#USER#", "#HASHTAG#" and "#URL#" -replace multiple spaces by a single space convert emojis to text with the Python emoji package normalize all text into lowercase -remove punctuation and numbers remove stop words
The TF-IDF features are obtained by word-based 1-to 3-gram and character-based 3-to 5-gram models. Then the truncated singular value decomposition (SVD) [35] 5 reduces the feature dimension to 1000 to reduce the computational complexity. Specifically, the TF-IDF features are extracted by the TfidfVectorizer function from the scikit-learn library 6 . The minimum document frequency is 2, and the maximum is set to 100%, i.e., the terms occurring in all documents or in less than two documents are ignored. The word-based and character-based models are obtained separately, each producing a 1000-dimensional vector of every input tweet. Finally, the two vectors are concatenated as a representation of the tweet.
As introduced in Section 2, the BERT model is limited in handling long documents. However, profiling authors based on long manuscripts can benefit the accuracy. To address this conflict, a re-segmentation strategy is adopted to make the sequence length of the sub-document fit the maximum input token length of the BERT model. The continuity of the segmented sentences is guaranteed by overlapping segmentation. The specific steps are as follows:
β’ concatenate all 200 tweets of each author β’ the new sub-document is segmented with the same text length π and with an overlap of π. To simplify the program, only the number of words in the text is considered here instead of the number of tokens. Therefore, π should be smaller than the maximum token length of the BERT model to guarantee that there are no text segments that are too long for BERT. β’ the author's label is assigned to every sub-document of that author Two external irony detection datasets were utilized to pre-train both BERT models for better performance. One is the Ironic Corpus [36], the other is the SemEval-2018 irony detection dataset [37]. The datasets were labeled on each document. Only 0.168% document exceeds the BERT maximum token length limitation. Therefore, we have not applied the sub-segmentation strategy for these two corpora.
Besides two BERT-based models, we also consider a CNN model that builds on the BERT embeddings. In addition, three traditional classifiers are also trained as baselines.
The models below are used as baselines. Among those, you can also find the winning models of the PAN author profiling task in previous years, in our respective re-implementations.
β’ SVM, LR, and RF models: All tweets of an author are concatenated and preprocessed according to Scheme 3. The TF-IDF features are extracted from the processed document. Finally, the scikit-learn library is utilized to train the models.
β’ CNN : We re-implemented the winning model of the PAN 2021 author profiling task [2]. Again, all the author's tweets are concatenated, and Scheme 3 is adopted for text normalization. The model structure is the same as in [2]. The embedding layer projects each input token into a 100-dimensional vector. A 1D-convolution layer with 64 filters of size 36 was applied to the embedding tensors. Then an average pooling with a size of 8 reduces the features' complexity, and the global average pooling decreases the dimension of the features. Finally, a fully connected layer outputs the results in the desired size.
Text-pre-processing
We initially chose the RoBERTa model (twitter-roberta-base-irony) [33] and pretrained it on the SemEval2018 irony detection database [37]. However, the leaderboard of TweetEval on GitHub 7 indicates that the BERTweet model (bertweet-large) [34] outperforms other candidate models for the sarcasm identification task. Therefore, the BERTweet model is also considered. Both base models are first trained with external datasets, and these pre-trained models are marked as RoBERTa-ext and BERTweet-ext. Subsequently, these two pre-trained models are fine-tuned with the PAN challenge dataset to obtain the final models RoBERTa and BERTweet. The input text π· π is processed using the above text preprocessing and re-segmentation strategy. The BERT model takes one sub-document at a time and gives two outputs. One is the probability predicted based on the current sub-document π(π π π |π· π π ), and the other is the corresponding word-embeddings E π π . As stated in [3], there are many benefits to training a model using BERT embeddings as fixed features. On the one hand, the BERT model structure cannot suit all tasks, rather, sometimes it needs to add some task-specific design to increase flexibility. On the other hand, featurebased methods can speed up the computation because the text representation only needs to be computed once. This paper adopts the CNN layer for classification (BERTweet-CONV ). The word embeddings of all sub-documents of an author are concatenated to train the CNN model, where the embeddings are extracted from the pre-trained BERTweet model. The CNN model structure here is principally the same as in the CNN described in Section 4.1, the only difference is the input dimension. The Bi-LSTM layer is not considered because the word-embeddings of all sub-documents are concatenated. The concatenated word-embeddings could be regarded as a word-embedding of a long document, and its token length could be as long as 4000. For such a lengthy document, BLSTM layers could face the problem of vanishing gradients and exploding gradients.
Figure 2 illustrates the details of the BERTweet-CONV model. On the left, the CNN model based on the BERTweet embedding is shown. The concatenated embedding is the input of the CNN model. On the right side, the predicted probabilities of the BERTweet model for each sub-document of that author are utilized. The average of the probabilities of each sub-document π π΅πΈπ π (π π |π· π ) and the probabilities predicted by the CNN model π πΆπ π (π π |π· π ) are input to an attention weighting block. The probability π π΄π‘π‘ (π π |π· π ) on the right is obtained as follows:
where
During the first two epochs, only the CNN model parameters are trained because in comparison with the CNN, the initial BERTweet model predictions are too accurate early in the training, which could dominate the entire model. After two epochs, the final probability is predicted using attention-weighting for both model type outputs.
Participants can access only the training set at the beginning. For each DNN model, our work first uses the Python RAY8 package to find the best hyperparameters. The training set has a total of 420 author samples. To find the best set of hyperparameters, 100 authors were randomly selected as the internal test set and the remaining 320 authors were used for the internal training set. When the best set of hyperparameters is found, 4-fold cross-validation is applied to test the robustness of the model. For this purpose, 105 authors are used as the internal test set in each fold. We used the scikit-learn library GridSearchCV package to find the optimal hyperparameters for the SVM and LR models, while the RandomizedSearchCV package is applied for the RF model. Table 2 lists the optimal hyperparameters of different DNN models. As previously described, we use the re-segmentation strategy to improve the performance of the BERT model. However, the re-segmentation leads to a problem: the originally balanced dataset is now imbalanced. The cross-entropy (CE) loss function for binary classification
generally does not perform well on imbalanced data [38]. In contrast to the cross-entropy, the focal loss function [39] for binary classification
assigns a larger weight to poorly estimated training samples.
This work adopts the Cross-Entropy Weighted Focal (CEWF) loss function [38] to deal with this problem during the model training. It is defined as
where π π is the estimated target probability. The CEWF is a compromise between the CE and focal loss functions. When the classifier is very confident about the classification probability, the CEWF is close to CE; otherwise, the CEWF is close to the focal loss function. In our work, we set π‘ = 4 and πΎ = 5 in Equation 5.
The RoBERTa and BERTweet models make predictions π π π for each segment π· π π because of the re-segmentation strategy. The prediction of one author's class π π is obtained by applying hard voting (HV) over all of the sub-segments of this author:
To implement the ensemble approach, we also use the soft voting method (SV) to obtain the class probability of each author:
All our models are trained by the PyTorch library 9 . Early stopping prevents overfitting. Specifically, the training is terminated if the evaluation accuracy does not improve within four epochs. The AdamW optimizer optimizes the parameters of the BERT model; the get_linear _schedule_with_warmup from the transformer model is used to schedule the learning rate. The learning rate has a warm-up process in the first four epochs, its upper limit is given in Table 2. Other models are optimized by Adam; the learning rate is scheduled by ReduceLROnPlateau, i.e., the learning rate is reduced by 50% if the evaluation accuracy does not improve. For the BERT fine-tuning models, the segment length π is 500 with an overlap of π = 128. Since the BERT-CONV model is based on the trained BERTweet model, the order of the author's tweets is shuffled to avoid overfitting during the BERT-CONV model training phase; the segment length π is still 500, while the overlap length π is 64. The output dimension in all models is 2 with a softmax output function. The training process is carried out on NVIDIA's Volta-based DGX-1 multi-GPU system, using 2 TeslaV100 GPUs with 32 GB memory each.
Our experimental results are presented in this section. Having established this, we focus on the BERTweet model with different loss functions. However, the results of the three loss functions are not significantly different. The best average accuracy is the BERTweet model with the CEWF loss function. Due to the outstanding performance of the BERTweet + CEWF model, the feature-based BERTweet-CONV model utilizes the BERTweet + CEWF to extract the embeddings as the fixed features. However, the feature-based strategy did not improve model accuracy on average. This conclusion is also in line with what was claimed in [3], although, interestingly, the BERTweet-CONV model achieves the best accuracy in one of the folds. Ultimately, the soft voting ensemble learning method is implemented to boost the final performance. The author class probability π(π π |π· π ) of the BERTweet model is obtained through Equation 7. We integrate the BERTweet models with different loss functions, as well as the BERT feature-based CNN model BERTweet-CONV. The ensemble model achieves the best accuracy on average (ensemble1,2,3,4 in Table 4).
For comparison and completeness, we also give the results of the 'classical' baseline modelsthe SVM, LR, and RF models-in Table 5. These three models are trained on the same internal training and test sets, i.e., 320 author samples are used for training; 100 author samples form the test set. Comparing Table 4 and Table 5, all results of the BERTweet model are much better than those of the SVM, LR, and RF models. Among these three models, the LR, and RF models are inferior to the SVM model.
The evaluation on the official test set was performed on the TIRA platform [40]. The hard voting of the 4-fold BERTweet + CEWF model achieves an accuracy of 98.89%. The soft voting of the 4-fold ensemble1,2,3,4 model improves the performance to 99.44% on the official PAN test set.
This work describes our models for the PAN 2022 challenge, which poses the task of identifying whether an author is spreading sarcasm in their tweets. By using a re-segmentation strategy for lengthy documents, we can overcome the text length limitation of the BERT model. We compare two BERT models, specifically looking at the Roberta model in comparison to the BERTweet model. Our experiments show that the BERTweet model is clearly more suitable for the sarcasm discrimination task in twitter data. Based on the BERTweet model, a feature-based model is also designed. However, the feature-based model can not improve the accuracy compared to fine-tuning the BERT model on average. Nevertheless, the advantage of the feature-based model cannot be neglected: Compared to fine-tuning a BERT model, training a feature-based model is faster, while the results are also comparable to the fine-tuned BERT model. In this work, we consider three different loss functions, seeing that the cross-entropy weighted focal loss function as a compromise between the cross-entropy and the focal loss function yields slightly better results. Three conventional classifiers are also evaluated, namely, SVM, LR, and RF models, but the BERTweet model far outperforms these traditional classifiers. Finally, our experiments demonstrate that an ensemble approach based on soft voting of BERTweet models with different loss functions and the feature-based CNN model can further boost performance on the PAN challenge task.
The work was supported by the PhD School "SecHuman -Security for Humans in Cyberspace" by the federal state of NRW, and partially funded by the Deutsche Forschungsgemeinschaft (DFG -German Research Foundation) [Project-ID 429873205] and by the German Federal Ministry of Education and Research [Grant No: 16KIS1518K]. The authors are responsible for the content of this publication.