In this article, we report the solution of the team BERT 4EVER for the Irony Detection in the Portuguese language task in IberLeF 2021, which aims to identify irony news articles or tweets in the Portuguese language spread via digital media. We propose the BERT (Bidirectional Encoder Representations from Transformers) model to tackle the problem. In addition, we adopt weight loss and ensemble learning to improve the generalization capability. Experimental results as well as the leading position of our team on the task leaderboard demonstrate the effectiveness of our method in the field of news.
Irony refers to the use of words contrary to the original meaning to express the
meaning, which is a form of figurative language with strong emotional color. The irony
detection is a key-challenge in various tasks involving Natural Language Processing
(NLP). In the field of Opinion Mining, for instance, Luís Sarmento et al. [
Most of the research studies in Irony Detection focus on English language [
Recently, the NLP community also focuses on other languages for the need to
develop linguistic and computational resources, which has spawned a multitude of
irony detection competitions for other languages, such as Arabic [
Portuguese is a low-resource language, which limits the amount of research done for this language. Freitas et al. proposed a set of patterns that might suggest ironic/sarcastic statements by observing a corpus constituted by tweets [9]. In particular, they developed special clues for irony detection, through the implementation and evaluation of a set of patterns. Fabio Ricardo Araujo da Silva [10] proposed a Convolutional Neural Network (CNN) adapted for automatically detecting irony/sarcasm in Brazilian Portuguese, which was trained and tested by datasets from Twitter obtained by the author and from thirds. For the reason that there is no website corpus in Portuguese corpus (only the corpus from social network twitter), Gabriel Schubert Marten et al. [11] developed a corpus in the Portuguese language to sarcasm and irony detection task. 3
As shown in Figure 1, based on the BERT model, we train three kinds of models with different strategies. In the prediction stage, we fuse the prediction results of the three models for each field (news/twitter). These three strategies are as follows: (1) We fine-turn the BERT model separately for the training set in each field. (2) On the basis of (1), we adopt the Loss Weight strategy for the training set in each field to solve data imbalance.
(3) We combine the data from these two fields together and fine-tune the BERT model, so as to make use of the information from the other field to assist classification and improve the generalization ability of the model. 3.1
BERT
BERT (Bidirectional Encoder Representations for Transformers) is designed to pretrain deep bidirectional representations from unlabeled text by jointly building models both left and right context in all layers [12]. The BERT model structure is shown in Figure 2. where the value of is e.
Model Fusion
It consists of two pretrain tasks, namely Mask Language Model (MLM) and Next Sentence Prediction (NSP) :(1) MLM is defined as masking some words in the input sequence, and then predicting the masked words according to the context; (2) NSP refers to predicting whether the second sentence is the follow-up (next sentence) of the first sentence. 3.2 Weight Loss We can see that in the field of news, there are more data that are not ironic, while in the field of tweets, there are more data that are ironic. In this paper, we adopt a class weight adjustment method [13] to tackle the problem of data with class imbalance. Assuming that the given labels = (!, ", . . . , #), the class weight $ of the -th label is calculated as follows.
$ = %&(
$ ∑$'(! $ ) (8)
We train multiple models through multiple strategies. Each model predicts the test data separately. For each sample , the predicted probability of the model is ! + " + ) ! + " + ) = [ , ]
3 3 in which $ is the “no irony” probability of -th BERT model, $ is the “irony” probability of -th BERT model. 4
Based on five-fold cross-validation, we reported the result of BERTs model with three strategies and other machine learning algorithms. We used Transformers2 library and Pytorch3 library as backend to construct BERT-based models and scikit-learn4 to construct machine learning models. The BERT5 model we used was pre-trained by Souza et. al [14]. When using other machine learning algorithms, we selected the text features with TFIDF. We used Bacc (Balanced Accuracy) as the evaluation indicator and the results were shown in Table 2. We could see that the performance of most algorithms was excellent, and the BERT models with three strategies were more than 0.99 on the validation set. 2 https://github.com/huggingface/transformers 3 https://github.com/pytorch/pytorch 4 https://github.com/scikit-learn/scikit-learn 5 https://huggingface.co/neuralmind/bert-base-portuguese-cased
Due to the limitation of the number of contest submissions, we submitted three results, the results of the test set were shown in Table 3 and Table 4. In the field of news, we respectively submitted the result of BERT (Strategy 1), the result of BERT (Strategy 3), and the fusion result of three BERT models. It can be seen that when multiple models were fused, the performance dropped instead. Among the three strategies, the strategy 3 with the worst performance on the validation set had the best performance on the test set. This was because the model used more data for training, which increased the generalization capability of the model to some extent. The result reached the best performance in this evaluation competition. However, in the field of tweets, our models were overfitting to the training data. Although the fusion strategy had brought a certain improvement, the result of our test set was still very low, the Bacc value was only 0.4975.
In the Irony Detection in the Portuguese language task in IberLeF 2021, we train three kinds of models with different strategies based on the BERT model. Experimental results as well as the leading position of our team on the task leaderboard in the field of news demonstrate the effectiveness of our method. However, in the field of tweets, our models are overfitting to the training data. In the future, we will try to solve the problem of overfitting in tweets field in order to achieve better results on the Irony Detection in the Portuguese language task.
This work was supported by the Key Field Project for Universities of Guangdong Province (No. 2019KZDZX1016), the National Natural Science Foundation of China (No. 61572145) and the National Social Science Foundation of China (No. 17CTQ045). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. 9. de Freitas, L. A., Vanin, A. A., Hogetop, D. N., Bochernitsan, M. N., and Vieira, R.: Pathways for irony detection in tweets. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing, pp. 628-633 (2014). 10. DA SILVA, F. R. A.: Detecção de ironia e sarcasmo em língua portuguesa: Uma abordagem utilizando deep learning. (2018). 11. Marten, G. S., and de Freitas, L. A.: The Construction of a Corpus for Detecting Irony and
Sarcasm in Portuguese. Brazilian Journal of Development 7(5), 47973-47984 (2021). 12. Devlin, J., Chang, M. W., Lee K., and Toutanova K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In: Proceedings of NAACLHLT 2019, pp. 4171-4186. (2019). 13. Wang, L., Lin X., and Lin, N.: Research on pseudo-label technology for multi-label classification. In: 16th International Conference on Document Analysis and Recognition, (2021). 14. Souza, F., Nogueira, R., and Lotufo, R.: BERTimbau: Pretrained BERT Models for Brazilian Portuguese. In: Brazilian Conference on Intelligent Systems, pp. 403-417. (2020).