In this article, we report the solution of the team BERT 4EVER for the Humor Analysis based on Human Annotation task in IberLeF 2021, which aims to identify humorous articles from a computational perspective. We propose the BERT model to tackle the problem. In addition, we leverage various strategies including pseudo-label technology, Task-Adaptive Pre-training and ensemble learning to improve the generalization capability. Experimental results as well as the leading position our team on the task leaderboard demonstrate the effectiveness of our method with the first place in two subtasks
Humor, a complex phenomenon in human communication that results in amusement or laughter, not only serves to interchange information or share implicit meaning, but also engages a relationship between those exposed to the funny message. However, while humor has been historically studied from a psychological, cognitive and linguistic standpoint, there have been only few attempts to create computational models for humor recognition or generation. Besides, the existing research mainly focuses on highresource languages such as Chinese and English and a characterization of humor that allows its automatic recognition and generation is far from being specified.
Luckily, HAHA @IberLEF 2021 propose the task “Humor Analysis based on
Human Annotation” [
To our best knowledge, the existing researches on humor detection are mainly focus on identifying whether the text is humor or not and humor rating.
Previous research for humor recognition is mainly based on taking the problem into
account as a classification problem. Barbieri et al. [
Although humor recognition has commonly been regarded as a binary classification
task, recent works have further toward humor detection as a relative ranking task.
Semeval-2017 Task 6 [
As regards to Spanish language, Castro et al. [
BERT. BERT [
Task-Adaptive Pre-training. Following Gururangan et al. [
MSE Strategy. According to Hinton et al. [
Five-fold Cross-validation Models Fusion. In our conducted experiment, in order to
fairly increase the robustness of the model, we leveraged 5-fold cross-validation in
which we divided all the datasets into 5 parts to obtain an ensemble model with a better
generalization performance. 4 parts of them are for training and the remaining 1 part is
for verification. Afterwards we leverage the average results of 5 cross models as an
estimation of the effectiveness of the strategy.
XGBoost. XGBoost [
LightGBM. Being a variant of XGBoost, it outperforms the XGboost model in terms
of accuracy and speed. For the shortcomings of the XGboost model, the LightGBM
[
Multiple Models Fusion. To our best knowledge, different models usually focus on different information about the same task, which will cause the difference performance on regression tasks. Based on this, in order to further increase the robustness of the model, we merge these three models with five-fold Cross-validation during the task of Funniness Score Prediction.
Although manually annotating dataset is expensive, it is relatively easy to collect
massive unlabeled data in the target domain. Hence, it becomes desired to improve the
generalization of the model by leveraging unlabeled data and limited labeled data
[
In this task, we also use the task-adaptive pretraining model in the humor detection task. We further use data to train a humor mechanism classification model 1. We use model 1 to predict the label of the unlabeled data , and keep samples with label probability greater than 0.8. Through this strategy, we obtained a total of 1940 pseudolabeled data . We merge the labeled data and the pseudo-labeled data to train a new model 2 which has the stronger generalization capability. In the final evaluation phase, we use 2 to predict the test data. 3.4
Similar to the humor mechanism classification task, we use pseudo-label technology to solve the task. However, we did not use the task-adaptive pretraining model in the humor detection task. At the same time, because humor target classification is a multilabel classification task, that is, a sample may have zero or more labels for humor target, we added a “None” label on the basis of the original label set. In addition, since many of the 4800 annotated samples have no labels, when generating the pseudo-labeled data set, we only selected the data whose prediction results contained one or more labels. The pseudo-labeled data set contains a total of 774 samples.
We use Transformers2 library using Pytorch3 as backend to construct BERT-based models and scikit-learn4 to construct machine learning models. What’s more, we leveraged BETO5 [16] as our base model. The hyperparameters are shown in Table 2. In our conducted experiment, in order to fairly explore the effectiveness of different strategies, we leveraged 5-fold cross-validation in which we divided all the datasets into 5 parts to obtain an ensemble model with a better generalization performance.
The experimental results of humor detection task are shown in Table 2. The experimental results show the effectiveness of the task-adaptive pre-training strategy and the MSE strategy. Among them, the model that uses two strategies at the same time has the best performance, and the F value is 86.45% on the final test data set.
3 https://github.com/pytorch/pytorch 4 https://github.com/scikit-learn/scikit-learn 5 https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased 6 We only used the four folds fusion as the final submission result since the fifth fold did not perform well.
The results in Table 3 show that on the funniness score prediction task, among the three base models, XGBoost has the best performance. The results on the validation set and test set are 0.6470 and 0.6615, respectively. In the end, the multiple models fusion strategy brought further improvement to the task. The result on the test set was 0.6587, ranking fourth place among all teams.
We achieved the first place in the leaderboard in both the humor mechanism classification task and the humor target classification task. Whether on validation and test sets, we can see that pseudo-label technology has brought significant improvements to the model. 5
Aiming at humor analysis task for Spanish tweets in HAHA@IberLEF 2021, we adopt a monolingual pre-trained Spanish BERT model as our base model and fine-tune it with the labeled tweets. In addition, for different tasks, we leverage different strategies to enhance the classic fine-tuned model. Experimental results demonstrate the effectiveness of our method. In the future, we will further try more strategies to achieve better results on the humor analysis task for Spanish tweets.
This work was supported by the National Social Science Foundation of China (No. 17CTQ045), the Soft Science Research Project of Guangdong Province (No.2019A101002108), the Science and Technology Program of Guangzhou (No.202002030227), the National Natural Science Foundation of China (No. 61572145) and the Key Field Project for Universities of Guangdong Province (No.
F1-test 37.20% 29.42% 42.28% 2019KZDZX1016). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. 16. Cañete, J., Chaperon, G., Fuentes, R., Ho, J., Kang, H. and Pérez, J.: Spanish Pre-Trained BERT Model and Evaluation Data. In: Preceedings of ICLR 2020. (2020).