This paper describes the system submitted to the Humor Analysis based on Human Annotation (HAHA) task at IberLEF 2021. This system achieves the winning F1 score of 0.8850 in the main task of binary classi cation (Task 1) utilizing an ensemble of a pre-trained multilingual BERT, pre-trained Spanish BERT (BETO), RoBERTa, and a naive Bayes classi er. We also achieve second place with macro F1 Scores of 0.2916 and 0.3578 in Multi-class Classi cation and Multi-label Classi cation tasks, respectively, and third place with an RMSE score of 0.6295 in the Regression task.
Humor Analysis based on Human Annotation (HAHA) 2021 [
Incorporating voting in ensembles is a machine learning algorithm. These
algorithms have been utilized in various domains ranging from Early diabetes
prediction, heart diseases prediction [
We were provided with a corpus of crowd-annotated tweets separated into three subsets: training (24,000 tweets), development (6,000 tweets), and testing (6,000 tweets).
The columns present in the corpus utilized for training and testing are as follows: { text - Text of the tweet. { is-humor - binary value (0 or 1) indicating if the tweet is humorous or not. { humor-rating - Real value (between 1 and 5) representing the average score the annotators gave to the tweet. { humor-mechanism - Label for humor mechanism. Only a subset of the tweets have the humor mechanism annotated.
{ humor-target - Zero or more labels for humor target, separated by ";". 4
This challenge3 proposes four sub-tasks which are as follows:
Humor Detection: The main aim is to classify if a tweet is humorous.
Funniness Score Prediction: Regression task which aims to rate a tweet in terms of humor.
Humor Mechanism Classi cation: A multi-class classi cation task with the primary goal of predicting the mechanism by which the tweet conveys humor.
Humor Target Classi cation: A multi-label classi cation task which aims at exploring the content of the joke based on its target. 5
We have released our code4 and experiments for easy replication. All the following models were ne-tuned using the AdamW optimizer, with a learning rate of 4e-5 and batch size of 8. These models were trained on the NVIDIA Tesla T4 GPU. The results for this task are summarized in Table 2. The baseline provided by the organizers for this task uses Naive Bayes with TFIDF features for Binary Classi cation of tweets achieving an F1 score of 0.6619 over the testing corpus.
In the nal solution, we tried a series of ensembles of pre-trained models. We use the Simple Transformers classi cation model, Classi cationModel for this task which uses a pre-trained model for this task of Binary Classi cation.
The nal model is based on hard voting in an ensemble of 5 models:-
Multilingual cased BERT (mBERT) [
While training our models on the given 24,000 tweets, we observed that BETO outperforms all other pre-trained models. We experimented with various ensembles from these pre-trained models based on hard voting. We used a 90:10 split for the training corpus without any preprocessing. 2 We have solved this problem with the technique of classi cation voting ensemble, predicting the results based on the majority vote of contributing models (preference is given to BETO and multilingual BERT with high individual F1 scores). 5.2
The results for this task are summarized in Table 3. Here the baseline is SVM with TFIDF features which achieves an RMSE of 0.6704 over the test corpus.
2 We observe that preprocessing reduces the F1 score.
In this task, we tried a series of ensembles of pre-trained models, and
results are predicted utilizing the technique of regression voting ensembles. We
combine our model with a regression head. Our ensemble comprises of 6
pretrained models:- Multilingual Base cased BERT (mBERT), ALBERT base
v2, RoBERTa base, DistilBERT base cased [
The results of task 3 are summarized in table 4. The baseline provided by the organizers for Task 3 achieves a macro F1 score of 0.1001 over the training corpus, which is based on Naive Bayes with TFIDF features.
Our model, with a Macro F1 score of 0.2916, utilizes BETO [
We use MultiLabelClassi cationModel from Simple Transformers for this task. Our nal system comprises of a pre-trained Spanish BERT cased 6 Combining BETO Cased and Uncased: The BETO model classi er outputs Softmax probabilities for all the classes. We choose the top 3 classes i.e. the classes with the highest probabilities for both the models. Next, from these 6 classes, we choose the class which appears maximum times as the nal prediction. model, which is ne-tuned for 4 epochs on approximately 2000 tweets. It took approximately 5 minutes for the complete training process per model. Various ensembles and their results are listed in the above table. 6
This paper describes the winning solution for Task 1, the second-place solution for task 3 and task 4, and the third-place solution for Task 2 in the evaluation phase of the Humor Analysis based on Human Annotation (HAHA) challenge at the Iberian Languages Evaluation Forum (IberLEF) 2021. During the development phase, our models achieved rst place in all four tasks. The combined results for both phases are mentioned in Table 4.
In all the tasks, we tried to exploit the power of voting in ensembles to get excellent results. For Task 1, 6 of our ensemble models outperform the second and third place solutions. Similarly, in other tasks, our models outperform the next place solutions by a high margin.
Further work can be done in preprocessing the Spanish tweets to analyze the e ects of various preprocessing methods on Humor prediction. An interesting approach is the translation of Spanish tweets to English and back to Spanish (i.e., Back Translation) as a method of preprocessing, which is a domain open for further experimenting and research. 7 Pre-processing includes cleaning, tokenizing, and parsing:- URLs, hashtags, mentions, reserved words (RT, FAV), emojis, and smileys. Sample preprocessor can be found at https://pypi.org/project/tweet-preprocessor/