This paper describes the winning solution to the Humor Analysis based on Human Annotation (HAHA) task at IberLEF 2019. The main classi cation task is solved using an ensemble of a ne-tuned multilingual BERT (Bidirectional Encoder Representations from Transformers) model and a naive Bayes classi er. Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). IberLEF 2019, 24 September 2019, Bilbao, Spain.
Humor Analysis based on Human Annotation (HAHA) challenge [
The main classi cation task is solved using an ensemble of a ne-tuned
multilingual BERT (Bidirectional Encoder Representations from Transformers [
The regression task is also solved by ne-tuning a multilingual BERT model. The nal submission is a weighted average of the regression BERT model and a LightGBM model (https://github.com/microsoft/LightGBM) estimated on TFIDF features. The solution achieves the root-mean-square error (RMSE) score of 0.736 with the second-place team achieving the RMSE of 0.746.
The rest of the paper is organised as follows. Section 2 describes the challenge
and Section 3 describes the solutions for each of the tasks. Section 4 concludes.
The challenge asks to classify tweets in Spanish as humorous or not, and rate
how funny they are on a scale from one (not humorous) to ve. The dataset [
For every model the nal predictions were obtained by averaging predictions from each fold of a ve-fold cross-validation schemes, in other words by averaging predictions from ve models each estimated using 80% of the data. The validation scores reported below are F1 scores and RMSEs calculated on out-offold predictions for the full train set. 3.1
Classi cation task The results are summarised in Table 1. The baseline provided by the organisers considers a tweet a joke randomly with a probability of 0.5.
The main model used in the nal solution is ne-tuned multilingual cased
BERT model (12-layer, 768-hidden, 12-heads, 110M parameters) [
For the classi cation task we use binary cross-entropy loss and one-cycle
learning rate schedule [
To further reduce over tting we ne-tune the pretrained weights by using
unsupervised learning and text data from both train and test sets. The idea
comes from the ULMFiT paper [
We then average these predictions with predictions from a naive Bayes model
following [
The model alone achieves only 0.771 on cross-validation, however an ensemble of the BERT model above and the naive Bayes model with weights 0.72 and 0.28 respectively scores 0.833 on cross validation and 0.821 on the leaderboard. The optimal weights are derived from cross-validation. This ensemble was the nal solution for the classi cation task. The results are summarised in Table 2. Here the baseline is a constant prediction of 3 assigned to all items in the test. For the regression task we also ensemble two models: a ne-tuned BERT model and a LightGBM model based on gradientboosted trees.
The only di erence between the BERT model for the regression task and the BERT model for the classi cation task is the loss function - for former we use the mean-squared loss given that the challenge metric is RMSE. This model achieves RMSE of 0.726 on cross-validation and 0.746 on the test set. 3 Based on the code nb-svm-strong-linear-baseline from
https://www.kaggle.com/jhoward/
The second model is a LightGBM model estimated on the same features as the naive Bayes model above. The loss function is also mean-squared error and we set bagging and feature fraction parameters to 0.7 to add regularisation. This model scores 0.795 on cross validation.
For the nal solution we average predictions from these two models linearly with weight of 0.71 assigned to the prediction from the BERT model and 0.29 - to the LightGBM model, with weights coming from the cross-validation. This ensemble scores 0.712 on cross-validation and 0.736 on the test set. This paper describes the winning solution for both classi cation and regression tasks of the Humor Analysis based on Human Annotation challenge at IberLEF 2019, which consists of an ensemble of a ne-tuned BERT model and a complementary model estimated on TFIDF features derived from uni- and bi-grams.
Firstly, we can see that a high score can be achieved by solely appropriately ne-tuning a BERT model. In fact that model alone would rank second in this competition, and with longer hyperparameter search or longer language model ne-tuning - possibly rank rst.
Secondly, unsupervised learning can help to reduce over tting and improve the score signi cantly when the size of the dataset is small.
And, nally, adding weaker but di erent and diverse models to the ensemble helps to boost the score as has been demonstrated above.
Further work can be done both in improving the quality of the predictions and studying how the models assign them. For instance it would be interesting to examine predictions from di erent layers of the neural network to study where does attention fall and what token combinations make the model decide whether a tweet is humorous or not.