This paper describes a system created for the HAHA 2021 shared task, framed within the IberLEF 2021 workshop [6]. We present an approach mainly based in ne-tuned and hyperparameter-optimized BERT models for binary, multi class and multi label classi cation. Our models far outperform the baselines and achieve results close to to the state-of-the-art. We also present a SHAP-values based model to explain predictions on what is humorous and what is not.
Humor research has been done historically from di erent domains such as
linguistics, history, literature and psychology. Machine learning and computational
linguistics are some tools that have been implemented on certain studies [
This article shows the process of using a BERT [7] model in Spanish to predict some of the present tasks such as humor prediction, humor mechanism and humor target. For all the tasks, ne-tuning was performed for binary, multiclass and multilabel problems. Additionally, for the binary task, a hyperparameter optimization was performed.
After predictions were performed, some explicability models were used to show the true portions of the text that was giving the humor to give us some insights on why some of them produce laughter. Most of the work in humor detection is dated before the appearance of Transformer [16] models. After this milestone, the state-of-the-art models started to be based in Transformers.
In a generic way, Sun et al. [15] propose very interesting techniques for text classi cation. However these are more focused in longer text. Some of the conclusions they achieve are that he top layer of BERT is more useful for text classi cation, within-task and in-domain further pre-training can signi cantly boost its performance and that a preceding multi-task ne-tuning is also helpful to the single-task ne-tuning, but its bene t is smaller than further pre-training.
Speci cally, for short and humorous texts, ColBERT [
For the Spanish language, we can underline the results of the 2019 HAHA shared task [5] where the best results were BERT-based models.
Some features have been evaluated to detect humor in texts. For example, in
[
The main corpus consists of 24000 texts in Spanish for training and 6000 for evaluation. For each text we predicted if it was humorous or not, the mechanism of the joke and the target or targets the joke had.
The rst task, Humor Detection consisted on determining if a tweet is a joke or not (intended humor by the author or not). The performance of this task was measured using the F1 score of the `humorous' class.
The second task consisted on Funniness Score Prediction. It consists on giving a rating from 1 to 5 for how funny is a tweet. Here we did not made a competitive model.
The third task was Humor Mechanism Classi cation, which consisted on predicting the mechanism by which the tweet conveys humor from a set of classes such as irony, wordplay or exaggeration. In this task, only one class per tweet was allowed. The performance of this task was measured using the Macro-F1 score.
The fourth and last task was Humor Target Classi cation, which consisted on predicting the target of the joke (what it is making fun of) from a set of classes such as racist jokes, sexist jokes, etc. The performance of this task was measured using the f1-macro score.
As we can see in Table 1, the dataset is unbalanced. That is the main reason why the F-measure is used as the ranking metric.
Class non-humor wordplay reference exaggeration unmasking misunderstanding absurd irony analogy embarrassment parody stereotype insults humor
In the table below, we can see some illustrative examples of the data and their labels:
We performed a simple texts preprocessing where we substituted some expressions with a more normalized form: { Every URL was replaced with the token \[URL]" so we do not get strange tokens when the tokenizer tries to process a URL. Furthermore, no semantic information about humor can be inferred from a URL, the only information relevant for the model is that there is a URL in that token. { The hashtag characters (\#") were deleted (\#example" ! \example") because the base language models we will use, are trained in generic text and might not understand their meaning. Furthermore, most of hashtags are used the same way as normal words. { We replaced every username with the generic token \[USER]" because the exact name of a user does not really add any information about the humor. The only relevant feature is knowing if someone was mentioned or not, but not who. { Finally we normalized every laugh (\jasjajajajj" ! \haha") so we minimize the noise of the misspellings, common in social networks. 4.2
The competition owners provided some baselines to compare our models with. The baselines consisted on the following models: { task 1: Naive Bayes with t df features. (0.6493 F1 over the dev corpus) { task 2: SVM regression with t df features (0.6532 RMSE over the dev corpus) { task 3: Naive Bayes with t df features (0.1038 macro-F1 over the dev corpus) { task 4: Assign label X if the tweet contains one of the "top" words for label X on the training corpus (top words were selected as the 50th to 60th most frequent words for the label) (0.0595 F1 over the dev corpus) 4.3
For our main language models we selected BETO [
In addition, for the ne-tuning process, on Task1 we carried out a Grid-search optimization over the main parameters of the neural network: learning rate, batch size and dropout rate. The search was performed with a 5-fold strati ed cross-validation with the following grid: Learning rate, (1e 6; 1e 5; 3e 5; 5e 5; 1e 4); batch size, (8; 16; 32) and dropout rate, (0:08; 0:1; 0:12). The best parameters for both models were: learning rate, 1e 5; batch size, 16 and dropout rate, 0:1.
On the task 2, 3 and 4, the default hyperparameters were used.
On task4 for nal predictions we computed the f1 metric through di erent thresholds on the validation set in order to convert the logits to classes. The threshold values evaluated were 0.2, 0.3 and 0.4. The best one on validation was 0.2 but on test the best result was on a 0.4 threshold.
Finally, the epochs performed for the ne tuning of the models were 5 for task1, 2 for task2, 7 for task3 and 8 for task4.
For explainability, shap values were calculated for each token of the sentences. Random sentences were evaluated for insights look-up. 5 5.1
We trained all the models with a NVIDIA Tesla P100-PCIE-16GB GPU and a Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz CPU with 500GB of RAM memory.
The software we used was Python3.8, transformers 4.5.1 [17], pytorch 1.8.1 [12], shap 0.39.0 [10] and scikit-learn 0.24.1 [13]. 5.2
In the Table 3, under Task1, we can see the results for our models in the test set of the rst task, where we obtained the second place.
In the Table 3, under Task2, we can see the results for our models in the test set of the second task.
For the third task, the results dropped in terms of F1, this for the di culty of a multiclass model. In the Table 3, under Task3, we can look at them in more depth.
Finally, for the task 4 results varied a lot. The baseline was very poor and the winner was very far on top of the other competitors. Our model did good compared to the baseline, but there is a long way to reach the winner. Results are shown on Table3, under Task4.
For the explicability shap model we visualized the sentences, highlighting the
important parts that led the model to make a "humor" prediction. We found 3
main cases:
{ case 1: Some jokes have a very speci c format, such as dialogues between
characters. For example on Fig 1 the "{" that characterize a dialog, is very
important on a prediction for a humor sentence. As mentioned before in the
Related Work section, Castro et al. [
Both texts are very similar, so the model over ts. { case 3: Finally there is the case that we think represents the best kind of prediction. Where the model understands relations between words and set the context as a humorous one. One example can be seen on Fig 3 Through this shared task, we have seen that NLP can be of great help in detecting and classifying humorous and non-humorous texts and there is still a long way to go. As was explained before, when analyzing which parts of the text the models use for deciding whether the text is humorous or not are based on heuristics such that whether or not the tweet represents a conversation. This shows that there is still much work to do until language models are able to understand the inherent semantics of the text so well that it can really understand the aspects of the texts, independently of the text form, that causes laughter. However, humor is an expression of high-level intelligence, expressed in sophisticated communication techniques, therefore only understanding the text meaning is probably not enough for many cases.
The results obtained by our systems are very promising given their great performance and their simplicity. Furthermore, the use of explicability models can really help get some insight on models behaviour for this kind of data. All this is very signi cant and could lead to much better results when combined with other improvements from the state-of-the-art.
We believe that our results could improve a lot using speci c language models trained with corpora from social networks like TWilBert [8] for Spanish tweets. Finally, we have proven that good hyperparameters are also key for a good neural network so a better search, like the Population Based Training, though computationally expensive, [9], would further improve the model.
This work has been partially funded by the Instituto de Ingenier a del Conocimiento (IIC) and the hardware used was also provided by the IIC.