Giving computers basic notions of humour can result in better and more emphatic user interfaces that are perceived more natural. In addition, hate-speech systems that understand humor are more reliable, as humor can be used as a Troy Horse to introduce oppressive-speech passing them o as harmless jokes. Understanding humor, however, is challenging because it is subjective as well as cultural and background dependant. Sharpen forms of humor, moreover, rely on gurative language, in which words loss their literally meaning. Therefore, humor detection has been proposed as shared-task in workshops in the last years. In this paper we describe our participation in the HAHA'2021 shared task, regarding ne-grain humor identi cation in Spanish. Our proposals to solve these subtasks are grounded on the combination of linguistic features and transformers. We achieved the 1st position for humor rating Funniness Score Prediction with a RMSE of 0.6226, the 8th position for humor classi cation subtask with an 85.44 F1-score of humours category, and the 7th and the 3rd position for the subtasks of humor mechanism and target classi cation with an macro averaged F1-score of 20.31 and 32.25 respectively.
According to the Merriam Webster dictionary, humor can be de ned as a mental faculty of discovering, expressing, or appreciating the ludicrous or absurdly incongruous. Therefore, humor requires sharp mental abilities that are developed starting in the childhood [13]. In addition to social and cognitive skills, other factors that make identifying humor challenging include subjectivity, as what it is fun for some people may not be fun for others, and cultural background, that in uences how humor is perceived and expressed [6]. Moreover, in written communication humor can appear in many forms: from light-heart humor, such as anecdotes told with wit, to bitter forms such as sarcasm. To make humor identication even more di cult, humorists play with gurative language, making use of complex linguistic phenomena such as sharp analogies or double entendres. Figurative language, and its relationship with humor, has been a productive area of research as regards tasks such as satire identi cation [11, 12].
In this manuscript we describe our participation in the shared task HAHA Humor Analysis based on Human Annotation (HAHA 2021) [2] from IberLEF (Iberian Languages Evaluation Forum) [9]. From a computational linguistic point of view, humor have been studied in recent years [8], but these studies mainly focused on their identi cation. As far as we know, the most complete work regarding humor identi cation and categorisation was the shared task HaHaCkathon 2021 [7], focused on English. 2
The evaluation campaign of HAHA proposed four subtasks to the participants
regarding humor detection and categorisation. These subtasks are the following:
(
For the rst two subtasks, the organisers provided a gold-standard corpus
divided into three splits: (
We calculate the correlation of the humor mechanism and humor target1 and
we observe that there is a strong correlation between (
Our methodology is grounded on the combination of linguistic features and transformers by mean of neural networks. Our pipeline includes a pre-processing stage for cleaning the texts, a feature extraction for the linguistic features and the transformers, a selection stage for selecting the best features and, nally, a hyperparameter evaluation stage for tuning the models. For text pre-processing we transform the documents into its lowercase form, we remove hyperlinks, emojis, quotations, mentions, and hashtags. We also replace digits with a xed token, and we x misspellings and remove word elongations. Next, both the pre-processed and the original version of the text are used to obtain the features. The cleaned version is used for obtaining the sentence embeddings and the majority of the linguistic features. The original version of the documents, on the other hand, are employed for extracting linguistic features related to misspellings, correction and writing style, and the percentage of uppercase letters. Next, we extract sentence embeddings from Spanish fastText [5] (SE) and transformers based on Spanish BERT (BF) [1]. For fastText, we use the their online tool for extracting xed vectors of 300. For transformers, we ne-tune the model for each subtask, and then obtained a vector of length 768 from the [CLS] token.
It is worth mentioning that during our experimentation, we also evaluated Convolutional and Recurrent Neural Networks because they achieved competitive results in classi cation tasks such as Sentiment Analysis [10]. Our results with a validation split, however, indicate the accuracy and results achieved by convolutional and neural networks were similar to the ones obtained with xedvectors, that are several orders of magnitude faster to train.
We use UMUTextStats [3, 4] for extracting a total of 365 features related to stylometry, phonetics, morphosyntax, discourse markers, gurative language, lexical, psycho-linguistic processes, register, and to detect patterns commonly used in social networks. The linguistic features are scaled using a MinMax strategy. Next, we apply a feature selection based on mutual information. As we faced four challenges, we perform this process four times: Mutual Information for subtasks 1, 3, and 4 and univariate linear regression test for subtask 2. Next, we perform a hyper-parameter tuning for evaluating di erent combinations of neural networks and select the best one for each subtask based on the main metric. We evaluate 125 neural network models for each feature set fLF, SE, and BFg and for the combinations of features fLF, SE, BFg and fLF, BFg. The hyperparameters include the number of hidden layers and their number of neurons organised in several shapes, the dropout rate, some activation functions, and di erent learning rates. Neural networks are provided with an early stopping mechanism, a learning rate scheduler, and the initial weights of the neural network in order to reduce the e ect of class imbalance. Source code is available at https://github.com/Smolky/haha-2021. 4
The competition was divided into development and evaluation and the organisers provided a split for each stage. However, as the labels with the grounding truth of the development split were not released, we generate a custom validation split dividing the training set in a ratio of 80% for training and the remaining 20% for validating. Table 1 includes the o cial leader board to compare our results with the rest of the participants. Our team achieved positions 8th, 1st, 6th, and 3rd for subtasks 1, 2, 3 and 4 respectively.
Regarding the rst subtask, humor detection, all participants outperform the baseline (Naive Bayes classi er based on TFIDF features). Our team achieve a F1-score over the humor class of 85.44, reaching position 8th, only a 3.06 below the best result (Jocoso, F1-score of 88.50). For this subtask we train a neural network with LF and BF. Each feature set is connected to a shallow neural network composed of 2 hidden layers and 8 neurons per layer. We use a dropout of 0.3, a linear activation function, and a learning rate of 0.01. For understanding the relevance of the linguistic features, we obtain the mutual information2 and observe that the most relevant features are related to number of questions and the length of the corpus. We also identify as relevant the number of personal pronouns and main verbs.
Regarding the second subtask, humor scoring, we achieve 1st position. For
this subtask, only 12 of the 16 participants sent runs. Our results, 0.6226 of
RMSE are similar to second (ColBERT, 0.6246) and third (Jocoso, 0.6296) best
results. For this subtask we combine LF, SE, and BF in a neural network
composed by 6 hidden layers in a long funnel shape, with the rst hidden layer
composed by 47 neurons. We use a dropout of 0.2 and a learning rate of 0.01.
The activation function is tanh. We achieve bad results in our rst and second
runs, obtaining an RMSE score of 1.19713 and 1.21332 respectively. To improve
our results, we replace the loss function from MSE to RMSE and we retrain the
neural network with our custom validation set for 10 more epochs. As a
additional strategy, we observe than our neural network sometimes outputs values
lower than 0. As we knew the score could not be negative, we convert them to 0.
Moreover, in order to understand how LF performs in this subtask, we calculate
Mutual Information Gain. Figure 1 contains the values of the 20 top rated LF.
It is worth mentioning that subtask 2 contains only the documents rated as
humor and, therefore, the LF are used to discern how funny and which agreement
have the documents among the annotators. We can observe that there are
linguistic features from several categories, such as stylometry (STY), highlight the
number of sentences that are questions. Regarding morphosyntax (MOR), there
are features related to the (
For the third subtask, humor mechanism classi cation, a total of ten users participated. We achieve the 6th position in the o cial leader board with a macro F1-score of 20.87. We improve the baseline based on Naive Bayes classi er with TF-IDF features and a macro F1-score of 10.01. The best result is for BERTForever, with an macro F1-score of 33.96. Similar to subtask 1, our neural network consists into a shallow neural network that connects separately LF and BF to a hidden layer composed of 16 neurons. We use a dropout of 0.2 and a learning rate of 0.001, with a sigmoid as activation function. To handle class imbalance, we include in the train split 400 documents from the non-humor class.
We calculate the confusion matrix over the o cial evaluation split3. We observe that analogy, misunderstanding, and wordplay are the classes with the most hits. Embarrassment, on the other hand, is confused with unmasking. A similar problem can be found between stereotype and wordplay. Another humor mechanism with poor performance is parody, wrongly classi ed as reference, stereotype, and wordplay. Irony is worst performing class. Only 9% of ironic documents were successfully classi ed. Moreover, ironic documents are wrongly classi ed uniformly among the rest of the humor mechanisms. Irony consists on suddenly shifting the expected outcome of events. This nding suggests that neither linguistic features nor transformers are able to catch words that deviates from their conventional meaning.
Finally, for the fourth subtask, humor target classi cation, a total of nine participants sent their results. As our team does not have much experience dealing with multi-label classi cations, our approach consists into transforming the problem in a multi classi cation task. We consider this strategy as we observed that not much of the documents were labelled with more that one tag. Despite the simplicity of our approach, we reached 3rd position with an F1-score of 32.25. In this case, our best neural network consist into a shallow neural network that linked separately each feature set (LF, SE, and BF) to hidden layers of 256 neurons. In this case, we use tanh as activation function, a dropout of 0.3, and a learning rate of 0.001.
0.00 0.01 0.02 Fig. 1. Mutual Information of linguistic features concerning subtask 2: humor scoring fam
aeg isyoanhgdbm iiii/ttrcyoennhg iilli/ftrssyoaaehnpm ltaehh ltgb enm ifrsssooenp iliroeng ilf-trscaeeengpd lrrssssxoaaeeugg littssscoaau tssscaeeunub ltcyooenhg oenwm This manuscript contains the description of the participation of UMUTeam at HAHA 2021 shared task regarding ne-grain humor identi cation in Spanish. We are happy with the results achieved in this shared task, as we consider that we are improving our methods and results. As promising research directions, we will improve the linguistic features and we will analyse in which cases the linguistic features and the transformers does not agree in the nal prediction, in order to get major insights of the strengths and weaknesses of each method. We also considered interesting to evaluate data augmentation for improving the accuracy of the model. In case of humor this is challenging as it is possible to include more examples during training with translated jokes from other languages. However, as humor is background and cultural dependant, this should be analysed with caution. 6
This work was supported by the Spanish National Research Agency (AEI) through project LaTe4PSP (PID2019-107652RB-I00/AEI/10.13039/501100011033). In addition, Jose Antonio Garc a-D az has been supported by Banco Santander and University of Murcia through the industrial doctorate programme.