This paper is a description of a system used to classify tweets in Spanish as humorous or not and rate the level of humor of each tweet. The system developed by the team TECHSSN uses binary classification techniques to classify the text as humor or not (subtask1) and ensemble learning regression model to rate the funniness score of the tweet (subtask2). The data undergoes preprocessing and is given to a modification of BERT [1] (Bidirectional Encoder Representations from Transformers) for the subtask1. The model is retrained, and the weights are learned for the dataset provided. XGBoost ensemble model is used to predict the funniness score on the BERT output for subtask 2. These systems were developed for the HAHA subtasks for IberLEF2021.
Humor is an experience that makes a person happy or amused. Throughout history, humans have been studying it from a psychological or linguistic perspective, but to see it through the eyes of a computer, which is basically figuring out the patterns and sequential repetitions in the textual content, is a challenging task for the field of NLP. One of the main reasons for this is the subjective nature of humor, as the humorousness of a joke depends on various factors such as age, gender, and cultural background of an individual. To make advancements in virtual assistants and chatbots, the integration of automated humor detection has become a necessity, which would make the conversations between them and human users more convenient and make their interactions look more human-like.We have participated in subtask 1 (humor detection) and subtask 2 (funniness score prediction).
Humor is a well-studied topic in the fields of psychology and linguistics, but in the field of computer science continuous research has been going on and the humorrecognition systems are getting better every year for us to get a better understanding of the factors that makes a conversation humorous.
Mihalcea, R., and Strapparava, C. in their work Making Computers Laugh (2005)
[
UO UPV system was developed for the Humor Analysis based on Human
Annotation (HAHA) track proposed in IberEval 2018 [
Santiago Castro et. al. [
In the HAHA task of IberLEF2019, Chiruzzo et. al. [
Orion Weller and Kevin Seppi [
Omar Khattab and Matei Zaharia [
There are other papers which describe systems that detect humor in non-English
text, like Ismailov A. et. al. [
For the classification of text, we have chosen the BERT-Base multilingual model which has 12 layers with the last layer’s activation function as the sigmoid function, as we are performing binary classification. 3.1
The classification model uses a separate line of hidden layers especially designed to extract features from each sentence. The used model is a neural network that includes two parallel lines of hidden layers: One to view text as a whole and another one to view each sentence separately. Figure 1 displays the architecture of the proposed method. It is comprised of a few general steps: 1. The sentences are separated and are tokenized individually, to analyse each sentence separately. 2. To convert the text to proper numerical inputs for the neural network, they are encoded using BERT sentence embedding. This step is performed individually on each sentence and on the whole text (shown in Figure 1). 3. The resultant BERT sentence embeddings for each sentence that we get from the previous step are then given as an input to the partial hidden layers of the neural network, whose purpose is to extract mid-level features for each sentence (could be related to context, type of sentence, etc). 4. While our main idea is to detect relationships between sentences (especially with punchline), it is also required to examine word-level connections in the whole text (such as synonyms and antonyms) that may have meaningful impacts in determining congruity of the text. Like the previous step, we feed BERT sentence embeddings for the whole text into hidden layers of the neural network. 5. Finally, three sequential layers of the neural network conclude our model. These final layers combine the output of all previous lines of hidden layers to result in the final output. In theory, these final layers should determine the congruity of sentences and detect the transformation of reader’s viewpoint after reading the punchline. 6. For predicting the humor level (funniness score) of a tweet we factor in the votes (votes_no to votes5) instead of the binary labels, which would give us a 5dimensional vector as a result, which is given as an input to a XGBoost Regression Model to predict the humor rating of the given tweet. 3.2
The dataset used for training our model is the one provided in the Codalab competition page, Training Dataset (haha_2021_train.csv). ─ Id - tweetId ─ text - tweet ─ is_humor – 0 or 1 ─ votes_no, votes_1...votes_5 – 0-1 ─ humor_rating – 1-5 humor_mechanism - {absurd, analogy, embarrassment, exaggeration, insults, irony, misunderstanding, parody, reference, stereotype, unmasking} ─ humor_target – {age, body shaming, ethnicity/origin, family/relationships, health, lgbt, men, professions, religion, self-deprecating, sexual aggressors, social status, substance use, technology, women} For the pre-processing the data is tokenized using the BERT Tokenizer (pre-trained on the BERT-Base multilingual model) and then it undergoes stemming (SnowballStemmer) and lemmatizing (WordNetLemmatizer). The tokenized input is encoded into ids, masks, and segments, for the transformer (BERT) to accept it as an input. For the model we have chosen BERT-Base multilingual, which is pre-trained in English language. This model is compared against some of the existing techniques such as Support Vector Machine (SVM), Decision Trees (DT) and Multinomial Naïve bayes (MNB). For training of the model, we loop over the folds in gkf (Group K-Fold) and train each fold for 3 epochs with a learning rate of 3e-5 and a batch size of 6.As we have performed binary classification for the humor detection task, we have set the loss function as a simple binary cross-entropy, and we have chosen Adam as the optimization function. For the second task the loss function is a Mean Squared Error (MSE) function, and the optimization function remains the same. 4
As mentioned in section 3.4, the performance of our model (BERT) was tested against various machine learning techniques (SVM, DT, MNB). These models were tested on the test set with gold labels provided in the Codalab's competition page.
Table 2 shows the results for the various models used for the subtask 1. This table we can infer that there is a trend, where all the models show high precision and low recall. The model has a very low false positive rate and an average false negative rate, which is illustrated in the Figure 2. And the same trend is seen amongst other models, which means that classifying a text as non-humorous is harder as compared to classifying it as humorous. Figure 2 shows the confusion matrix formed for the BERT model. Humor detecting systems in Spanish with high accuracy can help serve to the Spanish audience on various social media platforms. It can be used to make interaction with chat-bots and virtual assistants affable. The HAHA subtask 1 and 2 for IberLEF2021 involves classifying tweets in Spanish as humorous or not and rate their humor level on a particular scale. We used a model that is built on top of BERT which is used to classify such sentences (text) into humorous or not. XGBoost regression model is used to predict the humor level or funniness score in the tweet.