This paper describes in detail the four subtasks of HAHA@ IberLEF 2021 [7] : Humor Analysis based on Human Annotation. Subtask 2 is a regression problem, and the other three subtasks are all text classi cation problems. The data comes from the Twitter social platform, and the language is Spanish. The classi cation problem is mainly solved by integrating the Multilingual Bert model and the LSTM model, and the regression problem is solved by the GPT-2 Model. According to the o cial evaluation results, the result of the method proposed in this paper ranks fourth, eighth, fth, and sixth on the four subtasks, respectively. For this task, I have uploaded the code to GitHub kuiyongyi for easy reference by others.
Humor is a very common phenomenon in human communication. It is relatively
easy for humans to understand whether the content of a text is humorous, but
the computer can learn the characteristic information in the text, only after
learning a large amount of corpus and then detect whether the text is humorous
[
Humor detection has been a relatively hot eld for many years.
Semeval2015 Task 11 proposes the in uence of gurative language such as metaphor
and irony on sentiment analysis. Semeval-2017 Task 6 requires predicting the
ranking of comedy shows based on humorous tweets of comedy shows. Both
IberEVAL 2018 and IberLEF 2019 include two subtasks: humor detection and
interest score prediction. Castro [
The rest of the paper is organized as follows. In the second part of the paper, I give an overview of the task and the data. The third part describes the model used in this task. In the fourth part, the experiments of the four subtasks are described. The fth part gives the test results of the model. Finally, the sixth part of the article is the conclusion. 2 2.1
HAHA@IberLEF2021 includes four sub-tasks, which aim to predict the humor classi cation or humor degree of the text. The four subtasks are de ned as follows:
Subtask 1: Given the Twitter text in Spanish, the goal is to predict whether each tweet is humorous. This is a binary classi cation problem. This question is measured by F1 score.
Subtask 2: To measure the degree of humor on the text predicted to be humorous, this is a regression task. The evaluation standard of this subtask is the root mean square error.
Subtask 3: It is required to predict a humor mechanism category for tweets among the twelve humor mechanisms given. This is a multi-classi cation problem. The performance of this task will be measured using the Macro-F1 score.
Subtask 4: Given tweets and fteen humorous target tags, the goal is to predict the corresponding humorous target tags for each tweets (at least zero and at most 15 tags), which is a text multi-label classi cation task. The evaluation standard for this question is also Macro-F1. 2.2
The Dataset [
Each record in haha 2021 train consists of twelve columns of data, which are respectively Id, the tweets corresponding to each Id (text includes punctuation and emoticons), non-humorous voting, the number of votes for each of the ve levels, is humor, humor rating, humor mechanism, and humor target. 2.3
Since the text in the HAHA Dataset comes from Twitter social media, the length of the text varies, most of the text is generally short, and the content or format of the tweets is informal, (including spelling errors, character, and vocabulary repetition). It is necessary to clean up the tweets in the Dataset, because in this way the Language Model can predict the content to be covered based on the context and semantics. The data cleaning methods involved in this paper are as follows: { Replace the repeated characters or words (three times or more) in the tweet with a single character or word; { Delete Delete the emoticons appearing in the text; { Delete Delete the HTML tags that appear in the text; { Delete Replace the newline character in the text with spaces. 3
In this section, I will give an overview of the system that implements these four subtasks. We can simply regard the task of Humor Analysis based on Human Annotation in IberLEF 2021 as text classi cation and regression problems, given input text, predict one or more labels, and predict the humor score value of the text.
One disadvantage of the LSTM model is that the input information is compressed into a vector with the same length as the number of LSTM memory units, so the LSTM model cannot remember long tweets. However, the Twitter text length of this task is very short, so I consider using LSTM to further extract text feature information after the output of the Pre-training Model. 3.1
Since the text of the Dataset is Spanish, this paper uses three Cross-language
[
The Bert model uses Masked Language Modeling [
Albert uses Transformer and GELU activation function. Albert uses such as parameter sharing and matrix decomposition to reduce the number of model parameters. Albert uses Sentence Order Prediction Loss to replace Next Sentence Prediction Loss.
XLM is a Cross-language model, similar to Bert, it is also a Masked Language Model, and its Pre-training method is the next token prediction. In this task, XLM-mlm-tlm-xnli15-1024 is used. The model uses fteen corpus including Spanish for Cross-language sentence training.
XLM-Roberta is a model trained on a corpus of one hundred di erent languages. Unlike XLM, it doesn't require lang tensors to understand which language is used and should be able to determine the correct language from the input ids. In this task, XLM-RoBerta-base is used.
XLNet mainly optimized the Bert model in three aspects. First, the
Autoencoding Model [
The GPT-2 model is usually padding the inputs on the right and was trained using causal language modeling targets. The GPT-2 model and the Bert model are constructed through the decoder and encoder modules of the transformer, respectively. 3.2
In subtask 1, the optimizer chooses AdamW, train for 50 epochs with a 3e-5 learning rate and a 32 batch size. And the weight decay parameter is set to 1e-2, the maxlen parameter value is set to 64, and the loss function is selected as Crossentropy.
In subtask 2, the loss function used is mse loss, the learning rate is set to 1e-5, the maxlen parameter value is set to 64, batch size is set to 32, and after every sixty-four steps of training, a veri cation is carried out.
In subtask 3 and subtask 4, the learning rate and weight decay parameters are both 5e-6 and 1e-2, the epoch is 50 and 100 respectively, the loss function uses CategoricalCrossentropy and BinaryCrossentropy respectively, both the drop out parameter and the optimizer are set to 0.5 and Adam. 4 4.1
Divide haha 2021 train according to the ratio of 4/1 as the Training Dataset and Validation Dataset of the models in subtask 1. The actual Training Dataset, Validation Dataset, and Testing Dataset lengths of subtask 1 are 19200, 4800, and 6000 respectively. Next, the data is cleaned, word segmented, and coded before it is ready to be input into the model.
In subtask 1, First, I used three Pre-trained Models (Albert-base-v2, XLNetbase-cased and Bert-base-multilingual-uncased) for text binary classi cation. The results show that the Multilingual Bert model is used to obtain the highest score in the text classi cation problem of Spanish materials. The F1 score of this model on the Validation Dataset is 0.8712, and then add a 4-layer unidirectional LSTM network after the model to further extract text features, and nally input the LSTM results into the fully connected layer for classi cation. This combination has an F1 score of 0.8785 on the Validation Dataset, compared to a single pre-trained model, the score increased by 0.0073. Thus, the prediction result of this combined model is the answer I nally submitted on subtask 1. The performance of these models on the Validation Dataset is shown in Table 1. 4.2
There are 9253 pieces of data available for subtask 2 in the haha 2021 train. Divide the data according to the method of subtask 1. The Training Dataset length of subtask 2 is 7402, the Validation Dataset length is 1851, and the Testing Dataset length is 6000.
In subtask 2, I compared the performance of three Cross-language Pretraining Models and the GPT-2 model in the regression problem. These four Pre-training Models are Bert-base-multilingual-uncased, XLM-mlm-xnli15-1024, XLM-RoBerta-base, and GPT-2. The experimental results show that the performance of the GPT-2 model in subtask2 is slightly better than the other three models, therefore, I use the prediction result of the GPT-2 model as the answer I nally submitted in subtask 2. The speci c performance of these four models on the Validation Dataset is shown in Table 2. 4.3
In haha 2021 train, humor mechanism column and humour target column have 4800 and 1629 non-empty data respectively. Divide the data according to the method of subtask 1. The Training Dataset lengths of subtask 3 and subtask 4 Bert-base-multilingual-uncased Bert-base-multilingual-uncased+BiLSTM are 3840 and 1303, respectively, the Validation Dataset lengths are 960 and 326, respectively, and, the length of the test Dataset for both is 6000.
After dividing the data, rst, encode the labels of the Training Dataset and
Validation Dataset in subtask 3 and subtask 4 into one hot vectors. Next, the
Dataset is processed in the same way as subtask 1. Then, use the sequential
function to build a double-layer BiLSTM [
The results of subtask 1 show that the Multilingual Bert model performs slightly better than Albert and XLNet in the Spanish Twitter text classi cation task. So in subtask 3 and subtask 4, I use the multilingual Bert model as the basis. First, input the processed data into the Multilingual Bert model; next, input the result of the Bert model into the two-layer BiLSTM network to further extract features; nally, input the result of the LSTM network into the fully connected layer (The number of neurons in the fully connected layer of subtasks 3 and 4 are twelve and fteen respectively) for classi cation. Subtask 3 nally outputs a twelve-dimensional vector, taking the element with the largest value upwards to 1, and setting the remaining elements to 0 (The 1 and 0 in subtask 3 and 4 respectively indicate that it is predicted to be or not predicted to be a certain category). The output of subtask 4 is a fteen-dimensional vector, and the threshold is set to 0.5. For each element in the output vector, if it is greater than the threshold, it is set to 1 upwards, otherwise, it is set to 0 downwards. I used the prediction results of the combined model of Multilingual Bert and BiLSTM as the answers I nally submitted on the last two subtasks. The performance of my solution on veri cation sets of subtask 3 and subtask 4 is shown in Table 3. 5
Among all the teams participating in this task, the result of my solution ranked fourth, eighth, fth, and sixth among the four subtasks. Table 4 lists the scores of the best performing teams in the four subtasks of the IberLEF 2021 HaHa competition, and the scores of my method on the o cial Testing Dataset. This paper describes the data processing, the comparison of Pre-trained Models, and the nal model construction in the HAHA@ IberLEF2021 challenge. Although the solution I proposed has achieved good results, it is undeniable that there is still a lot of room for improvement. Due to time constraints, I can not do error analysis at present. In my future work, I will conduct a detailed error analysis to understand the limitations of the program.
In future work, rst of all, we can try to extract the emoticon information
in tweets instead of deleting it directly. If both the emoticons and the text
information can be extracted, it will promote humor classi cation. Secondly,
during the experiment, I found that the model had over tting problems. For
this reason, I will try to use transfer learning [