Our team (the team-name is qu) participate in the classi cation task of EmoEvalEs@IberLEF 2021. The task requires to classify the emotion of the Spanish tweets (anger, disgust, fear, joy, sadness, surprise, others). To solve the problem, we propose a model (XLM-RoBERTa combined with Bidirectional Gated Recurrent Units). In our model, the accuracy is 0.449879 and the averaged precision is 0.618833. In the nal evaluation results, our team ranked 15th.
With the development of society and technology, the number of users on the
network has increased sharply, and communication on Internet has become an
important way of communication in modern society [
Since most of the tweets' messages do not have intonation and expression.
So it is a di cult task to analyze the emotion of users on social media. To
encourage the development of this eld, EmoEvalEs@IberLEF 2021 [
This article uses a model combining XLM-RoBERTa [
In recent years, in the elds of NLP (Natural Language Processing) and DM (Data Mining), researches about emotion analysis have always existed. Scholars at home and abroad are mainly doing researches and learning from emotional dictionaries, Machine Learning, and Deep Learning. The keys of the analysis method based on the emotion dictionary are to use the emotional dictionary and to judge the users' emotional tendency through a series of calculations (semantic correlation). This method requires to extract the features manually. For "Big Data", the method can no longer meet daily needs.
To make up for the shortcomings of emotional dictionary, the papers
compared SVM (Support Vector Machines), CRF (Conditional Random Field) and
other methods [
For the shortcomings of Machine Learning, the methods based on Deep
Learning can solve it well, and it has been proved that Deep Learning is more
prominent in emotion analysis than Machine Learning. Kim [
Zhu et al. [
In this mission, we use the o cial data set provided by EmoEvalEs@IberLEF
2021 [
In order to choose a better model, we get the scores of three di erent models on the development set under the same conditions. As shown in Table 1 below.
In the Table 1, we observe that XLM-R combined with Bi-GRU model has more advantages than other models. LSTM network or GRU network can effectively deal with the long-range dependence problem of length sequences, and are widely used in natural language processing. However, the two networks only consider the correlation between sequences in one direction, and do not consider the possible correlation of future information, so they have great limitations in the application of strong round-trip correlation.
This paper uses the XLM-R model to extract the features of the training
set and then inputs the acquired features into the Bi-GRU model to extract the
emotional features of comments. Finally, we classify the emotion tendency by
the Softmax method [
In the GRU, there are only two gates: update gate and reset gate. The facts have proved that compared with LSTM, GRU reduces the parameters and improves the e ciency of the model. The speci c structure is shown in Figure 2.
In Figure 2, the Zt represents the update gate and the rt represents the reset gate. The Zt can determine the renewal of the cell state and the rt decides to write some information into eht. The smaller the number of rt, the less information to be written into the candidate set, which means that the amount of previously useful information is less. Zt is the update gate, which determines to update the previous information or not. It can discard useless information to avoid the problem of long-term dependence. Zt is the logic gate when the model renews the activation. The equation is as follows:
Zt = (!z [ht 1; xt]) (1)
In the Equation 1, is the activation function (Sigmoid) and the ! is the weight matrix.
The rt is the reset gate, which determines whether the previous information needs to be reset. The rt decides whether to abandon the previous activation ht when the candidate is activated. The equations is as follows: rt = (!r [ht 1; xt]) (2)
In the Equation 3, the tanh is the activation function and the ht is the hidden layer. The calculation formula for receiving [ht 1 ,ht ] in the calculation process is as follows: ht = (1
zt) ht 1 + zt eht
In the model (XLM-R combined with Bi-GRU), the XLM-R layer is the embedding layer of the Bi-GRU layer, and the Bi-GRU layer is the hidden layer. The input sequence is embedded in the model from two directions and the twoway information is stored. Last the information (ht(i)) is nally output. ht(i) represents the bidirectional GRU information of the i-th text, and !ht represents the forward GRU information of the i-th text. ht represents the reverse GRU information of the i-th text. The equations are as follows: eht = tanh(! [rt ht 1; xt]) (3) (4) (5) (6) !ht = GRU (xt; ht!1) ht = GRU (xt; ht 1) Transformer encoder, we can get the vector of the tweet text (T1, T2, . . . . . . , Tn 1, Tn). The vector obtained through the XLM-R model training is expressed as: Wi = (w1(i); w2(i); w3(i); : : : ::; wn(i) 1; wn(i)), where the Wi is the vector-matrix of the i-th sentence. The Wj(i) is the feature of each word, and the n is the maximum length of the sentence.
As shown in Figure 1, the entire XLM-R combined with Bi-GRU model is mainly divided into 5 layers. The rst layer is the input layer. We input the o cially provided data into the model; The second layer, we use the XLMR model to vectorize the tweets; The third layer, we input the obtained word features into the Bi-GRU network to realize the emotion analysis of the text; The fourth layer, we use the function (Softmax) to classify the features; The fth layer, the model output the emotion labels of the tweets (anger, disgust, fear, joy, sadness, surprise, others). 4 4.1
The data come from the o cial competition. We nd that the data are based on events related to entertainment, disasters, politics, commemoration and global strikes that occurred in April 2019. Since the data set comes from tweets, the content of the data are not formal texts. So we need to process the data before training and testing. (1) Use "HASHTAG" to replace the tags in the data. (2) Because the username has no e ect on emotion analysis and may a ect the judgment of the model, a uni ed standard is adopted and "#@USER" is used instead of the username. (3) There are emoticons in the text. Replace all emoticons with text. (4) Uniform case. 4.2
The experimental equipments used in this experiment are as follows: Intel CORE i7 CPU (16G), Hard Disk 1T, NVIDIA RTX 3080Ti. The operating system is Windows 10. The tool of editor is the Pychrom2020. The framework of Deep Learning is PyTorch. The optimizer used is Adam, and the learning rate is 2e-5. The speci c parameters of the model are shown in Table 2. The evaluation indexes for the Spanish emotion multi-classi cation model are F1 score, averaged recall, averaged precision and accuracy. The partial results in this competition are shown in Table 3 below.
The team GSI-UPM win the rst place in the competition. However, our team qu only win the 15th place in this competition. In the competition subtask 3, we not achieve the ideal results. We think there may be three main reasons for this result: (1) The value of the hyper-parameter is unreasonable. The value of the epoch is too large and it can increase the number of iterations of the weights in the network, and cause the model to over- t. (2) The distribution of training data is uneven. The distribution of the data amount of each label in the training data is too uneven, resulting in poor performance of the trained model. Because the linear classi er used in this model is biased to most classes, it causes the deviation of the model. (3) The training set has only 5723 pieces of data. So the generalization ability of the model is poor. 5
In this paper, we mainly propose a model (XLM-R combined with Bi-GRU) for emotion detection of Spanish text (tweets). The performance of this model on the development set is good, but the performance on the test data set is not satisfactory. Therefore, I think there is still a huge room for improvement in the performance of the model. In the next work, I will reset the hyper-parameters and use the K-fold ensemble method to improve the generalization ability of the model.
The completion of the paper is attributed to many people's support. First and foremost, I want to extend my heartfelt gratitude to my supervisor, Yanhua Yang. He gives me much help and advice during the whole process of my writing. My thanks also go to the authors whose books and articles have given me inspiration in my writing of the paper. Last but not least, I would like to thank the organizers for their hard work.