This article describes HAHATeam participation in the IberLEF 2021 EmoEvalEs task: Emotion detection and Evaluation for Spanish. In this task, we use Masked Language Model-based data augmentation to enhance the data to increase the training data set and prevent over tting. We improve the performance of the model through a series of data processing and data augmentation technologies. We use three di erent cross-language models, BERT, XLM, and XLM-RoBERTa for comparative experiments. Very competitive results have been achieved on both the development set and the test set. The best model achieved the third place in the development set and the fth place in the test set.
Sentiment analysis (SA) is the process of analyzing, processing and categorizing
subjective text. According to the emotional tendency of the text, the text can be
classi ed. In addition to the traditionally recognized positive and negative
emotions, there should also be anger, disgust, fear, joy, sadness, surprise and other
emotion types. This article describes the emotion classi cation task (EmoEvalEs
[
The earliest work on SA largely relied on feature engineering [
Recently, pre-trained language models, such as ELMo [
In order to solve this task, we proposed the use of pretrained classic deep learning models to apply to downstream task. Furthermore, we not only tested the data on a single model, but also compared three language models. We use the three mainstream cross-language models: bert-multiling-cased (BERT), xlmmlm-100-1280 (XLM), xlm-roberta-base (XLM-RoBERTa). These three models all come from the transformers package, and the hugging face provided us with great help. The system will sort according to accuracy and the weighted-averaged versions of Precision, Recall, and F1.
The rest of the article is structured as follows. Section2 proposes the task to be solved and introduces the corpus of the task. It analyzes the data set and proposes some tricks. In the Section3, we introduce the methods and techniques we used. The rst is to use some conventional operations to process the data set, the second is to introduce the data augmentation method we use, and the third is to introduce the language model we use. Section 4 presents the results and discuss the performance of the system. In Section 5, we give the conclusions and future work. 2 2.1
The organizer only proposed one task: EmoEvalEs [
The datasets [
To achieve good results, data preprocessing is essential. The rst step in our
model is to process these tweets. For how to preprocess the data, we refer to
IberLEF 2019 [
We ignored spelling errors in text tweets. After data processing, the sentence length in the tweet is distributed within 60. At the same time, we also keep the version that not remove the stop word in Spanish, so that we can judge whether removing the stop word in Spanish is good for our model. 3.2
Data augmentation technology is widely used in computer vision, but it is rarely e ectively used in Natural Language Processing (NLP). The essential reason is that some data augmentation methods in the image will not change the meaning of the image itself while augmenting the data. In this article, we try to use a data augmentation method: Masked Language Model-based data augmentation. Masked Language Modeling (MLM) can not only predict words, but also can be used for text data augmentation, and other methods, such as based on synonym substitution and word embedding. Compared with the replacement of, the text generated by data augmentation based on the Masked Language Model is more grammatically uent, because the model considers the context information when making predictions. We can easily use HuggingFace's transformers library. One place to pay attention to in this method is how to determine which part of the Mask text is to be used. Do data augmentation for each processed tweet, and determine the number of masks according to the length of each sentence, details as follows: { Fixed 1 [Mask], random position, repeatable. { Fixed 2 [Mask], random position, repeatable. { Fixed 3 [Mask], random position, repeatable. { 15% of the length of the sentence [Mask], the position is random, not repeatable. { 20% of the length of the sentence [Mask], the position is random, not repeatable.
Next is a concrete example: `This is [Mask] cool' `score': 0.515411913394928, `sequence': `This is [pretty] cool' `score': 0.1166248694062233, `sequence': `This is [really] cool' `score': 0.07387523353099823, `sequence': `This is [super] cool' `score': 0.04272908344864845, `sequence': `This is [kinda] cool'
\score" indicates the degree of similarity between the word and [Mask], and \sequence" indicates the sequence after data augmentation. The sequence with the highest score is selected as the sentence augmented by our data. 3.3
For SA Task, we combine some classic deep learning models. In the deep learning model, we mainly use the most popular cross-lingual models. These cross-lingual models can be better used to process Spanish text. These models include two parts: tokenizer and SequenceClassi cation. Among them, the tokenizer divides the input into words and processes it into the input required by the model. SequenceClassi cation is applied to speci c downstream tasks.
{ tokenizer: First of all, the tokenizer was splitting strings in sub-word token strings, converting tokens strings to ids and back, and encoding or decoding. Adding new tokens to the vocabulary in a way that is independent of the underlying structure (Byte Pair Encoding). Managig special tokens (like mask, begining-of-sentence, etc). The model needs language embedding during inference. We will use the tokenizer module to process the input before entering the model. The input required for each cross-language model is different, and the tokenizer module will generate the corresponding embedding according to the speci c model. The tweet rst undergoes data preprocessing, and then uses the tokenizer to vectorize the input. According to di erent models, Token embeddings, Segment Embeddings, and Position Embeddings are obtained. { SequenceClassi cation: The model with a sequence classi cation/regression heads on top(a linear layer on top of the pooled output). We add a dropout layer and a linear layer to the last layer. The input feature of the linear layer we added is 768, the output feature is 384, and the p value of the dropout layer is set to 0.1. The input feature of the last layer is 384, and the output feature is 7, to satisfy our 7 classi cation task. The model does not require language embedding at inference time. They can identify the language used in the context and infer accordingly.
XLM-RoBERTa [
XLM [
BERT: The bert-base-multilingual-cased model is based on BERT, which has
obtained state-of-the-art performance on most NLP task [
We have fully experimented with the above three model. The process and result of the experiment will be described in detail below. We rst experiment with di erent data augmentation methods to obtain better data augmentation methods. Then we use the best data augmentation method on the three crosslanguage models and compare them on the development set. Finally, we use the best model to experiment on the test set and get the nal result. 4.1
During the development set experiment, we divided the training set into a new training set and a development set, where the division ratio was 0.9 for the new training set, and the development set was used as the test set.
We will do the same preprocessing before sending the data to each model to ensure the comparability of the experiment, that is the ablation experiment. Use ablation experiments to evaluate the quality of our di erent data processing methods. And here some hyper parameter sett of the model; the max length of a single sentence in each batch is adjusted according to the preprocessed results, the max length does not exceed 60; and add \[CLS]" and \[SEP]" to each input; and padding for sentences with insu cient length; We use a batch size of 32, a base learning rate of 1e-5; the activation function is AdamW; and the number of ne-tuning is set to 3. For every 10 batch-size training set data processed, model.eval will be executed to process the development set data. And save the model and print the accuracy rate. 4.2
Use learning rate warm-up. During warm-up, the learning rate linearly increases from 0 to the initial learning rate of the optimizer. After the warm-up phase, a schedule is created to linearly reduce the learning rate from the initial learning rate in the optimizer to 0. The number of steps for the warm up phase is set to 0, that is, the learning rate is warmed up from the beginning. The total number of training steps are set to N.
N = len(train dataloader) num epochs (1) 4.3
We conduct data augmentation experiments on the XLM-RoBERTa model to get the best data augmentation method, and use the data augmentation method in all subsequent models. From Table 2, we can see that when using 15% of the length of the sentence [Mask], XLM-RoBERTa is the best among the three evaluation indicators Accuracy, Recall weighted and F1 weighted score. But when the XLM-RoBERTa model uses 20% of the sentence length [Mask], the precision of the model is the best. The choice of pre-training model has a great impact on the nal performance, even though they are all based on transformers. 4.4
Next, we conduct ablation experiments on three cross-language models for data augmentation. From Table 3, we can see that the four evaluation indicators
Data Augmentation Accuracy Precision weighted Recall weighted F1 weighted Yes No Yes No Yes No of the BERT model and the XLM model during the data augmentation operation are less than those without the data augmentation operation. In the XLM-RoBERTa model, data augmentation can improve the performance of the model to a certain extent, except for Precision weighted scores. Furthermore, whether data augmentation is performed or not, the performance of the XLMRoBERTa model is better than the other two models. The biggest advantage of XLM-RoBERTa is the signi cant increase in the amount of training data. And the e ect is especially obvious in languages with fewer resources.
Finally, we use the best performing XLM-RoBERTa model on the test data set, with data augmentation and no data augmentation at the same time. From Table 4, we can see that data augmentation will e ectively improve the performance of the model in test data set, especially in Accuracy, Recall weighted and F1 weighted scores. But it will also reduce the Precision weighted score of the model. We use three popular cross-language models BERT, XLM, and XLM-RoBERTa for comparative experiments. It can be seen from Figure 3 that the XLMRoBERTa model surpasses BERT and XLM in all evaluation indicators. Because the XLM-RoBERTa model increases the number of languages and the number of training data sets, it uses more than 2TB of pre-processed CommonCrawl data sets. And during the ne-tuning of the XLM-RoBERTa model, based on the ability of the multi-language model to use multi-language annotation data to improve the performance of downstream tasks. Beyond this, although we use data augmentation technology to only double the amount of data, the performance improvement is still very obvious. Accuracy and Recall weighted have increased by 0.6%, and F1 weighted has increased by 1.2%. Moreover, the data processing method and Learning Rate Warm-up technology we use also play a certain role in the improvement of model performance and the speed of model inference.
This article introduces the EmoEvalEs task: Emotion detection and Evaluation for Spanish in the IberLEF 2021. We use three cross-language models BERT, XLM and XLM-RoBERTa. Using the cross-language pre-training model can handle Spanish tweets well, and the pre-training model only needs to modify the nal output layer to be well applied to downstream tasks, which greatly reduces the training time. We improve the performance of the model through a series of data processing and data augmentation technologies. As can be seen from Table 5, The results and ranking of our model on the o cial test sets (Ranked by Accuracy). Our nal model got an Accuracy of 0.692, a Precision weighted score of 0.680, a Recall weighted score of 0.692, and an F1 weighted score of 0.664 on the o cial test set, ranking 5th, 6th, 5th, and 8th among all participating teams.
We are very satis ed with our rst participation in IberLEF 2021 and our performance in the EmoEvalEs task, but some work still needs improvement. The future work is to explore more advanced technology, use better models, and achieve better results in the next competition. For example, use better data augmentation technology to increase the amount of data set, select the best hyperparameter adjustment technology, and use other pre-training models such as Longformer, ELECTRA, ALBERT or even T5.
First of all, we would like to thank the organizers for the opportunity and organization, as well as the help of teachers and seniors. Finally, we would like to thank the school for supporting my research and the patient work of future reviewers.