This paper describes the participation of the Department of Computer Science at Universidad Católica San Pablo (UCSP) for the TASS 2020 Workshop. We have developed sentiment analysis algorithms for the monovariant and multivariant challenges. In both cases, our approach is based on transfer learning using BERT language modeling. We also propose a procedure based on this language model to generate contextual data augmentation aimed to increase the training dataset and prevent overfitting. Our design choices allow us to achieve comparable state-of-the-art results regarding the TASS benchmark datasets provided.
Deriving an efective algorithm for Spanish Twitter sentiment analysis has been long pursued
since the first TASS challenge in 2012 [
In this paper we still rely on transfer learning to solve the classification task, but our approach has been carefully designed bearing in mind that the Spanish language has several variants.
In NLP it is common to use text input encoded as word embeddings. Those embeddings,
which allow us to encode semantic similarities among words, can be defined using several
approaches such as Word2vec [
Nowadays, this kind of encoding has evolved to a language model encoding. The idea is to
use language context in order to better encode words [
Our proposed classifier relies on three components: a multivariant Spanish corpus, a general
language model and a data augmentation step. Thus, we start by training a general language
model based on BERT [
Those simple design choices allow us to obtain comparable state-of-the-art results regarding
the two subtasks of polarity classification task presented in the 2020 TASS challenge [
The rest of the paper is structured as follows. In Section 2, we describe the task at hand. In Section 3, the system is explained. The experimental setup is described in Section 4. Results and conclusions are presented, accordingly, in Section 5 and 6.
The aim of the task 1 is the correct classification of sentiments (N:Negative, P: Positive, NEU: Neutral) in tweets written in Spanish and variants. This year, the task has been sub-divided in 2 subtasks1: • Subtask-1: Monovariant. In this challenge, we have to predict the sentiment in tweets of 5 Spanish variants included: Spain, Peru, Costa Rica, Uruguay and Mexico, and for each one we have three datasets: Training, Validation and Test. Moreover and even when we could use any additional corpora or linguistic resource, we have to submit the predictions of every country Test dataset in a diferent file to the system evaluation. • Subtask-2: Multivariant. Complementary to the subtask-1, we have been provided with one additional Test dataset extracted from the diferent datasets of countries commented before. Again, it is possible to use any corpora or linguistic resource.
Furthermore, informal language used in tweets and lack of context because the limited characters permitted in those add up several challenges to this competition.
Finally, the metrics used in the evaluation system are the macro-averaged versions of MacroPrecision, Macro-Recall and Macro-F1. However, Macro-F1 was used to rank the diferent submits to the system.
We propose a single system for improving polarity classification in small datasets based on
the high performance Language Model (LM) referred to as BERT developed by Devlin et al [
In order to do so, we start by pre-training a general LM on a multivariant Spanish corpus
related to the target task. Subsequently, to enhance our small target dataset and prevent
overfitting, we use the LM to generate Unsupervised Data Augmentation using a novel technique
called Conditional BERT Contextual Augmentation [
Due to the huge impact of BERT in NLP, we decided to use it as our initial LM in its base form.
By doing so, we have fewer parameters to train and it is easier to fine-tune with less samples.
According the guidelines presented by Devil et al. [
• Masked Language Model. Mask some percentage of the input at random and then predict those masked tokens. • Next Sentence Prediction. Given a first sentence, the model must predict the one that would follow.
Also, this process is made using a free source dataset described in spanish-corpora repository2.
Based on the Conditional BERT Contextual Augmentation algorithm [
Finally, we add up the new formed sentences into the original training dataset and perform downstream task on it.
In this final step we build a classifier, a simple fully connected layer, on top of the initial LM and a Softmax which allow us to predict the correct class. However, in order to prevent overfitting, we use the new enhanced dataset created using the aforementioned Semi-supervised Data Augmentation process.
A detailed description of hardware and software requirements for replicating our research results. In addition, we describe the hyper-parameters tuned during experimentation that allow us to train and converge without overfitting and regularization.
The experiments were executed on Jupyter Notebooks running Python 3.7 kernel and PyTorch 1.4.0. Moreover, all models were trained on a GPU 2080 Ti with 11 GB GDDR6.
For a complete description about dependencies we refer to our public project repository3.
The datasets used in this work are freely available to everyone to use.
4.2.1. Pre-Training
4.2.2. Target
The dataset used to pre-train the LM is a compilation of Large Spanish Unannotated Corpora
[
3https://github.com/dpalominop/atlas 4http://tass.sepln.org/2020
1. The text was converted to lowercase and every accent mark was removed. 2. Repeated characters were replaced by single characters. 3. User references were transformed to a specific token.
4. Useless spaces were removed.
As will be shown in Table 3, using a multilingual LM decreases the performance of the system. In contrast, if we use a monolingual LM, results are noticeably improved. Furthermore, we have to use a LM trained on a similar dataset w.r.t. the target task and the best option was training one using the general corpora described in section 4.2. The main hyper-parameters used through this process are: 1. Backpropagation Trough Time (BPTT): 70 2. Weight Decay (WD): 1 − 2 3. The batch size (BS): 64
2. Weight Decay (WD): 1 − 2 3. The batch size (BS): 32 4. Number of randomly masked words (k): 4 5. Repetitions of the process (n): 2
The main hyper-parameters used through this fine-tuning LM process are:
Similar to the sub-section before, the main hyper-parameters used through this fine-tuning process are:
2. Weight Decay (WD): 1 − 2 3. The batch size (BS): 16
Results for TASS 2020 Task 1 - Monovariant are shown in Table 1. Our submission (referred to as daniel.palomino.paucar) was ranked 1st or 2nd in all variants of Spanish presented in the competition among all competitors (M-F1 score).
Furthermore, results for TASS 2020 Task 1 - Multivariant are shown in Table 2. Our submission (referred to as daniel.palomino.paucar) was ranked 1st among all competitors (M-F1 score).
On the other hand, given the possibility to send 3 diferent submissions to the competition, we decided to use that to test three variations of our system and perform ablation experiments in order to get a better understanding of the competitive results obtained.
The output of these experiments is shown in Table 3. Thus, we can observe a significant positive impact when using a monolingual LM instead of a multilingual LM. Regarding the M-F1 metric used, there is 10 percent diference between the two methods. Moreover, if we include the additional step of Unsupervised Data Augmentation, the results further improve.
We have presented a novel sentiment classification system based on BERT that includes an additional step of Unsupervised Data Augmentation. The system has been applied to sentiment analysis on Spanish tweets and its variants. Despite its simplicity, this approach allowed us to be ranked 1st or 2nd on the TASS 2020 Task 1 - Multivariant and 1st on the TASS 2020 Task 1 - Monovariant.
Furthermore, the ablation experiments have shown that a careful choice of the language model can improve the results drastically. Thus, we have chosen to pre-train the language model using a similar dataset to the target task. Moreover, the use of data augmentation allowed us to further improve our previous results for most variants of the Spanish language. However, the performance on the Mexican variant decreased after using this technique —probably due to overfitting during the fine-tuning process.
This work was funded by CONCYTEC-FONDECYT under the call E041-01 [contract number 34-2018-FONDECYT-BM-IADT-SE].