In this paper we describe the system designed by the Mathematics Research Center (CIMAT) for participating atMEX-A3T 2020. In this work, we addressed the Aggressiveness Detection (AD) task by exploiting Bidirectional Encoder Representations from Transformers (BERT) and Data Augmentation. BERT fine-tuning has shown outstanding performance in a wide range of language tasks. However, according to recent research fine-tuning BERT on small size datase<t1s0(K instances) often results in unstable models. In other words, even when only the final layer is randomly initialized, distinct random seeds lead to substantially diferent results. In this paper, we use two strategies that are helpful in producing more stable classification models based on fine-tuned BERTs. The first strategy take advantage of ensembles, whereas a second strategy relies in data augmentation. The experimental evaluation showed that our proposals outperforms all baselines by a wide margin, and has the overall first place for Aggressiveness Detection in Mexican Spanish Tweets.
diferent structure in documents, our approach wassiample baseline, only for our reference for future research: a Bag-of--grams at word and character level fed into an Support Vector Machines (SVM).
The AD task is commonly approached as a supervised classification proble4m,5[
One of the most successful approaches is the Bidirectional Encoder Representations from
Transformers (BERT), which was first proposed in1[
The remainder of this document is organized as follows: Section 2 presents the proposed strategies. Section 3 describes experimental settings. The experiments and results are presented in Section 4. Finally, Section 5 outlines the final conclusions and future work.
This section describes our two proposed strategies to alleviate the instability of fine-tuning BERT on few sample and unbalanced datasets. These strategies are built on fine-tuned BERT models that only vary the random seed of an extra classification layer for domain adaptation. The first strategy is based on BERT ensembles and two well known voting schemes. The second strategy uses data augmentation to improve even more the efectiveness and stability. In our evaluation, we empirically show that both methodologies provide benefits for the AD and both ranked first place of the challenge.
There are many ways to combine the information of several models, but we are particularly interested in those based in ensemble theory. The idea of having an ensemble is that several models (possibly weak models) can make a strong on1e7][. One of the key ideas in successfully build ensembles is that the prediction space should be diver1s8e].[ This is that individual models can have diferences among decisions; we hypothesize those are the unstable individual BERTs. In general, when the latter conditions are met, it could be possible to obtain a stronger model with simple strategies. In our case we consider the following two straightforward voting schemes: • Majority Voting Scheme: we predict the most voted class among the classifiers of the ensemble. In case of tie, we perform random prediction among the classes in question. • Weighted Voting Scheme: we aggregate the confidence prediction for classes in each model of the ensemble to build a final weighted vote. This confidence prediction, in our case is the output of the last Softmax layer.
These strategies based on ensembles and simple voting schemes have resulted very convenient
and have been explored with diferent base models in several domain17s,[
A common and efective technique in deep learning for image related tasks is that of data augmentation, in which the goal is to create a new training data by means of a transformation: sometimes simple like rotation, reflections and cropping, sometimes more complex techniques are used with better result1s9[].
Data augmentation for text based tasks is a very diferent scenario. The information in
documents is sequential and the word, usually taken as the basic unit, has a syntactic and semantic
meaning that depends on the context. Thus, changing the individual words or their order could
result in noisy data that hurts the performance. This is specially true for approaches beyond
the Bags-of-Words and inspired in language modeling like BERT where the order, context and
structure of the text matters. Fortunately, there have been some advances demonstrating sightly
improvements in some scenarios2[0]. In this work, we have carefully adapted two methods, and
proposed a new one to perform data augmentation. The explored data augmentation strategies
are the following:
• Easy Data Augmentation (EDA) [
• Unsupervised Data Augmentation (UDA) [
It is worth to mention that previous described strategies, originally were designed for English language and therefore rely on dependent language tools. Thus, for each strategy we did several adaptations in order to exploit them in Spanish.
The MEX-A3T Team provided us the data set, which has 5222 no-aggressive and 2110 aggressive tweets. For the experiments in this paper we split the training data into 80% for training and 20% for test.
To compare the proposed strategies in this paper, we have trained two baselines that are commonly used in the literature: • N-grams ensemble with SVM: We used a grid search to find a suitable number of unigrams, bigrams and trigrams at word level. We also find the number of 2-6-grams at character level. Those features were fed into an SVM to explor e hthyeperparameter. • Bi-LSTM: This baseline is a neural architecture with a Bi-LSTM and a classification layer.
We use the pre-trained word2vec vectors in Spanish from Caro and Cuervo Institute Linguistic Research Grou1p. We fix the learning rate in 1e-3, and we used the Adam Optimizer.
Regarding fine-tuning BERT, we set the hyper-parameters as the authors in11[] recommend. We use Adam with a learning rate of 1e-5 and a batch size of 32 for three epochs. We use a classification layer and as loss function the weighted Cross Entropy Loss by using the proportion of each class. In this process, we used a BETO pre-trained model in Spanish t2e3x]tb[y using the widely known default implementation in24[].
This section describes the experimental evaluation that shows the main findings of this work. Firstly in Section4.1 we design a set of experiments to observe the benefits of building and ensemble of several BERT. Secondly in Sectio4.n2 we explore data augmentation strategies to improve even more their performance. In order to have comparable results in this section, we partitioned the trained dataset in the following way: 80% for training, and 20% for test. The metrics we use to compare the methods were th e1 in each class, the macro1 and the standard deviation.
The purpose of experiments in this section is to empirically show that BERT ensembles are helpful and more stable for classification. For the experiments in this section BERT was full finetuned (the 130 millions of parameters) two hundred times with diferent seeds that initialized the last layer to detect aggressiveness. This pool is then used to compute the averages over one hundred runs for each voting scheme. This means that, fSoirngle BERT the average performance of one hundred individual BERTs is reported with the standard deviations. For the case of a Ensemble ( ), diferent BERTs were randomly taken for each of the one hundred runs; we report the average performance and the standard deviation.
In Table1 we show the performance of BERT ensembles and other reference approaches. First of all, note that Single BERT clearly outperforms the two baseliness-:g1r)ams-SVM and 2) Bi-LSTM. Both of this approaches are very strong references, since we use well known strategies and heuristics to find suitable hyper-parameters (see Sectio3)n. The margin of improvement of Single BERT over the two baselines also shows that on average, the fine tuning have been successfully done.
From Table1 one can also note that Single BERT have a standard deviation of .836 in one hundred runs. However, when several BERTs are combined by using ensembles, the classification performance improves while the standard deviation decrease. In T1aebalech rowEnsemble ( ) ( -BERTs) has two values in each column metric. The left value was obtained by using a majority voting scheme, whereas the right value represents the performance when using weighted voting scheme. In Figure1 the left plot shows the 1 of the aggressive class as the number of BERTs in the ensemble increase up to one hundred. The red line is for majority vote, whereas the black line is for weighted vote. In a similar way, the right plot shows how the variance decrease as the size of the ensemble increase. The ensemble based on majority vote seems to have stable and higher performance until the number of BERTs is greater than sixty. Thus, the weighted voting scheme seems to be a better choice as the variance is consistently lower than the majority voting scheme. ThCeIMAT-1 run reported by the organizers in the ifnal ranking of the challenge is a simplEensemble (20). In the following Section, we will show how data augmentation can be used to improve the performance.
In this section we evaluate the data augmentation strategies described in Sec2t.i2o.nThe purpose of these experiments is to improve the classification performance of the previously evaluate ensemble approaches. In all experiments of this section we will use the weighted voting scheme for ensemble methods. Furthermore, as data augmentation strategies are computationally expensive in time and storage, the results of this section are based on fity runs instead of one hundred.
In Table2 experimental results shows that augmenting the data by using EDA and ADV improve the ensembles performance, while maintaining the standard deviation low compared to Single BERT. The UDA strategy actually seems to hurt the performance, but still having lower variance. The overall trend can be seen in the left plot of Fig2u,wrehere ADV strategy seems to be the best choice to augment text. In Figure3 we show the boxplot of fity runs of the Ensemble (20) for diferent data augmentation strategies. Note that the blue bloxplot, corresponding to the vanilla (no data augmentation) seems to have lower results than the red boxplot of the adversary data augmentation. Furthermore, the variance is still low in all data augmentation strategies.
Finally in Tabl3e and the right plot of Figur2e, we show the experimental result of removing the dropout in last layer of the fine-tuning process of BERT. Reportedly, it is better to have dropout, but if the test data comes from very similar distribution that could not be necessary. For example, note that the vanilla strategy (no data augmentation) has very similar results with and without dropout (see the first row of both tables). However, in Tab3lneote that data augmentation does not grow the performance at the same pace, and in some cases it hurts. Also note that the size of the ensemble helps, especially if the ADV strategy is used. Finally, note that UDA strategy improves thSeingle BERT model, which results in the best single model. These latter results suggest that while the UDA improves the behavior of the model by reinforcing positive examples every time, on the other hand, the use of adversaries might be doing it by learning a wider array of the dataset, specifically, the hard to learn examples. That would explain that the ensembles behave better as they go larger when using adversarial augmentation. In the final ranking of the challengeC,IMAT-2 corresponds to anEnsemble (20) that used EDA data augmentation. As experiments in Tab2leand3 show, this is not the best strategy to data augmentation since the performance is lower than ADV with and a similar or higher standard deviation. The experiments that use UDA and ADV was obtained once the challenge was finished.
We proposed strategies for AD that are based on ensembles of fine-tuned BERTs. The weighted voting scheme seems to be helpful for combining the decisions of several models. This means competitive performance and low variance of the model. The experiments show that there is space for the data augmentation paradigm in the tool set of the deep learning specialist. However, even if the results suggest that there is some improvement in the results when using data augmentation, they also imply certain trade ofs. The results indicate that there is more to be gained when working with adversarial data augmentation on ensembles because the models seems to learn in a more heterogeneous way. But the cost of doing so is quite elevated because of the need of using costly methods for the adversarial examples’ generation.
Guzman-Silverio and Balderas-Paredes appreciate CONACYT’s support for scholarships 925934 and 928540 respectively.