This paper describes the participation of the ELiRF research group of the Universitat Politècnica de València in the TASS 2020 Workshop, framed within the XXXVI edition of the International Conference of the Spanish Society for the Processing of Natural Language (SEPLN). We present the approach used for the Monolingual Sentiment Analysis and Emotion Detection tasks of the workshop, as well as the results obtained. Our participation has focused mainly on employing an adaptation of BERT for text classification on the Twitter domain and the Spanish language. This system, that we have called TWilBERT, shown systematic improvements of the state of the art in almost all the tasks framed in the SEPLN conference of previous years, and also obtains the most competitive performance in the tasks addressed in this work.
to compare the results (Deep Averaging Networks). Section 4 summarizes the conducted experimental evaluation and the achieved results. Finally, some conclusions and possible future works are shown in Section 5.
Two tasks have been proposed by the organizers: Task 1 - General polarity at three levels [
Table 1 shows the tweet distribution according to their polarity in the training set for the first task. It can be observed a bias towards the N and P classes in some Spanish variants (Spain and Mexico). In general, the N class is the most frequent class and the NEU class is the less frequent (excluding some variants like Peru or Costa Rica). In Table 2 the tweet distribution for each emotion in the training set of the task 2 is shown. In this case, there is a large bias towards the class Others that acts like a sink of unconsidered emotions or combinations among emotions. The less frequent class, by far, is the Fear class.
We decided to use Deep Averaging Networks [
To compute the word embeddings, we used the Twitter87 model [
TWilBERT is a framework for training, evaluating and finetuning BERT-based models 2 in the Twitter domain. It also includes several techniques and improvements published in recent works for the BERT architecture. Furthermore, several pre-trained models for Spanish are freely released with the framework: TWilBERT-base and TWilBERT-large. Both models were trained with 94 million of (tweet, reply) pairs in several Spanish variants.
The purpose of TWilBERT is to adapt and improve the language modeling capacity of the
BERT architecture [
MLM+ROP
MLM+ROP
104 languages
✗ 12 12 768 768 64 64 64
To carry out the experimentation for both tasks, we used DAN as baseline in order to compare
the results with the TWilBERT model. TWilBERT is a pre-trained deep model while DAN is
trained from scratch on the tasks. To make a fair comparison between them, a grid-search has
been performed on the hyper-parameters of DAN: number of layers ({1, 2}), number of units for
each layer ({64, 128}) and batch size ({8, 16, 32, 64}). Furthermore, dropout [
Our team presented three diferent runs to the competition: run-1 (DAN), run-2 (TWilBERT trained with the training set) and run-3 (TWilBERT trained with the training and development set until the epoch where the M 1 was maximized on the development set with the run-2).
The results of our runs for each Spanish variant are shown in Table 5. These results support the competitive performance of TWilBERT, that obtains +6.42 M 1, in average, more than DAN when it is trained with the training set (run-2) and +8.01 M 1 when it is trained with the training and development sets (run-3). Our system TWilBERT-large, trained with all the available data of this edition, obtains the best results of the competition in the Spain, Mexico, Costa Rica and Peru variants. At this point, it is important to highlight that, to obtain these results with TWilBERT, we did not perform any exploration of its hyper-parameters. Therefore, these results could be improved by performing a more extensive experimentation.
Regarding the second task for Emotion Detection, we used the same systems than for the ifrst task. Table 6 shows the results of DAN and TWilBERT on the development set. In this case, the results are more similar than in the previous task on the development set, with a diference of 0.5 M 1. Also, it can be seen how the MP and MR are unbalanced for the DAN system, while for the TWilBERT system are similar between them, mainly due to the weighting of the cross-entropy.
We submitted two runs for the task 2: run-1 (DAN) and run-2 (TWilBERT trained with the training set). The results of each run in the test set are shown in Table 7. It can be observed how TWilBERT generalizes better than DAN on this set, obtaining +1.5 M 1 in comparison to DAN, mainly due to an increment of +3.3 MR. Both systems obtained the best results of the competition.
We have proposed the use of TWilBERT for the Sentiment Analysis and Emotion Detection tasks of TASS 2020. The results obtained by our system are very promising, being the first or second ranked system in almost all the Spanish variants of the Sentiment Analysis task and the ifrst ranked system in the Emotion Detection task. This is especially significant, considering that these results have been obtained without an exploration of the hyperparameters of the model and only a reasonable configuration was used for all the tasks.
This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) and by the GiSPRO project (PROMETEU/2018/176). Work of José-Ángel González is financed by Universitat Politècnica de València under grant PAID-01-17.