Due to the technical gap between the language models available for low-resource languages and the state-of-the-art models available in English and Chinese, a simple approach that deploys automatic translation and ensembles predictions from Portuguese and English models is competitive with monolingual Portuguese approaches that may demand task-specific preprocessing and hand-crafted features. We performed our experiments on ASSIN 2 - the second edition of the Avalia¸c˜ao de Similaridade Semaˆntica e Inferˆencia Textual (Evaluating Semantic Similarity and Textual Entailment). On the semantic textual similarity task, we performed multilingual ensemble techniques to achieve results with higher Pearson correlation and lower mean squared error than BERTmultilingual, and on the textual entailment task, BERT-multilingual could be surpassed by automatically translating the corpus into English and then fine-tuning a large RoBERTa model over the translated texts.
Although recent advances in Transformer architectures [
We demonstrate that, due to the current gap between the models available for Portuguese and these two languages, a simple approach that deploys automatic translation and ensembles predictions from Portuguese and English models is competitive with monolingual approaches that may demand task-specific preprocessing and hand-crafted features to achieve the same accuracy.
For this purpose, we examine the e↵ ectiveness of multiple ensemble
techniques on the Semantic Textual Similarity (STS) and Recognizing Textual
Entailment (RTE) tasks proposed on the ASSIN 2 dataset3 [
In section 2 we describe some of the related work for Transformer architectures in low-resource languages. The approaches to the Transformer architecture investigated in this paper are described in section 3. Section 4 shows the approach considered for ensemble modeling. The experiments carried out for evaluating our model for the ASSIN2 tasks are described in section 5. Section 6 finishes this paper with its conclusions and proposals for future work. 2
As Transformer architectures are reasonably recent, there are few works which have attempted to apply large pretrained English models to other linguistic domains. Fonseca fine-tuned the GPT-2 model in Portuguese, and his results5 indicate that this can become a viable approach for tackling text generation problems in the Portuguese language.
However, the strategy of automatically translating a dataset into a foreign
language and then ensembling models from distinct linguistic domains to address
a natural language processing task has already been throughly researched in the
past. In the field of sentiment analysis, Wan et al. [
Under di↵ erent circumstances, Araujo et al. [
Tian et al. [
Belinkov et al. [
Although machine translation may be e↵ ective, it may not be suitable for
certain industrial settings, specially when one cannot a↵ ord to call an external
translation service several times. Tang et al. [
Compared to the Portuguese language, a relatively large amount of Transformer models have been trained for Spanish and other major Romance languages. To the best of our knowledge, no large Transformer model trained exclusively for Portuguese has ever been made publicly available, and no other paper has ever investigated how Transformer architectures and machine translation can be leveraged for natural language processing tasks in the Portuguese language. 3
Only two transformer architectures were considered for our experiments:
BERTmultilingual and RoBERTa. We started from pretrained models, which were
fine-tuned to a STS task through the Transformers library developed by Hugging
Face [
BERT stands for Bidirectional Encoder Representations from Transformers [
In order to define its prediction goal, BERT utilizes two training strategies. One of them is called Masked LM (MLM). Before being fed into the model, 15% of all tokens are replaced by either a random token or a fixed [MASK] token. In this way, the model has the training goal to improve its ability to predict the original token that occupied these positions.
For its second strategy, the model is pretrained for a binarized Next Sentence Prediction (NSP) task with sentences A and B, where B has an equal chance of being either a randomized sentence or the actual next sentence after A.
In our experiments, we utilized the multilingual, cased BERT model, trained with 110 million parameters on 104 languages, including Portuguese. These languages were chosen because they are the ones with the largest Wikipedias, and the entire Wikipedia dump for each language was taken as the training data. Although overrepresented languages were downsampled during the training stage, no downsampling was performed among dialects within the same language. It should also be noticed that Wikipedia does not impose any Portuguese dialect for any of its articles, and therefore, it is not possible to di↵ erentiate between them on its dumps.
The flexibility of BERT has contributed to establish it as one of the main
paradigms for advancing the state-of-the-art in natural language processing
tasks. As of today, most state-of-the-art models are either directly or indirectly
based on the original BERT architecture [
After the release of BERT, several authors studied the model and pointed out
architectural choices that should be reconsidered. The main issues lied on its
training strategies, the Masked LM and Next Sentence Prediction tasks. In fact,
some of the limitations found within these strategies had already been
acknowledged on the paper that first presented the BERT architecture [
In our experiments, we have used one of the models that improved on BERT
while trying to reach a proper training strategy: RoBERTa, which stands for
Robustly Optimized BERT Pretraining Approach [
As demonstrated by [
We can also expect better performance even when comparing multilingual
versions of RoBERTa with multilingual versions of BERT. In fact, Conneau et al.
[
Although there are versions of RoBERTa available which have been previously fine-tuned on MNLI and the output of GPT-2, we performed our experiments only on a pretrained version of RoBERTa which was based on the architecture of BERT-large.
For both STS and RTE tasks, BERT-multilingual was fine-tuned on the original, Portuguese dataset, and RoBERTa was fine-tuned on the same dataset after it was automatically translated into English. In this way, it can be said that a stronger model (RoBERTa) is receiving an input of lower quality (an automatically translated dataset) while the weaker model (BERT-multilingual) is receiving an input of better quality (the original dataset).
After fine-tuning, we tested two distinct ensemble techniques to combine the predictions generated by both Transformers. The first one was averaging: the fine-tuned models generated predictions for the test set, and then these scores were averaged through an arithmetic mean to produce the final submission.
The second technique utilized is most commonly known as stacking. A careful
and diagrammatic description of the stacking ensemble technique has been made
by Kyriakides et al. [
The metadata was generated through K-fold cross validation over the entire training set. Both BERT and RoBERTa were fine-tuned on their corresponding training sets for each possible combination of K 1 folds. Therefore, an ensemble made through K-fold stacking will generate K distinct BERT models fine-tuned on the original dataset, and K distinct RoBERTa models fine-tuned on the translated dataset.
After generating 2K fine-tuned models, each one of them will generate predictions for the fold from the training set which was missing during its fine-tuning process. Although all labels from the training set are known to us, they are temporarily ignored for the sake of generating the metadata for the MLP, and we concatenate the predictions from all fine-tuned models. As a result, we will have produced metadata for the entire training set, which will be used to train the MLP.
The MLP, our meta-learner, is trained while taking the metadata generated by each transformer as its input, and the gold score for the sentence pairs of the training set as the training goal. During this process, the MLP will try to learn how to properly weight the contributions of each transformer on each score range.
After training the MLP, we fine-tune one BERT and one RoBERTa model over their entire corresponding training sets, without considering any division into folds. These models produce predictions, which are combined by the trained MLP to produce the final predicted scores. If successful, this trained meta-learner will have produced scores that are more accurate than the predictions generated by either base learner considered in isolation.
We did not test any averaging techniques other than a simple arithmetic mean, and our MLP was made only of two Dense layers of 64 neurons with a ReLU activation function followed by a single Dense layer of one neuron with a linear activation function.
It should be noticed that there is a trade-o↵ between improving the overall performance of the ensemble and the amount of computational resources required to fine-tune transformer models for a higher amount of folds. There are also diminishing returns for increasing the amount of folds: although the accuracy will initially increase, it is expected to stabilize after the folds become small enough for the MLP to learn the general patterns present on the metadata. 5
We evaluated the performance of our models on the ASSIN 2 dataset. Part of our experiments were submitted during the ASSIN 2 workshop, and then their accuracy was compared with the results obtained by the other participants. In this section, we examine some particular features of the dataset and how they influenced the obtained results. These results are described both in their entirety and, in the case of the STS task, also separately for five distinct gold score intervals. 5.1
The dataset consists of sentence pairs in the Portuguese language, with
humanannotated scores for Semantic Textual Similarity and Textual Entailment. The
Semantic Textual Similarity task is available in the GLUE benchmark [
During our experiments, RTE was treated as being a regression task, exactly like the STS task, rather than being a classification task. In other words, while STS was treated as the task of predicting scores ranging from 1 to 5, RTE was similarly treated as the task of predicting scores ranging from 0 to 1, taking ”None” as 0, ”Entailment” as 1. And then, before submitting our files to the o cial evaluation script for the ASSIN 2 dataset, the entailment scores were rounded up and converted back to their corresponding labels.
Datasets for Brazilian and European Portuguese were released during the
first edition of ASSIN [
It should be noticed that we did not apply any preprocessing steps to the datasets other than those required by the Transformer models themselves. We also did not take any measures to modify or increase the quality of the translated sentence pairs after they had been retrieved from the translation API. 5.2
The Transformers (BERT and RoBERTa) were fine-tuned both on the standard training and validation sets for ASSIN 2. After fine-tuning, they produced their own predictions for each one of the available test sets. These predictions were submitted to the standard evaluation script for the ASSIN datasets, both in isolation and combined through two di↵ erent ensemble techniques: averaging and 5-fold stacking.
It can be seen on Table 1 that the best Pearson correlation for the STS
task has been achieved by combining the predictions from both Transformers
through an aritmetic mean, although 5-fold stacking achieved a mean squared
error considerably lower than any other of the evaluated approaches. For the
entailment task, RoBERTa singlehandedly performed better than either BERT
or any of the ensembles. However, 5-fold stacking achieved results pratically
equivalent to RoBERTa.
Reimers et al. [
With these concerns in mind, we have measured both Pearson correlation and the mean squared error individually for five distinct gold score intervals.
Regarding the mean squared error, in Figure 1 we can see that, while the performance of BERT-multilingual linearly improves for higher gold scores, RoBERTa performs exceptionally better than all other models for the lowest score range between 1.0 and 1.4 . After this range, its performance suddenly decreases. And after decreasing, it also starts to linearly improve.
As sentence pairs on the lowest gold score intervals are very di↵ erent from each other, one possible explanation is that they were able to maintain their high degree of dissimilarity even after automatic translation. In this way, RoBERTa was able to provide results closer to its performance for sentence pairs which were originally written in English; and as expected, such results are consistently better than what can be achieved through BERT-multilingual.
Although averaging performed slightly better than 5-fold stacking on the lowest score range, 5-fold stacking was able to provide a mean squared error lower than any other model on all other score ranges.
Regarding the Pearson correlation, it is interesting to notice that RoBERTa provided results which were superior or otherwise competitive with all other models except on the higher score interval between 4.5 and 5.0, and BERTmultilingual has a performance inferior to all other models except on the same higher score interval. Although 5-fold stacking failed to notice such complementary behavior, it was possible to achieve results competitive with both models for all score ranges by simply averaging their predictions.
As we treated the RTE task as if it were a STS task with only two scores, the results obtained by RoBERTa on this task can be explained by its ability to accurately determine when two automatically translated sentence pairs are totally dissimilar to each other, that is, when their entailment relationship is equal to None. In other words, RoBERTa has a lower false positive probability than BERT-multilingual or our evaluated ensemble approaches. As a result, RoBERTa achieved not only high accuracy and Macro-F1 scores, but also a high Matthews correlation coe cient.
Therefore, for the RTE task, an ensemble will not be necessary, and we can simply take the predictions produced by RoBERTa as our final result. However, an ensemble made through 5-fold stacking is able to achieve equivalent results, as it can recognize that BERT-multilingual performs consistently worse than RoBERTa on this task, and thus it minimizes the weight given to the predictions made by BERT-multilingual as much as possible. 7 https://github.com/huggingface/transformers/issues/876#issuecomment-536792338 Fig. 1: We break down the dataset into five parts determined by the intervals in which their gold scores are located. Then we calculate the Pearson correlation and the mean squared error of all experiments listed on Table 1 separately for each one of these intervals.
We have discovered that, for the evaluated Portuguese RTE task, automatically translating the dataset into English and then fine-tuning a large RoBERTa model over the translated dataset can produce better results than BERT-multilingual or monolingual approaches that rely only on the resources natively available for the Portuguese language.
And although the same results could not be achieved in isolation by RoBERTa on the STS task, state-of-the-art results in the Portuguese language for this task can be achieved by combining the predictions made by BERT-multilingual and RoBERTa through an adequate ensemble technique, which in some cases may be as simple as taking the arithmetic mean of both predictions.
In spite of the simplicity of our approach, we have achieved the best results for the RTE task on the ASSIN 2 workshop. Later on, we also found out that simply taking the arithmetic mean of the predictions generated by BERT and RoBERTa was able to surpass all the Pearson correlation scores which had been achieved by the participants to the STS task. We did not perform any preprocessing on the training data, either before or after automatic translation. We also did not take into account any features other than the predictions generated by the Transformer architectures as they have been implemented in standard libraries.
Therefore, we believe that there is room for significant improvements to our
results. For both tasks, it may be convenient to test more sophisticated
techniques for ensemble building [
For future experiments, new English models that reach the top positions on
the GLUE Benchmark Leaderboard8 for the STS-B and RTE tasks should be
considered, and these models may be combined or compared with Transformers
trained in Romance languages. For instance, experiments may be performed on
CamemBERT [
However, as a long-term strategy, we should also consider solutions which entirely avoid the inevitable limitations of machine translation, such as training our own Transformer models exclusively for the Portuguese language.
The authors would like to acknowledge the support of the Institute of Informatics at the Federal University of Goi´as, Americas Health Co., B2W Digital, Datalawyer and Data-H Data Science and Artificial Intelligence.