Recognizing Textual Entailment, also known as Natural Language Inference recognition, aims to identify if it is possible to infer the meaning of a text from another fragment of text. In this work, we investigate the use of multilingual models, through BERT, for recognizing inference and similarity in the ASSIN 2 dataset, an entailment recognition and sentence similarity corpus for Portuguese. We also investigate possible features that could enhance the results, such as similarity scores or WordNet relations. Our results show that a multilingual pre-trained BERT model may be su cient to outperform the current state-of-theart in this task for the Portuguese Language. We also show that using other features did not necessarily improve the performance of the model, however deeper studies are needed to investigate the causes for this.
Recognizing meaning connections such as entailment relations and content similarity among di↵ erent statements is part of daily communication and usually done e↵ ortless by humans. However, automatizing such communication component has been a challenge. Overcome it can help many Natural Language Processing (NLP) applications such as Machine Translation, Question Answering, Semantic Search and Information Extraction.
Particularly, the task of recognizing textual entailment (RTE) has been widely
explored in natural language processing field. Inference recognition in NLP, also
called text entailment recognition, consist recognizing a directional relationship
between pairs of text expressions, in which a human reading the first text would
infer the second one is likely true.[
Initially spread by the Pascal Challenge [
In addition to the growth of available corpora, several techniques have been
tested to improve inference recognition in NLP, including probabilistic models
and rule-based approaches [
A broadly known one is the Bidirectional Encoder Representations from
Transformers (BERT). BERT has been used e↵ ectively in multiple tasks like
Semantic Text Similarity, Paraphrase detection, among others[
In addition to MLM, the authors also use next sentence prediction task that jointly pre-trains text-pair representations. BERT was the first fine-tuned representation model to achieve state-of-the-art performance for a large number of token and phrase-level tasks, outperforming models developed specifically for these tasks. As far as we know, there are two monolingual pre-trained models publicly available: one trained for English and one for Chinese, along with one multilingual model. The multilingual model was trained for the 100 languages with most articles on Wikipedia2.
This work presents the results achieved by the NILC group for the ASSIN 2 shared task. We firstly analyse the corpus in order to find correlation between features and classification labels. Then we fine-tune multilingual BERT on sentence-pairs from ASSIN 2 corpus for the RTE task and finally, use the generated embeddings and incorporate some linguistic features for the Semantic Textual similarity (STS) evaluation. In general, we rank 3rd place for the RTE task and 5th place for STS task.
This paper is organized as follows. Firstly, we discuss previous related work in section 2. Afterwards, we describe the ASSIN dataset in section 3, followed by our experiments and results in section 5. Finally, some conclusions are presented in section 6. 2
There are several works on textual inference for multiple languages. However, due to di↵ erences in corpora for other languages, we report mainly works done in Brazilian Portuguese. This way, the works reported here use the corpus ASSIN, thus making a closer comparison to our work.
Rocha and Lopes Cardoso [
Fialho et al. [
Barbosa et al. [
Reciclagem and ASAPP were proposed by Alves et al. [
Finally, Fonseca and Alu´ısio proposed the Infernal system [
The ASSIN 2 corpus consists of 10,000 pairs of sentences tagged with similarity grading and entailment classification. Tags for the Recognizing Textual Entailment (RTE) task are None, when both sentences are not related in any way and Entailment when the second sentence is a direct inference of the first one.
ASSIN 2 was also manually annotated for semantic textual similarity in a range from 0 to 5, for which the pair was considered more similar by the annotators as higher is the number.
The corpus was split in two parts for the shared task, 7000 pairs were provided in advance as a training set and the other 3000 pairs later. The dataset provided for training was balanced, with 3500 pairs labeled as None and 3500 as Entailment. 4
As the dataset provided for training was balanced for the two entailment classes, we verified if it was equally balanced for other features. We use OpenWordnetPT to extract the number of synonyms and hypernyms in each pair per class. Hypernyms were counted when the second phrase contained any hypernym of any word in the first one. Synonym values were calculated in a similar way. Results are shown in figures 1 and 2.
As can be seen, sentence pairs with entailment tend to have more hypernyms (as can be seen by the bars representing counts of 0 and 2). The same can be observed for synonyms: there are more sentence pairs without entailment with no synonyms than those with an entailment relation.
Since the corpus was also annotated for semantic similarity in a continuous range from 0 to 5, we investigate the relation between similarity index and entailment classes. As a result, we find that entailment pairs have lower dispersion than none ones, and, di↵ erently from none, its range is mostly concentrated in higher similarity values, as can be seen in Figure 3.
Although similarity values shows notable correlation to entailment classes,
it was not possible to considered them for the entailment recognition task, once
it was part of the expected predicted result. In other words, textual similarity
annotation were hidden in the test set. Therefore, in order to incorporate this
kind of knowledge into the model, some metrics have been explored, namely
BLEU [
Both metrics were calculated for each pair of sentences resulting in the distributions presented in figures 4 and 5. The figures show the distributions of the metrics according to the classes in the corpus. The results for the BLEU metric, as it is based on the calculation of string overlaps between texts, show greater values for entailment (higher similarity). Accordingly, Edit Distance values are lower for this class, as it computes how di↵ erent a pair of sentences is, i.e. their dissimilarity.
From these analyses, these four features (synonyms and hypernyms counts, BLEU and Edit Distance scores) seem applicable to text inference prediction. Thus, we try to combine these features with the model selected for this task: the BERT language model, as will be discussed later. 5
We analyze how BERT performs on Recognizing Textual Entailment (RTE) and Semantic Textual Similarity (STS) for Brazilian Portuguese in ASSIN 2 corpus.
It is worth noting that our methods were tested on three variations of the original dataset for RTE, which are the runs submitted to the shared task. On the other hand, our submission to the Semantic Textual Similarity task only was tested on the original dataset. 5.1
We use BERT to train the RTE classifier. Specifically, we use a pre-trained BERT model, add an untrained layer of neurons at the end, and train the new model for the RTE task. For this purpose, we use the pre-trained BERT multilingual model that includes Brazilian Portuguese3 along with other 103 languages. The model was trained for 7 epochs with a learning rate of 0.00002, a batch size of 22 and a maximum sequence length of 128 tokens4.
As mentioned previously, linguistic features like BLEU, Edit-distance, number of hypernymns and synonymns between the sentence in a sentence-pair were also used in the model, however, the introduction of these features did not contribute positively to the final result. 5.2
Due to the high correlation between the Entailment class and the similarity values, we used the previous trained model for RTE to obtain embeddings for each sentence-pair. Initially, we experiment using separate embeddings (an embedding for each sentence, obtained through BERT), but the results were poor. Thus, we use joint embeddings (size of 768) as input, obtained by providing both sentences in the pair as an input to the BERT model, which creates a single representation of the whole pair. Additionally, we incorporate the features BLEU, number of synonyms and hypernyms in common for each sentence-pair.
For experiments, we use a multilayer perceptron with a hidden layer (64 neurons), the logistic function as the activation function, the adam optimizer, a learning rate of 0.001 and a maximum number of iterations of 1,000. 5.3
As mentioned, our methods were trained and fine-tuned on three variations of the original dataset. The first variation (”own” in Table 1), consists in splitting the original training dataset (7,000 sentence-pairs) into 6,300 for training and 700 for development. This split has been done by a stratified sampling according to both entailment and similarity values, to guarantee that their distribution is also represented in the development set. The second one was provided by the organization and it contains 6,500 sentence-pairs for training and 500 for development (”assin-2” in Table 1). The last variation comprised all 7000 sentences (”all” in Table 1).
Table 1 shows the results of the best three teams (excluding our team), our results and the baseline results for RTE. In general, our proposal obtained the third place in the RTE task (being only surpassed by the Deep Learning Brasil and IPR teams) and the di↵ erence between our proposal and their proposals is small.
Concerning our proposal, it is important to highlight three regards. Firstly, the split performed by us shows the best results even containing fewer instances 3 The model is available at https://storage.googleapis.com/bert models/2018 11 23/ multi cased L-12 H-768 A-12.zip 4 It is worth noting that other hyperparameters were tested. However, the results were not shown improvements and they are no reported in this paper. Team
Run Deep Learning Brasil IPR Stilingue NILC NILC NILC Baseline Baseline Baseline
Ensemble 1 2 own assin-2 all BoW sentence 2 Word Overlap
Infernal in the training set than the original split, which could note the relevance of the splitting strategy. However, we cannot a rm this due to the small improvements.
Secondly, the introduction of linguistic features (BLEU, edit-distance, among others) did not contribute positively to the final result. Thus, we only fine-tuned the multilingual BERT on our RTE task. In principle, it could be thought that multilingual BERT learned all these features, this way by including them, the results did not improve. Another explanation could be that it is necessary to explore other ways to integrate this kind of information. However, a deeper study must be performed.
Finally, it is worth noting the potential of multilingual BERT. This model made our proposal easier in comparison to other approaches as it only needs the pre-trained model and adding an untrained layer to perform the fine-tuning process on the RTE task.
Concerning the Semantic Textual Similarity (STS) task, our proposal obtained smaller results than the other proposals. However, our results outperformed all baselines. It is interesting to note that fine-tuning multilingual BERT on the RTE task contributes positively to the STS task. However, some linguistic features had to be incorporated to obtain better results, showing that BERT could not learn this kind of information. In that sense, an interesting direction could be experimenting fine-tuning on STS task and then using this information to apply to RTE task or trying to learn both tasks in a multi-task learning approach.
Results
Acc. 88.32% 87.58% 86.64% 87.17% 86.85% 86.56% 56.74% 66.71% 74.18% This work presents the results obtained by the NILC group for the shared task ASSIN 2 on entailment and textual similarity recognition. We analyzed characteristics of the ASSIN 2 dataset according to its entailment classes and tested two approaches of classification using BERT.
Fine-tuning BERT, on the ASSIN 2 corpus without any extra feature presented the best results, largely outperforming the baselines. Therefore, we show that using a simple BERT model can provide satisfactory results in these tasks.
As shown in the corpus analyses, similarity, BLEU and Edit Distance metrics seem to be suitable for discriminating the entailment classes of the ASSIN 2 corpus. In particular, entailment pairs have higher similarity values and higher BLEU scores, as well as lower Edit Distance values, than none class.
The number of hypernyms and synonyms calculated consulting OpenWordNet-PT for each pair also indicates some level of distinction between the two classes, there were considerable more entailment pairs containing these relations. However, incorporating these features as values concatenated to BERT embedding vectors achieved poorer results.
We considered some possibilities for these negative results, such as the way features were incorporated, as a concatenation to BERT embeddings. Another reason may be that BERT embeddings already incorporate such knowledge (similarity, synonym and hypernym relations) within their representation. A future deeper analysis about the incorporation of these features may lead to further conclusions about these hypothesis.