The Semantic Textual Similarity (STS) algorithms have a key role in Natural Language Processing (NLP) studies since it can support various NLP tasks such as Text Summarization and Information Retrieval. Although we found several STS initiatives in the literature, just a few authors explored Siamese Neural Networks (SNN) to solve this problem, especially for the Portuguese language, even considering their lower need for training data and an architecture built for similarity tasks. We defined a set of lexical, semantic, distributional and graph-based feature groups to capture distinct aspects of the text and incorporated to a SNN architecture to perform STS in ASSIN 1 and ASSIN 2 datasets. The experiments indicate positive results since we improved the results of previous attempts of STS using SNNs in Portuguese texts.
The Semantic Textual Similarity (STS) task consists of quantifying the degree of semantic equivalence of one text to another. It is essential for many Natural Language Processing (NLP) research and applications, supporting tasks such as Plagiarism Detection, Text Deduplication, Text Summarization, Information Retrieval and Text Clustering [1].
The Neural Network (NN) architectures have been outperforming traditional Machine Learning (ML) models in several fields of study including NLP. One example of successful NN is the Siamese Neural Network (SNN), which is a type of NN used to calculate similarity in studies like [2–5], it has been successful in various tasks focused on both image, and more recently on text context, achieving good results using less data than other approaches. Additionally, it has shown less susceptibility to overfitting [5].
Copyright c 2020 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The SNNs layers are configurable to any layer type that suits the problem resolution, such as Convolutional Neural Networks or Long-Short Term Memory, the architecture is composed of two or more equal subnetworks that share the same configurations, parameters, and weights, which has the values updated simultaneously during the learning process [6].
Many features have been tested to tackle the STS problem, such as lexical with string-based approaches, semantical with corpus-based and knowledge-based approaches [7], structural syntactic or morphological information, and recently, several studies have used distributional and contextual Word Embeddings (WE).
The shared tasks have an important role in the STS task, for the English language, the SemEval initiatives released various STS tasks over the years (e.g., [1]). The Portuguese language is represented by the ASSIN 1 [8] and ASSIN 2 [9] shared tasks, which focused on STS texts from the journalistic domain. The winning team [10] of ASSIN 1 used an approach in which they combined TF-IDF calculation with WE. The SNN architecture was not fully explored, with just one group of ASSIN 1 exploring it [11] and just a few other studies applying to other languages [e.g., [4–6, 12].
We hypothesize that associating the SNN’s efficacy to train with data limitations (which is often the case in shared-tasks), and the use of a set of lexical, semantic, graph representation and distributional features, it could be able to capture different aspects of the text, establishing a consistent model.
In this work, we present a SNN architecture inputted with lexical, semantic, distributional and graph-based features, aiming to perform STS in ASSIN 1 & 2 datasets. 2 2.1
The ASSIN 1 & 2 datasets are composed of manually annotated Portuguese sentence pairs with their respective similarity/relatedness scores ranging from 1 to 5, where 1 depicts no similarity and 5 depicts equivalence. The ASSIN 1 dataset is divided into two parts, the Brazilian Portuguese (BR) and the European Portuguese (PT), while the ASSIN 2 contains BR sentences only. Table 1 presents the sizes of the datasets while Table 2 depicts some sentence pairs and their respective similarities.
Aiming to conduct exploratory data analysis and compare the data sparsity and density of both datasets, we applied a graph representation algorithm to the data and verified interesting aspects such as low vocabulary volume on ASSIN 2 dataset compared to the ASSIN 1, that despite having more sentence pairs, have less unique tokens (shown in Table 3). When we compare the number of edges of each dataset (see Table 4) it implies the high and low data sparsity of ASSIN 1 and ASSIN 2 respectively.
2342 1967 2542
8787 7090 10327
PT 11075 9282 15249 23673
PT 45218 35075 70965 129143 BR 10058 8675 13757
BR 40039 33441 63710 2.2
Our STS algorithm was based on formerly proposed SNN in [6], but the Manhattan’s distance was replaced with a 50-units dense layer since the addition of new features would not work well with the Manhattan’s distance. We used two 300sized BiLSTM subnetworks and trained for 70 epochs with the mean squared error (mse) loss function.
We decided to use the pre-trained Word2Vec CBOW 300-sized vector from NILC [13] since it was utilized in previous STS works for Portuguese and achieved good results [10, 14] (from now on named NILC Word2vec) . The 100-sized Word2Vec skip-gram vector ID 63 from NLPL WE Repository 1 was used as well, hereafter NLPL ID 63 Word2vec. A 100-sized vector was trained with each of the dataset groups texts using the Word2Vec algorithm [15], to work as our baseline, henceforth ASSIN Word2Vec.
A new parallel dense layer with 100-units was incorporated in the SNN to accommodate five new feature groups: (i) lexical-based similarities, (ii) knowledgebased wordnet tokens distances, (iii) distributional-based WE cosine distance between the sentences, (iv) the sum of the degree centrality of the tokens using a graph created from the dataset, and (v) overlap of common words around the sentences.
The lexical-based feature group is composed of three different equations, computed with the Jaccard index, Dice coefficient and Cosine distance [7]. They represent the overlap of common tokens of both sentences and have shown very similar results to the word overlap metric used as the baseline of the ASSIN.
We used the Open Multilingual Wordnet (OMW) 2 from the Natural Language Toolkit (NLTK) to build our knowledge-based feature group. Three different similarity calculations were used as features, such as the Wu-Palmer similarity, Leacock-Chodorow similarity and shortest path distance that connects the hypernym/hyponym taxonomy.
The third feature group is the cosine distance of both inputted sentences using the WE model selected for each experiment (more details on the different models later in this chapter). Each sentence position is the average position of their words in the WE model, as presented in Figure 1.
Intending to use the information found in the dataset itself as features, a directed graph of each dataset was generated to calculate both the degree centrality of a token and the overlap of common words around the sentences, each node is a token, and each edge is the pair of token (current token, next token), the edges possess some syntactic information.
The degree centrality of a token is the link incident over the token, for instance in Figure 2 the degree centrality of “jovem” is 8, because there are 8 links around it, this metric was selected as a way to measure the significance of the words. We have chosen the degree centrality equation due to the low computational cost and the high importance of words closely related, other centrality equations can be used and need to be tested.
The overlap of common words around the sentences is the number of common words directly around both sentences; the equation is normalized between 0 and 1 by dividing the overlap with the sum of tokens around the sentences.
Our experiments were executed in each of the following dataset configurations: (i) a concatenation of ASSIN 1+2, (ii) ASSIN 2, (iii) ASSIN 1 BR, (iv) ASSIN 1 PT and (v) a concatenation of ASSIN 1 BR and ASSIN 1 PT. Our experimental setup involved the algorithm evaluation using only the WE as features to our SNN, and the WE plus additional hand-crafted features. The performance was evaluated using the same metrics used in the ASSIN, the Pearson correlation (p) and the mean squared error (mse). 3
The results of our SNN algorithm are displayed in Table 5, which distinguishes the scores for each dataset configuration (i.e., ASSIN1, ASSIN2, ASSIN1-PT, ASSIN1BR, ASSIN1&2), WE model (i.e., ASSIN Word2Vec, NLPL ID 63 Word2vec and NILC Word2vec) and features used (i.e., WE plus five features and WE as only feature). The best scores for each dataset configuration are in bold (p value the larger the better, mse the smaller the better).
The five proposed features seem to improve most of the results if compared to the “WE as only feature” runs. The best p score was achieved by the five features model with the NLPL ID 63 Word2vec pre-trained model for all datasets, for ASSIN 2 it scored 0.72 p and 0.65 mse. Our feature selection has improved the scores mostly for the pre-trained WE.
In Table 6, we compare the performance of our current method with the languageindependent approach developed in [16]. The improvement is evident in all dataset configurations.
Table 7 shows a comparison with the five feature groups in isolation and pairs, that were evaluated exclusively with the NLPL ID 63 WE in the concatenation of ASSIN 1 & 2 datasets, giving that the NLPL ID 63 WE achieved the best results in most runs. Differently from previous experiments, which were trained with 70 epochs, this table was trained with only 50 epochs, due to time constraints and that the focus was only to compare the features results. basic feat basic feat basic feat basic feat basic feat basic feat
*lind denotes for the language-independent approach by [16]. *new denotes the best result of our current approach.
The degree centrality and the overlap of common words around are dependent on the generated graph, and the graph for ASSIN 1 and ASSIN 2 have very different aspects, for instance, the low data sparsity of ASSIN 2, shown in Table 3 and Table 4, may have led to the different performances for each WE on Table 5, contrasting with each other distinct dataset performance.
mse mse
The lexical and the WE cosine distance were the less costly features to calculate, while the overlap of common words around was the slower to calculate, despite that they have shown good results, while the WordNet have shown smaller score increase and the degree centrality have not shown good results.
The degree centrality got the worst results, this might be an indication of low representativeness of the significance of words for similarity tasks, nonetheless, other centralities and node influence metrics need to be tested and the graph representation features can be improved with contextual weighting.
One of the difficulties relative to the method selection was due to the lack of annotation guidelines released, therefore some aspects of the text were not fully presented, for instance, if the scores are about sentence relatedness or similarity. For example, in the sentences “O menino está tocando o piano” and “O menino não está tocando o piano” with score “3.0” the sentence is related, but due to the negation the meaning is opposite to each other, this score does not corroborate with the sentences “Não tem nenhum homem executando um truque em uma bicicleta verde” and “Um homem está realizando um truque em uma bicicleta verde” with score “4.8”.
As future work we intend to use at least the three best features of Table 7 with a contextual WE, such as BERT and ELMO, this could improve the scores by taking advantage of the contextual aspects stored in this kind of embeddings. We hypothesize that a contextual WE would improve the results mainly of ASSIN 2, due to the low data sparsity of the dataset, while a distributional WE could fail when dealing with words used in multiple contexts and with more similarity variation. 5
We trained a SNN architecture in association with pre-trained Word Embeddings and a set of features trying to cover different aspects of the text (e.g., lexical, semantic), aiming to perform the Semantic Textual Similarity task in Portuguese texts. The experiments showed promising results since we improved all the last attempts to use SNN for Portuguese STS, and checked each feature contribution by isolating each one, and training separated models.