Analogies have been characterized as fundamental to abstraction, concept formation, and perception, and are traditionally expressed as quadruplets in the form of proportional analogies : :: : read “ is to as is to ”. While Natural Language Processing (NLP) has primarily focused on word analogies and SAT problems, recent research has started exploring analogies between sentences and even documents. In this paper we explore the potential of identifying analogies between pairs of sentences via the identification of common latent relations between them. We exploit three diferent datasets generating pairs of sentences which can either share the same latent relation-forming thus an analogy-or not. We encode phrases into a higher dimensional vector space using embeddings from GloVe, BERT, and RoBERTa which we then feed to both a Multi Layer Perceptron (MLP) and a Convolutional Neural Network (CNN). Results show that architectures using contextual embeddings as inputs outperform those based on static embeddings.
Analogies have preoccupied humanity at least since antiquity [1]. In recent years they have
been characterized as being at “the core of cognition” [
Traditionally analogies have been expressed as a quadruplet : :: : read “ is to as is to ”. Such quadruplets then form valid analogies if pairs (, ) and (, ) share the same underlying relation, forming thus a proportional analogy. The underlying relation has been viewed as the symbolic counterpart of arithmetic or geometric proportions: − = − and
In Natural Language Processing (NLP) various approaches adopt the framework of quadruplets
focusing mostly on word analogies, such as man is to woman as king is to queen [
In this paper we are interested in further exploring the potential of analogies between
sentences via the identification of common latent relations between them. We exploit three
diferent datasets, namely the Microsoft Research Paraphrases Corpus (MSRP) [
The rest of the paper is structured as follows. In Section 2 we present the related work. In Section 3 we present the datasets that we have used in order to perform our experiments. The methodology for these experiments is described in Section 4 while the results are presented in Section 5. We conclude in Section 6.
Initial work on analogies in NLP was performed by [
More recently Mikolov et al. [
In terms of word analogy classification [
Recently, several researchers have explored sentential analogies. [
In another approach, [
In order to perform our experiments we used three well known datasets. In what follows we provide a detailed description of the corpora used as well as the procedure which lead us to the creation of analogical quadruplets that were later used in our experiments. We should mention that we used the input datasets as they were released, no further additions or modifications were performed from us.
The first corpus that we used was the Microsoft Research Paraphrases Corpus (MSRP) [
A Support Vector Machine-Classifier (SVM-Classifier) is then used to identify a set of possible paraphrases from the 49375 sentences pairs. This set is validated by human annotators later. The SVM-Classifier is trained on a 10000 sentences pairs training set annotated by 2 human judges, and a 3rd who served the function of judge in case of disagreement. The distribution of this training set is 2968 positives examples and 7032 negatives. The classifier considered multiples features: string similarity, morphological variants, synonyms mapping with WordNet Lexical Mapping and Encarta Thesaurus, and finally composite features. The SVM-Classifier allowed to extract 20574 sentences pairs as possible paraphrases from the 4959375 previously considered. The number is high because the classifier’s role was to separate possible sentences pairs to be evaluated by human judgment and not discriminate all non-paraphrases pairs, so the classifiers tend to classify inputs as positive rather than negative, at the assumed cost of having more false-positives.
Human judgment was applied to a 5801 subset of the 20574 previously extracted sentence
pairs. Two judges annotated each sentence and a third one was used in case of disagreement.
Each judge was asked if the pairs’ sentences were semantically equivalent. About 3900 (67%) of
the sentences pairs were labeled as semantically equivalent.
3.2. PDTB
The second corpus that we used was the Penn Discourse TreeBank (PDTB) [
The inter-annotator agreement was high: 90,2% for explicit relations and 85,1% for implicit
when exact match metric was considered; and respectively 94,5% and 92,6% when partial
match metric was considered. Class level disagreement was resolved by a team of 3 experts,
disagreement at lower levels were resolved by providing a tag for the direct higher level.
Agreement for the class level reached 94%, 84% for type level and 80% for subtype level.
3.3. SNLI
The Stanford Natural Language Inference (SNLI) corpus [
The indeterminacies of event and entity co-reference are two well known issues during labeling of NLI data degrading the quality of the annotated corpus. They represent respectively a possible confusion between an Entailment and a Neutral relation, and between a Contradiction and a Neutral relation. This confusion comes from the fact that an assumption may or not have been made.
In order to solve this problem the annotation process was made in a grounded scenario aiming to reduce assumptions. Annotators were then able to generate sentences in the same scenario in order to illustrate the relations instead of relying on automatic data augmentation techniques. The work of 2500 employees permitted the data collection phase. When presented with an image caption without the matching image, the annotators had to write three sentences, one for each relation (the exact instructions are described in the SNLI paper). The image captions came from the Flickr30k corpus containing 160k captions from 30k individuals images. The validation phase is completed on 10% of the 570k pairs of sentences by a set of 30 trusted workers. They were presented pairs of sentences and had to label them, each pair being presented to 4 annotators so there is 5 judgments considering the label from the data collection phase. The gold-label has been assigned to the pairs with at least a 3-annotators consensus, representing 98% of the data. The corpus is then separated in three individual files : test and dev (10k pairs each), train (the rest of the pairs).
In order to create our analogical quadruplets we proceeded as follows. For each of the aforementioned datasets we randomly selected two pairs of sentences each one linked with a relation. Since our input datasets do not contain relations that have as arguments the same sentences, we never have analogies of the form : :: : . In case the relation linking the two pairs is the same we have a positive instance of an analogy otherwise a negative instance. For the SNLI corpus we considered neutral as not being a relation. For each input dataset we create a balanced training, test and development datasets containing the same number of positive and negative instances. Training consists of 400K instances while testing and development 40K instances each.
Our problem can be formalized as follows. Given a set of quadruplets of sentences : :: :
which can either form an analogy (pairs : and : share the same latent relation) or not we
need to estimate a function that predicts whether a new instance of four sentences is an analogy
or not. Each quadruplet is represented by the input tokens of its sentences = { 1, . . . , | |}
with ∈ {, , , } and | | representing the length of the sentence. With each quadruplet we
associate a ∈ {0, 1} which represents whether the quadruplet is an analogy or not. For each
quadruplet we obtain embeddings using GloVe [
In order to perform classification we need to provide embeddings for each sentence. In the case of GloVe3 static embeddings are provided for each word, while in the case of BERT4 and RoBERTa5 embeddings are dynamic. In order to obtain embeddings that represent sentences from the ones representing words a common approach [20, for example] is to take the mean of the embeddings representing each word. This is the approach that we have used as well. For 2Our code is available at https://github.com/ThomasBARBERO/EXPLO_ANALOGIE 3The Glove embeddings that we used are the following: https://huggingface.co/sentence-transformers/average_ word_embeddings_glove.6B.300d 4https://huggingface.co/bert-base-uncased 5https://huggingface.co/roberta-base each sentence ∈ {, , , } we obtain an embedding with emb ∈ {glove, bert, roberta}. Thus four diferent embeddings a, b, c and d are obtained for each of the sentences , , and . In the case of BERT we have also examined the use of the representation obtained for the final hidden state of the special symbol [CLS]. Embedding dimensions for each method are shown in Table 1. No further fine-tuning was performed on BERT or RoBERTa.
Multi-layer perceptron (MLP) The first classifier that we use is a multi-layer perceptron. The MLP takes as input the concatenation of the representations for the four sentences a, b, c and d as a vector [a; b; c; d] and has two hidden layers, the first has a dimension of 100 and the second of 50.
z1 = W1 [a; b; c; d] + b1 z2 = W2 z1 + b2 ^ =
1 1 + −z2 with W1, b1, W2, b2 learnable matrices. The output layer is of dimension 1 and we use a sigmoïd function providing a score for the final prediction Convolutional Neural Network (CNN) The second classifier architecture that we have used are Convolutional Neural Networks which are widely used for image and audio processing, but are useful for Natural Language Processing tasks too as well, including analogies [27, 10, inter alia]. CNNs aim to recognize patterns, extracting features from the initial tensor given as input. The core of CNNs is their convolutional layers applying filters called kernels on the whole input. Kernels’ weights and bias are learnt parameters, convolution between a kernel and a tensor allows to extract a learnt feature from the tensor. Parameters define the method for the application of kernels: the kernels’ size, the stride which indicates the spatial distance between two kernel applications and padding which indicates the number of pixels to add on the considered tensor’s borders. Our CNN’s implementation is as follows, illustrated in Fig. 1: 1. The input goes through a first convolutional layer with 2×1 kernels and 2×1 stride allowing to firstly get feature maps for the pairs (, ) and (, ). At the end of this process (, ) and (, ) are reduced to one dimension regarding the width while the other dimension represents the embedding size. The size of the output is 2 × _ . 2. We fed the output to a second convolutionnal layer with 2 × 2 kernels and 2 × 2 stride so the features maps of (, ) and (, ) are now reunited in one single dimension across width, the embedding size is divided by 2 too. 3. We then apply dropout and feed the output to a singular linear layer, we use sigmoid activation function to compute a confidence score for the 2 sentence pairs being in an analogical proportion relation.
For both architectures we ranged learning rate between 10−4 and 10−5, and dropout from 0.1 to 0.3. We used Adam optimizer with default PyTorch settings and Binary Cross Entropy Loss. Results for both architectures and combinations of embeddings are shown in Table 2.
Transformer-based Language Models outperform GloVe almost constantly in terms of accuracy and F1-score (ability to recognize valid analogical proportions). While the scores are not significantly higher, we can still conclude that contextual embeddings provide better handling of latent relations and analogies between sentences in comparison to static embeddings. Let us note also that representing a sentence by the mean of its contextual word vectors outperforms the CLS sentence representation. Precision Recall F1
Accuracy
Precision Recall F1
Accuracy GloVe-mean BERT-base-mean BERT-base-CLS roBERTa-base-mean ccllaassss 10 roBERTa-base-CLS GloVe-mean BERT-base-mean BERT-base-CLS roBERTA-base-mean ccllaassss 10 roBERTa-base-CLS GloVe-mean BERT-base-mean BERT-base-CLS roBERTa-base-mean ccllaassss 10 roBERTa-base-CLS
Overall scores for the SNLI dataset are the highest with accuracy ranging from 62.708 to 68.01 and F1-score peaking at 68.2 across MLP and CNN. Scores for MRPC are a bit lower with accuracy ranging from 57.537 to 63.992 and F1-score peaking at 62.662 considering CNN only as it constantly outperforms MLP. The classifiers had a harder time grasping analogies on the PDTB corpus with accuracy ranging from 53.773 to 56.47 and F1-score peaking at 57.564, F1-score being below 50 for roBERTa-base-CLS/CNN and GloVe-mean/MLP. The classifiers had a harder time grasping analogies on the PDTB corpus with accuracy ranging from 53.773 to 56.47 and F1-score peaking at 57.564, F1-score being below 50 for roBERTa-base-CLS/CNN and GloVe-mean/MLP. This can be explained by the fact that the number of latent relations that we had to handle in PDTB is much higher (5 latent relations) than the latent relations that we have in the MRPC or the SNLI corpora. We assume that providing more data will yield better overall results.
One main diference between BERT and roBERTa is respectively the presence and absence of the Next sentence prediction training task. While BERT considered this task to be beneficial for the learning of long range dependencies roBERTa considered this task counter-productive. Although roBERTa performs slightly better than BERT (considering the mean-pooling sentence representation method) we cannot draw a definitive conclusion about the utility of the Next Sentence Prediction training task for analogical properties learning. A bigger training set may have enforced the tendency. roBERTa outperformed BERT for SNLI and MRPC, the two corpora for which the sentences from the sentence pairs do not follow each other in a natural context. The next sentence prediction may be detrimental in this case. 5.4. CNN vs MLP Both MLP and CNN are relevant for the classification task we performed as they have almost similar results with the CNNs usually outperforming the MLPs, but not always. However the MRPC’s results show a large diference in performance between the 2 classifiers, the CNNs performing significantly better than the MLPs. Meaningful features were extracted from the sentences representations. As we described Section 4 features are first extracted from the (, ) pair and the (, ) pair in tandem. This could probably be attributed due to teh fact that paraphrases use semantically similar words which probably are closer to the vector space which is better captured by CNNs than MLPs, although further analysis is needed in order for this claim to be verified.
In this paper we have focused on the problem of identifying analogies between pairs of sentences based on common latent relations that exist or not between the pairs. We have used both contextual embeddings (BERT en roBERTa) as well as static embeddings (GloVe). Both BERT and roBERTa outperformed GloVe at the binary classification task we performed. We believe an error analysis or a diferent classification task might shed more light on those results. In conclusion this work scratches the surface of Transformer-based Language Models’ ability to encode analogical properties. Our experiments show that embeddings issued from Transformerbased architectures can better capture analogies via the identification of common latent relations, in comparison to static embedding approaches. Nonetheless it is premature to conclude that such architectures can indeed capture more broadly the mechanism of analogy making.
The authors would like to thank the anonymous reviewers for their thoughtful and constructive comments. This work has been partially funded by the ANR AT2TA project, grant number ANR-22-CE23-0023. [1] Aristotle, Poetics, 384—322 BCE.