Analogies between 4 sentences, “ is to as is to ”, are usually defined between two pairs of sentences (, ) and (, ) by constraining a relation holding between the sentences of the first pair, to hold for the second pair. From a theoretical perspective, three postulates define an analogy - one of which is the “central permutation” postulate which allows the permutation of central elements and . This postulate is no longer appropriate in sentence analogies since the existence of ofers no guarantee in general for the existence of some relation such that also holds for the pairs (, ) and (, ). In this paper, the “central permutation” postulate is replaced by a weaker “internal reversal” postulate to provide an appropriate definition of sentence analogies. To empirically validate the aforementioned postulate, we build a LSTM as well as baseline Random Forest models capable of learning analogies based on quadruplets. We use the Penn Discourse Treebank (PDTB), the Stanford Natural Language Inference (SNLI) and the Microsoft Research Paraphrase (MSRP) corpora. Our experiments show that our models trained on samples of analogies between (, ) and (, ), recognize analogies between (, ) and (, ) when the underlying relation is symmetrical, validating thus the formal model of sentence analogies using “internal reversal” postulate.
Analogy plays a crucial role in human cognition and intelligence. It has been characterized
as “the core of cognition” [
Although analogies between words have been extensively studied, analogies between sentences have received very scant attention by the community, to the best of our knowledge. Instead of dealing with words, dealing with sentences leads to 2 challenges: • How to embed sentences in a vector space? • How do we define a sentence analogy?
We expect sentence embeddings to be dense vectors supposed to reflect semantic properties
of a sentence. Various approaches are available: embed each word and get the average of the
vectors. In that case, the order of the words is lost. Another option, described in [
The question of defining sentence analogy is especially delicate. Indeed the aforementioned
parallelogram model used for words reflects the usual postulates of analogies, namely if :
:: : holds, : :: : (symmetry) and : :: : (central permutation) should hold
as well. This latter postulate (already questionable between words [
John sneezed loudly (a). Mary was startled (b).
Bob took an analgesic (c). His headache stopped (d).
In that case, the implicit relation between sentences in a pair is a kind of causal relation. This example indicates that central permutation makes no sense here and raises the question of defining a weaker notion of analogy obeying another system of postulates. By which postulate to replace the central permutation? In this paper, we propose to introduce a postulate we call “internal reversal” that expresses that if : :: : holds then : :: : holds as well, and we study its consequences. So our main goal is to: • theoretically investigate the formal consequences of this new model, • empirically validate the model by implementing various classifiers of sentence analogies.
After presenting in Section 3 the standard formal definitions of analogies, including the “central permutation” postulate, and their immediate consequences, we focus on the replacement of the “central permutation” postulate by the internal reversal postulate. Having a better fit with what is accepted as sentence analogies in the NLP community, this postulate also impacts the machine learning perspective that we implement.
For natural language sentences, “internal reversal”, as a formal postulate, may have some limitations. For instance, if = −1, where is the common relation that holds between two pairs of sentences (, ) and (, ) (e.g. , is a paraphrase of ), one would expect that internal reversal holds straightforwardly. In that case, a machine learning model trained to recognize : :: : , should also recognize : :: : .
We investigate the conditions under which a machine learning model containing quadruplets
of sentences (, , , ) representing positive and negatives instances of analogies, is capable
of identifying analogies for which the operation of internal reversal has been performed. We
have then devised several series of experiments using various underlying models and datasets.
The paper is structured as follows. After reviewing the related work (Section 2), in Section
3 we recall the formal definitions of analogical proportions and investigate the new case of
sentence analogies, suggesting “internal reversal” postulate as a better fit and examining its
consequences. In Section 4, we consider the consequences of the formal definition in a machine
learning perspective, by suggesting a rigorous extension of an initial training set. Sections 5 and
6 are dedicated to the description to the context, protocol and results of our experiments. This
work is an extension of [
Due to the advent of neural models and distributed representations of words, lexical analogies
have been the focus of various works in computational linguistics. [10, 11, 12, 13, for example].
In terms of analogies on the sentential level few works exist. [
(||( − ) − ( − )||) . The authors limit the question/answer pairs to ℎ − questions from WikiQA and TrecQA. They use a Siamese bi-GRUs as their architecture to represent the four sentences. In this manner, the authors learn embedding representations for the sentences which they compare against various baselines including random vectors, word2vec, InferSent and Sent2Vec obtaining better results with the WikiQA corpus. Most of the tested sentence embedding models succeed in recognizing syntactic analogies based on lexical ones but had a harder time capturing analogies between pairs of sentences based on semantics.
Instead of training a model to select the best candidate amongst a given set of candidates
([
In this paper, the aim is to empirically validate the “internal reversal” postulate (without focusing on accuracy). To our knowledge, such a study has not been conducted before.
We briefly recall the formal definition of analogy such as found in [
Given a set of items , a (proportional) analogy is a quaternary relation supposed to obey the 3
following postulates (e.g.,[
∈ : 1. : :: : (reflexivity ); 2. : :: : → : :: : (symmetry); 3. : :: : → : :: : (central permutation).
These postulates have straightforward consequences like: • : :: : (identity); • : :: : → : :: : (internal reversal); • : :: : → : :: : (extreme permutation); • : :: : → : :: : (complete reversal).
Among the 24 permutations of , , , , the previous postulates induce 3 distinct classes each containing 8 distinct proportions regarded as equivalent due to postulates: : :: : has in its class : :: : , : :: : , : :: : , : :: : , : :: : , : :: : , and : :: : . But : :: : and : :: : do not belong to the class of : :: : and are elements of the two other classes.
In the NLP community, the 4 items , , , are sentences in natural language, not necessarily the same. It is widely admitted that the sentences are in analogy (i.e., : :: : ) as soon as there is a relation , the relation between sentences, such that (, ) and (, ). The example from the introduction is a perfect illustration of this definition where the relation is just causality:
John sneezed loudly (a). Mary was startled (b).
Bob took an analgesic (c). His headache stopped (d).
But:
Il fait beau aujourd’hui (a). Today we have nice weather (b).
Il vaut mieux eviter la guerre (c). It is better to avoid war (d).
is another example of analogies between sentences where the implicit relation is “ is the
English translation of the French sentence ”. From a logical viewpoint, this can be expressed
as:
: :: : if ∃ s.t. (, ) ∧ (, )
(1)
where ∧ is just the formal notation for the connector. This definition can be considered
as quite vague because, as advocated in [
John sneezed loudly (a). Mary was startled (b).
Bob took an analgesic (c). His headache stopped (d).
Definition 1 or 2 still applies to : :: : :
Bob took an analgesic (c). His headache stopped (d).
John sneezed loudly (a). Mary was startled (b). which is then a valid analogy. Nevertheless, central permutation is not satisfied with the above definitions 1 or 2.
Let us now focus on the “internal reversal” postulate as a alternative to “central permutation”: : :: : → : :: : (internal reversal) By definition, if (, ) holds then −1(, ) holds. Definition 1 supports “internal reversal”: for instance, if relation (, ) is interpreted as “ is a cause of ", −1(, ) can be the passive form “ is a consequence of a”. But Definition 2 does not support “internal reversal” except if, for each relation in the set of built-in relations, we also have its counterpart −1. A simple way to ensure this property is to consider relations such that = −1. For instance, (, ) is defined as “ is a paraphrase of ”.
In the general case, a proper definition of a sentence analogy supporting the 3 postulates (reflexivity, symmetry, internal reversal) would be: : :: : if ∃ ∈ s.t.
︃{ ( (, ) ∧ (, )) ∨( −1(, ) ∧ −1(, )) (3)
1. : :: : (reflexivity ); 2. : :: : → : :: : (symmetry); 3. : :: : → : :: : (internal reversal).
As immediate consequences, we get that : • there are only 4 equivalent forms (instead of 8 with the central permutation postulate) for an analogy: • : :: : , : :: : , : :: : , and : :: : . • : :: : → : :: : (complete reversal). • : :: : (full identity) is still satisfied.
• : :: : (identity) is no longer a consequence of the new postulates.
Let us assume that we have at our disposal a repository of pairs of sentences (, ) with their associated relation . From this repository, we need a training set of examples for the classifier. Given the previous section, several steps can be implemented.
1) Building an initial training set of analogies : :: : can be done by joining 2 pairs (, ) and (, ) belonging to the same relation . This constitutes a set of positive examples + such that for every quadruplet (, , , ) = x ∈ + the training instances are {x, } with = 1. In terms of negative examples, joining 2 pairs (, ) and (, ) belonging to diferent relations leads to build a set of negative examples − such that for every quadruplet (, , , ) = x ∈ − the training instances are {x, } with = 0. The training set = + ∪ − is then a set of quadruplets of sentences , , , such that:
) ∈ − • if the implicit/explicit relation between the pair (, ) also holds for the pair (, ), then (, , , ) ∈ + • if the implicit/explicit relation between the pair (, ) does not hold for the pair (, ), then (, , ,
Applying symmetry postulate allows to double the size of +, just by adding (, , , ) ∈ + as soon as (, , , ) ∈ +. We then improve the theoretical unbalance between + and −.
2) The same method applies with internal reversal postulate, by adding (, , , ) ∈ + as soon as (, , , ) ∈ +. This again doubles the size of +.
At this stage, we have multiplied by 4 the initial size of our positive training set + by introducing common sense analogies deducible from the initial ones, but not necessarily related to the initial list of relations . Can we do more? The Identity Relation For completeness sake, one could argue that it is still possible to extend the set of positive examples since it seems acceptable to consider : :: : as a valid sentence analogy even though identity relation likely does not belong to . But identity relation holds between the pairs (, ) and (, ). Although recognition of analogies based on the identity relation might seem trivial from an NLP perspective it could still be a useful task in case that we want to evaluate the quality of our classifier. In other words, if a potential classifier is not able to identify analogies based on the identity relation, one should probably reconsider the underlying approach.
The Inverse Relation A scenario that appears quite often in Natural Language Processing, although far from being a generalized phenomenon, is that a relation between sentences (or larger proportions of text for that matter) is its own inverse −1.
Instances of such a relation can be, for example, that of the paraphrase. If is a paraphrase of , obviously is a paraphrase of . The same hold for the operation of translation. If sentence is a translation of then again is a translation of . Following our initial definition of analogy, we will have to accept: : :: : when R is its own inverse
Before moving to the details of the empirical validation, we describe the datasets we use in the following section.
following sets of experiments.
As explained earlier in this paper, our main goal is the empirical validation of internal reversal for sentential analogies, using various corpora. To investigate this postulate we devise the and a test set (
) = ({x } =1, { } =1) with typically being a tenth of and x representing a quadruplet of sentences : :: : 3 and ∈ {0, 1} we learn a model ℋ capable of identifying analogies with a certain accuracy. Crucially, |{ : = 1}|= |{ : = 0}| both for training and testing sets. Due to the huge size of instances at our disposal, there is no need at this stage to implement any further data augmentation process, as explained in the previous Section. In other words, we have an equal number of positive and negative instances in training and testing sets, for a total of 4M instances.
Internal reversal on the test set (Experimental setting 1) In this series of experiments, we used the same training set ( construct the test set, we perform internal reversal on all the instances of the train set that we have used in base setting. Our goal is to see whether we get similar results on analogies for the internal reversal.
Test set from train distribution with internal reversal (Experimental setting 2) For this series of experiments we use the same training set ( the base setting ℋ . The test set though is constructed in the following way: for every positive
) = ({x } =1, { } =1}) for testing set (
) we add the internal reversal pair (x : :: : , 1) to the new ) whose size thus is / 2. In contrast to experimental setting 1 where the underlying sentences between train and test distributions are diferent, in this series of experiments we want to see how well a trained model can detect analogies after performing internal reversal on the same set of pairs of sentences. 3Henceforth, we will denote a representation for a quadruplet of sentences : :: : by the vector x : :: : . Augmenting training and test sets (Experimental setting 3) In the series of experiments we learn a model ℋ using (
( (x : :: : , 1) in train or test set we add the following instance (x : :: : , 1). In other words, we double only the positive instances by adding the internal reversal of a quadruplet as a positive Augmenting test set (Experimental setting 4) In this series of experiments the train set and thus the model learnt is the same as the base setting. In other words, we have a training set have a new test set ( , ) = ({x } =1, { } =1) which results from ( base setting by keeping only positive instances. This subset is then augmented with instances ) of the
) = ({x } =1, { } =1}) from which we learn a model ℋ . For testing though we (x : :: : , 1) for every instance (x : :: : , 1) we have in that subset, resulting thus in total and a test set instance. positive instances.
To perform our experiments, we used three corpora: Penn Discourse TreeBank (PDTB), Stanford Natural Language Inference Corpus (SNLI) and the paraphrase dataset MPRC.
The first dataset that we use is PDTB version 2.1[
SNLI is a corpus of pairs of sentences from [
There are well-known word embeddings such as word2vec [
- The simplest method is to average the word embeddings of all words in a sentence. Although this method ignores both the order of the words and the structure of the sentence, it performs well in many tasks. So the final vector has the dimension of the initial word embedding.
- The other approach, suggested in [
Random Forest (RF) We have tested our hypothesis on a classical method successfully used
for word analogy classification [
Bi-LSTM architecture Given a quadruplet of sentences : :: : which can be an analogy or not, we represent each sentence by its input tokens = { 1 , . . . , }, = { 1, . . . , }, = { 1, . . . , } and = { 1 , . . . , }. Although sentences can have different lengths we have empirically fixed = 35; if a sentence has less than 35 word tokens we use padding. Each word token (with ∈ {, , , } and ∈ [1 . . . ]) is represented by a Glove vector of 300 dimensions. In this series of experiments, LSTM did not use averaging or DCT since the recurrent nature of LSTMs themselves accounts for the structure of a sentence. Our architecture is composed by four Bi-LSTMs whose output is passed over to a feed-forward network. More precisely, for each sentence we recursively calculate ℎ = ⊗ ℎ ( ) with ⊗ representing the Hadamard operation and = (W · [h −1, x ] + b ) = ⊗ −1 + ⊗ C˜ = (W · [h −1, x ] + b ) ˜ C = ℎ
(W · [h −1, x ] + b ) = (W · [h −1, x ] + b ) h = (W h
+ b) h
−→ ←− −→ = −ℎ→ ++ ←ℎ− ++ ℎ ++ ℎ ++ −ℎ→ ++ ←ℎ− ++ ℎ ++ ℎ
←−
In the above, x represents the vector for token in a given sentence. These representations are obtained on both directions. Thus for each sentence the following representations are −→ ←− −→ = { } = −ℎ→ ++ ←ℎ−; = { } = ℎ ++ ℎ ; = { } = −ℎ→ ++ ←ℎ−; = { } = ℎ ++ ℎ ←− with ++ representing the concatenation operation.
The above representations are given as input to a single layer feed forward network: where and obtained: with using Rectified Linear Unit (ReLU) as activation function. Finally, the prediction is performed using a sigmoïd function: ˆ = (W h + b) =
1 1 + −W h +b The architecture is guided by a standard binary cross entropy loss function.
Results of our experiments for LSTMs and RFs are shown in Tables 1 and 2 respectively. In all cases, we randomly generated quadruplets (, , , ) which we annotated as analogies (class 1) if pairs (, ) and (, ) shared the same relation, or with class 0 if they did not. For PDTB and SNLI we randomly generated 2 million instances for training; testing and development corpus contained 200.000 instances each. For the paraphrase corpus, we generated 4 million instances for training and testing and development corpus contained 200.000 instances each. Each dataset contains an equal number of positive and negative instances. As we can see, base settings for all datasets perform quite moderately, which is to be expected since our aim was not to create a general model for sentential analogies, which would require much more data and powerful models with billions of parameters. Instead, our goal was to examine under which conditions internal reversal holds. As we can see, in experimental setting 1, for which the test set is the same as the train but with internal reversal, results on PDTB and SNLI, which contain relations that are not symmetric, are worse than the base setting. This is not the case though for the paraphrases corpus for which results are better to the base setting.
In the second set of experiments, we decided to focus solely on the positive instances and examine if learning analogy : :: : also implicitly learnt internal reversal, that is : :: : . We used the same base setting that we have learnt, but for testing, we created a new dataset. Starting from an empty set, we took every positive instance of the training set and performed an internal reversal; we then add it to the new test dataset. The resulting dataset has no common instances with the training dataset, but every instance of it is an internal reversal of the positive instances of the training set. As we can see there is almost no diference in scores for PDTB and SNLI, but the results for the paraphrases corpus (93.412% 1 for LSTMs and 87.544% for RFs) clearly show that when a relation is symmetrical the model almost makes no diference between an analogy and its internal reversal. It is interesting to observe that the trend for LSTM is similar to RF. However, the results from LSTM appear to be more stable. In the third series of experiments, we augmented both the training and testing datasets with the internal reversal. All three datasets showed a significant increase—of almost 20 percentile points for some cases—for the detection of analogies. In the fourth and final set of experiments, we used the same base setting that we had used initially. The test set was constructed based on the same test set as the base setting but we removed all negative instances and focused solely on the positive ones augmented with the internal reversal. Again here we can see a significant increase in the results for the detection of analogies, further showing that the model learns internal reversal as well. On Table 1 , Experiment setting 2 for MRPC has the highest F1: this corpus has more symmetrical relationships when compared to PDTB and SNLI. We observed with Table 2 the trend already observed with LSTM: Experiment setting 2 has the highest F1. base setting ccllaassss 10 5543..237242 Exp. Set. 1 ccllaassss 01 4498..29514 Exp. Set. 2 class 1 100.0 Exp. Set. 3 ccllaassss 10 7404..10638 Exp. Set. 4 class 1 100.0 base setting ccllaassss 10 6677..885662 Exp. Set. 1 ccllaassss 01 5500..11111 Exp. Set 2 class 1 100.0 Exp. Set 3 ccllaassss 10 8648..438695 Exp. Set. 4 class 1 100.0 base setting ccllaassss 10 5534..46571 Exp. Set. 1 ccllaassss 10 8806..445246 Exp. Set. 2 class 1 100.0 Exp. Set. 3 ccllaassss 10 6398..083332 Exp. Set. 4 class 1 100.0
Accuracy
In this paper, we have suggested a new formal model dedicated to sentence analogies, replacing the standard model for word analogies. A weaker “internal reversal” postulate takes the place of the well-known “central permutation” postulate. From a purely formal viewpoint, we have investigated the consequences of this new model and to what extent it fits with sentence analogies. To validate this approach in practice, we have implemented sentence analogies classifiers, using well-known machine learning algorithms. We have also designed two machine learning protocols involving diferent ways to build a training set, all derived from the formal expected properties. Our results show that an “internal reversal” sentence analogy is recognized by our algorithms as a valid analogy as soon as the underlying relation between sentences is symmetric (e.g. “to be a paraphrase of”). When this relation is not symmetric (e.g., “to be a consequence of”), “internal reversal” sentence analogies are not always recognized. Maybe, in the general case, learning is not the same as learning −1. Alternatively finding a more accurate postulate might be a valid track of research for the future. Analogy postulates could also be used for further constraining the classifier.
The authors would like to express their gratitude to the anonymous reviewers for their valuable comments. They would also like to thank the organizers of this workshop.