We propose a method for solving sentence analogies using an embedding-to-embedding method. The method involves the pretraining of an autoencoder with a denoising decoder that generates sentence embeddings and reconstructs sentences. To generate solutions to analogical equations in the sentence embedding space, we introduce a network architecture that learns analogy properties from the dataset instead of relying on predefined formulas. The embeddings of the solutions are then decoded back into sentences using the decoder of the pretrained autoencoder. We conduct experiments on a set of semantico-formal analogies and purely-formal analogies datasets in English, French, and German. The results show that our method achieves state-of-the-art performance in most cases and to some extent provides evidence of the limitations of the 3CosAdd formula in handling longer sentences.
please tell us about : please tell me about :: what do you expect : what do you expect it. it. us to do? me to do? he never saw his : theer naegvaeirn.saw his sis- :: thheenreavgearinsa.w his fa- : he never saw his brother again. mother again. semantic changes, such as the brother corresponding to the sister and the father corresponding to the mother. The ratio in this example contains some gender-related information. We refer to such examples as semantic analogies.
The above two important concepts indicate that we can deduce the fourth term based on any three terms of the quadruplet. This property has led to the gradual application of analogy to some natural language processing related tasks, such as natural language inference, question answering, and machine translation, especially EBMT (Example-based machine translation).
The application of analogies in natural language processing mainly involves two tasks that
need to be addressed. The first one is analogy detection, i.e., determining whether a quadruple ,
, , and constitute an analogy. Since the concept of analogy still lacks a standard definition,
we mainly refer to the analogy property proposed by [
The second primary task is analogy solving, the process of giving , , and in a quadratic group to obtain . That means we need to find the solution to the analogical equation: : :: :
⇒ = ?
Currently, in recent years, methods mainly rely on vector representations of sentences in embedding space. The approach involves using the parallelogram rule (if − = − , then = − + ) to find four vectors that satisfy the analogy property and simultaneously ifnd the solution of the analogical equation in the embedding space.
After obtaining the embeddings of the solutions in the embedding space, a commonly used
approach is to employ retrieval-based methods. These methods involve providing a set of
candidate sentences and retrieving the most similar sentence to the target based on metrics like
cosine similarity. One example of such a method is the 3CosAdd method [
Inspired by the work of [
In the remaining sections of this paper, we first introduce the related work in solving analogies, particularly sentence analogies, in Section 2. In Section 3, we describe the main approach we adopt, namely the embedding-to-embedding method. In Section 4, we present the experiments and results. In Section 5, we provide an overview of the contributions of this paper and propose further directions for future research.
In this paper, we primarily focus on solving sentence analogies, which involve deriving an
unknown sentence given known sentence analogies , , and . However, we can still draw
inspiration from recent word analogy tasks. As mentioned in Section 1, some retrieval-based
methods like 3CosAdd rely on predefined formulas and expected properties of the embedding
space. Their goal is not to learn the properties of analogies from existing actual data so as to
solve analogy. [
Unlike word analogy tasks, sentence analogies are more diverse and complex in terms of
vocabulary, syntax, and semantics, making them more challenging to solve. However, a sentence
can still be seen as a whole composed of multiple words. [
[
noise sentence embedding sentence decoder ~ + encoder sentence
sentence embedding space A B
Ratio C
Output predicted D
Ratio Extraction Network
Conformity Mapping Network
Offset network for solving analogies from the sentence vector. Additionally, they designed a simple linear fully connected network FCN to learn the mapping of analogical equation solutions in the embedding space. They tested diferent ways of combining vectors in semantico-formal analogy dataset, and the calculation formula = − + in 3CosAdd ultimately achieved the best performance.
Inspired by work of [
Similarly as in [
The method used in [
Additionally, because the task of the decoder is to decode embeddings that satisfy the constraints of analogical equations, there may be slight deviations in the numerical values of the generated embeddings, whether produced by neural networks or predefined formulas, compared to the true reference embeddings. This can cause the decoder to struggle in correctly decoding these embeddings. Therefore, during the training process of the autoencoder, we introduce a certain proportion of Gaussian noise to the sentence embeddings generated by the encoder, aiming to train the decoder to produce accurate sentences. This approach enhances the decoder’s robustness to small perturbations along the target embedding manifold, expands the range of manifolds the decoder can correctly decode, and mitigates overfitting to some extent.
After completing the pre-training of the autoencoder, we can obtain sentence embeddings using the well-trained encoder. Within the generated embedding space, we propose an Ofset network structure to learn predicting embeddings that satisfy the constraints of analogical equations. This neural network is based on two important concepts of analogies: conformity and ratio, and it is divided into two parts: the ratio extraction network and the conformity mapping network.
The ratio extraction network, the first part of the Ofset network, learns the ratio relationship in the analogy by taking the embeddings of sentences A and B as inputs.
The conformity mapping network, the second part, learns to map the ratio and the embedding of sentence C to obtain the embedding of sentence D.
These two parts of the network have a simple structure, consisting of only one layer of convolutional network and one fully connected layer. To some extent, this network structure achieves the ofset of embedding C by ensuring the conformity of the ratio between two binary tuples in the analogy. Hence, we refer to it as the Ofset network . Our expectation is that it can learn the properties of analogies from the dataset and solve analogies without relying on predefined formulas.
In our experiments, we want the generated sentences and the reference sentences to be as
similar as possible. So we use BLEU [
For decoding sentence embeddings, we keep the decoder part of the autoencoder consistent
with the decoder in [
During training, we employed MSE as the loss function.
During the pre-training of the autoencoder, we set the ratio of Gaussian noise added to the sentence embeddings as 0.1. Additionally, the dimension of the sentence embeddings was set to 300. The results on the three languages are shown in Table 2. In terms of accuracy, using sentence embeddings generated by the encoder of the autoencoder outperforms the simple summation approach by nearly 30% in all three languages. For English sentences, which are shorter with a smaller vocabulary, the decoding accuracy reaches 91.1%. Additionally, the 0.1 Levenshtein distance indicates that, on average, less than one word is incorrect when decoding English sentences. As for French and German, which have longer sentence lengths and vocabulary sizes two to three times larger than that of English, the decoding performance decreases slightly, but the decoding accuracy using encoder-generated sentence embeddings still surpasses the simple summation approach by a considerable margin more than 30%. en fr de en fr
de 100 80 EU 60 L B 40 20 0 5 EU 60 L B 100 80 40 20
Additionally, we investigated the impact of sentence length on the decoding of sentence embeddings. As shown in Figure 2, we can observe that both methods experience a gradual decrease in decoding performance as sentence length increases. However, the approach of using encoder-derived sentence embeddings exhibits relatively more stability. Furthermore, due to the larger vocabulary in German, the decoding performance is relatively lower compared to the other two languages.
For the performance on the semantic-formal analogy set, we first evaluated the performance of our pre-trained autoencoder in decoding the embeddings of the fourth item sentence D in the analogy. Both accuracy and BLEU score reached 100, indicating that all the target reference sentences were perfectly reconstructed. Then, as described in Subsection 4.2, we tested the performance of diferent method combinations on this dataset. As shown in Table 3, from the perspective of obtaining sentence embeddings, using the encoder to obtain sentence embeddings performs similarly to the simple summing method, with only a 1-point diference in BLEU score. This is because the average length of English sentences in this dataset is relatively short, and both methods show similar performance in decoding sentence embeddings. However, from the perspective of solving analogies, the FCN network using the 3CosAdd formula outperforms the Ofset network and ANNr. This indirectly indicates that methods relying on predefined formulas are more efective than learning analogy properties from a dataset when the data size is limited, sentences are short, and analogies are relatively simple in form.
For purely formal analogies, Table 4 presents the performance of diferent models in the three languages.
Considering the languages, although French and German have a larger vocabulary, the overall impact is mainly determined by the average sentence length. Since French has the longest average sentence length, followed by German, and English has the shortest, the performance of diferent models is generally lower in French compared to the other two languages. Especially, due to the extremely short average sentence length in English, when using the FCN network to solve analogies, the method of obtaining sentence embeddings using the encoder and the simple summing method show similar performance, with the simple summing method even outperforming it. In contrast, the performance trends of diferent models in French and German are roughly similar. First, the method of obtaining sentence embeddings using the encoder outperforms the simple summing method. Second, the FCN network performs better than the Ofset network.
It is worth mentioning that French has a longer average sentence length, with the longest sentence reaching around 10 words. The performance of the Ofset network is almost on par with the FCN network, with only a slight diference of around 1 in both BLEU score and accuracy. This suggests that when the average sentence length becomes longer, the Ofset network, which learns analogy properties from the dataset, may perform well. Therefore, we further conducted tests on the French dataset by selecting sentences with a length of 10 or more, and the results are shown in Figure 3. We observe that when the sentence length exceeds 10, the Ofset network performs better than FCN. We infer that when dealing with longer sentences, methods that learn analogy properties from the dataset are more reliable than using predefined formulas such as 3CosAdd. This could be because the application of the 3CosAdd formula in analogies of longer average sentence lengths requires the sentence embedding space to have more pronounced linear properties. On the other hand, learning from the dataset allows for lower expectations in the embedding space having linear properties, especially when there is a larger amount of data available.
We proposed an auto-encoder architecture that internally removes noise to generates sentence embeddings and reconstructs sentences, achieving high accuracy in decoding sentence embeddings. Building upon this, we devised an embedding-to-embedding method and a model that learns analogies from datasets in the sentence embedding space instead of relying on predefined formulas. Our experiments demonstrated that this approach performs better than a model relying on the 3CosAdd formula, especially in cases where the sentence length is longer.
Our method for analogy solving is a generation-based approach. It is still limited by the drawback of LSTM decoders in handling long sentences. In the future, we need to explore more advanced encoder-decoder architectures that are better suited for decoding longer sentences, as well as generating more meaningful sentence embeddings specifically designed for analogies.
This research has been partially supported by a JSPS grant Kiban C n° 21K12038 entitled « Theoretically founded algorithms for the automatic production of test sets in NLP."