In this study, we introduce a novel approach to enhance sentence embedding by leveraging word analogy. Compared with past methods that use word analogy on sentence-level tasks, our method is less afected by sentence patterns and pays more attention to semantic relations. By fine-tuning pre-trained models as BERT, RoBERTa and Sentence-BERT and evaluating their performance on inter-sentence downstream tasks, we demonstrate the eficiency of our method. Our experimental results show that each model, following fine-tuning using our approach, exhibits improvements across all inter-sentence tasks. In the STS task, our method increases the average result from 18.63% to 62.52% on BERT. This outcome substantiates that sentence relationships derived from word analogy contain valuable knowledge that can enhance the performance of sentence embedding models.
sound made by a cat"). In total, DSBATS based on semantic network (DSBATS-sn) consists of 20 clusters, each capturing a distinct relationship.
We employ contrastive learning and DSBATS-sn to fine-tune popular models, including BERT, RoBERTa, and Sentence-BERT. We do data augmentation on DSBATS-sn to get DSBATS for contrastive learning (DSBATS4CL). Through a series of experiments, we evaluate the performance of our approach in three inter-sentence downstream tasks. Firstly, we propose an intrinsic evaluation task called "sentence relationship similarity distinguishing", a task of identifying whether the two sentence relationships are the same. Fine-tuning the models with DSBATS4CL leads to performance improvements of 8.37%, 7.42%, and 7.87% on BERT, RoBERTa, and Sentence-BERT, respectively, compared to the performance of the original pre-trained models. Secondly, in the Semantic Textual Similarity (STS) task, our method achieves improvements of 43.6%, 21.46%, and 13.89% on the three pre-trained models, respectively. Additionally, our approach consistently produces modest improvements on the Microsoft Research Paraphrase Corpus (MRPC) dataset.
We prove that the language model can learn knowledge from sentence relationships generated
from word analogy to improve the performance on semantic analysis tasks. Compared with the
sentence relationships from NLI datasets, our method reduces the need for human annotation
and increases the diversity of inter-sentence relations efort by using semantic network and
word analogy data. Meanwhile, sentence relationship similarity distinguishing task proposed in
this paper is also a challenging evaluation metric for sentence embedding method.
2. Pretrained models and sentence embedding
Pretrained models have played a significant role in the advancement of natural language
processing (NLP) tasks. They are models that are pre-trained on large corpora of text data to learn
language representations that capture semantic and syntactic properties of words and sentences.
Transformer-based pre-trained model like BERT [
Many sentence embedding methods opt to fine-tune BERT or RoBERTa using sentence-level
pre-training tasks. Sentence-BERT (SBERT) [
We use word analogy as the relationship resource. The most common example of word analogy
is king : queen :: man : woman, it states that "king is to queen as man is to woman". Phenomena
like this are to be studied as an important process of human cognition, with the development of
language models, computational analogy has attracted more and more attention [
There have also been past studies on constructing sentence relationships through word
analogy, in [
We use semantic networks as the sentences resource. Semantic network is a kind of resource in a
graphical form that shows the relationships between concepts or entities. In a semantic network,
concepts are represented by nodes, and the relationships between those concepts are represented
by edges that connect the nodes. BabelNet is a multilingual semantic network and ontology
that provides a wide range of information about words and concepts in multiple languages [
With BATS as our relationship resource and BabelNet as our sentence resource, we build the dataset Definition Sentences from BATS based on sementic network (DSBATS-sn). The extraction process is as the Figure 1(a). We input word analogy clusters into BabelNet, and BabelNet will deliver several synsets for each word in the clusters. In Figure 1(b), we have "duck" and "quack" as a pair in the cluster of animal:sound relationship, the search for"duck" in BabelNet delivers 35 diferent synsets, including synsets that do not conform to the animal:sound relationship like the synset with number 0 in math area. We use a filter to select the valid synsets that refer to the concept corresponding to the relationship. The filter takes advantage of the information contained in BabelNet, like the domain category or the part of speech, to select the synsets that match the relationships. In Figure 1(b), orange synsets are chosen, and gray synsets are discarded. The definition sentences in the valid synsets will be organized as sentence relationship clusters as the output part in Figure 1(a). In DSBATS-sn, there are 20 clusters corresponding to 20 diferent relationships, the size of diferent clusters (categories) are shown in Table 2. Some additional examples are shown in Tables 3 and 4. “bee” “buzz” “dog” “bark” “cat” “mew” “duck”“quack”
BabelNet
Filter “Any of numerous hairy-bodied “Sound of rapid vibration.” insects including social and solitary species.” “The dog is a domesticated descendant of the wolf.“ “Domesticated mammal of the “The sound made by a cat.” Felis catus species.” “Name applied to several bird species of the family Anatidae. ” “The harsh sound of a duck” “The sound made by a dog. ” (a) Input a word analogical cluster in BATS and output a sentence relationship cluster in DSBATS-sn “Small wild or domesticated …swimming bird usually having …short legs.”, … “duck” “zero”, “nought”, “cipher”, “duck” “A mathematical element …another number yields the same number.”,
…
When using DSBATS-sn for fine-tuning, we aim to have similar relationships close to each
other and diferent relations far away from each other in the embedding space. For example, the
relationship between "Domesticated mammal of the felis catus species" and "The sound made by a
cat", and the relationship between "The dog is a domesticated descendant of the wolf" and "The
sound made by a dog" are both the relationship of animal:sound, they are positive examples
that should be close, and we can generate sentence pairs in another relationship sound:animal
as negative examples by exchange the position in the pair. This requirement conforms to
the basic idea of contrastive learning, which is to narrow the distance of relevant samples
and push the distance of irrelevant samples in a certain feature space. Contrastive learning
does not require very large-scale labeled data, and it can make the samples more uniformly
distributed in the feature space [
Our loss is basiclly InfoNCE [
) But there is a little diference, we only use the third pairs in the batch as negative examples, instead of using all the samples in the same batch except for as negative examples for , so
Relationship animal:sound animal:sound sound:animal loss = − log ( ∑︀ =1 sim(,− )/
) sim(,+)/ (2) − corresponding to the red example in Table 5. S1: Any of numerous hairybodied insects including social and solitary species.
S3: The dog is a domesticated descendant of the wolf.“ S2:Sound of rapid vibration.
S4: The sound made by a dog..
Sentence Embedding Ԧ1 Ԧ2 Ԧ3 Ԧ4 ( 1 − 2, 3 − 4, | 1 − 2 − 3 + 4 |)
Classifier Similar relationship
We designed the Sentence Relationship Similarity Distinguishing (SRSD) task as intrinsic evaluation. It inputs a pair of relationships at a time, which is 4 sentences, and predicts if they are two similar relationships. Figure 2 shows the process with two sentence relationships in the same category. The test set is a manually annotated DSBATS dataset diferent from the automatically extracted DSBATS-sn in Section 3. The manual version uses the same relationship resource BATS but diferent sentence resources like Oxford Dictionary, Webster’s Dictionary, and Collins Dictionary, so we call the manually annotated DSBATS as DSBATS-dic. The number of pairs of sentences in DSBATS-dic is shown in Table 6. After learning with DSBATS4CL, all three models improved accuracy on this task. The improvement is basically about 6%. The best performance is from Sentence-Bert with DSBATS4CL, which reaches 69.55%. Table 7 shows the result of SRSD task.
We conducted extrinsic evaluations using the Semantic Textual Similarity (STS) and Microsoft
Research Paraphrase Corpus (MRPC) datasets as extrinsic evaluations. We use SentEval [
In this work, we introduced a method to enhance sentence embedding using word analogy. We map the relationships between words to relationships between sentences by using definition sentences in semantic network. Compared with the past methods that use word analogy in sentence-level tasks by replacing words in sentences, our method is less limited by morphology and pays more attention to semantics. The improvements on downstream tasks like STS and MRPC prove that the sentence relationships from word analogy include the knowledge that can enhance the semantic understanding of sentence embedding models. Sentence relationship similarity distinguishing task proposed as an intrinsic evaluation in our work can also be a challenging evaluation task for other sentence embedding methods. We believe that it is worth further exploring ways to combine analogy with contrastive learning, as analogy relation has many equivalent forms suitable for contrastive learning to construct positive and negative examples.
This work has been supported in part by a research grant from JSPS Kakenhi Kiban C n° 21K12038 entitled “Theoretically founded algorithms for the automatic production of analogy test sets in NLP.”