Prompt learning, a recent thread in few-shot learning for pre-trained language models (PLMs), has been explored for completing word analogies in the extractive way. In this paper, we reformulate the analogy task as masked analogy completion task with the use of prompting to derive a generative model for analogies beyond words. We introduce a simple prompt-based fine-tuning paradigm for language modeling on answered prompts of analogies in the sequence-to-sequence framework. To convert discrete terms of analogies into linear sequences, we present a symbolic prompt template. The sequence-tosequence model is fine-tuned to fill in the missing span of masked prompts deduced from diferent masking schemes on phrase analogies extracted from a small corpus. We analyze the out-of-distribution performance on sentence analogies which are unseen cases. Our experiments demonstrate that promptbased fine-tuning with the objective of language modeling enables models to achieve significantly better performance on in-distribution cases than PLMs. Masked prompt learning with one-term masking exhibits the best out-of-distribution generalization on sentence analogies, with a diference of only 3 characters from references.
Analogy, a cognition mechanism that relies on relational similarity, is growing in prominence
in the field of artificial intelligence [
By formulating tasks in the manner of analogical reasoning, sentence analogy has
demonstrated its versatility in various tasks in the area of natural language processing (NLP), such
as machine translation [
Some work on completing sentence analogies has focused on extractive approaches, i.e.,
identifying the optimum solution from candidates (i.e., from a finite answer pool) [
Prompt learning [
In this paper, we present a preliminary study of generative completion of analogies beyond words by using LMs in conjunction with prompting. To learn analogical regularities, we introduce a novel prompt-based fine-tuning method to extract sequential features of answered prompts with the symbolic template by masked sequence-to-sequence learning. We conduct lightweight fine-tuning on phrase analogies with diferent masking schemes. By language modeling on answered prompts, fine-tuned models achieve over 97% accuracy on solving phrase analogies. The fine-tuned sequence-to-sequence LMs have more promising out-of-distribution generalization on sentence analogies than autoregressive LMs. In particular, one-term masking is more robust at extracting analogical regularities, which contributes to adapt efectively to unseen analogies.
Based on prompting, recent works have formulated the quadruplet problem as language
modeling. The GPT-3 work [
Recent eforts on prompt learning found that prompt-based fine-tuning of PLMs can improve
efective learning on downstream tasks. These works have the advantage of being specific to the
task by tuning the LM parameters entirely [
In this section, we introduce a generative method for completing analogies beyond words by using prompts with a symbolic template (Section 3.1). To adapt to the analogy task, we propose a novel prompt-based fine-tuning paradigm (Section 3.3) for PLMs to reconstruct the missing span in prompts processed by masking schemes (Section 3.2).
Prompt design is crucial in prompt learning. For analogies beyond words, it is easy to get the
tokens of analogical words mixed up with the context tokens of textual templates, as used
in [
Symbolic prompts for analogy are formally identical to analogical equations A : B :: C : D, where the ratio (:) and proportion (::) characters are two symbolic tokens in the prompt template. We employ the Unicode characters U+2236 and U+2237 for ratio and proportion in sequences. Let ( ∈ {, , , }) denote one of the four terms in an analogy. The length of an answered prompt can be calculated as | | + 1 + | | + 1 + | | + 1 + | | = ∑︀ ∈{,,, } | | + 3.
Compared to textual tokens, symbolic tokens that facilitate the distinction between the four terms are straightforward. Based on ordering characteristics, they directly delimit the four terms by detecting the order of symbolic tokens in sequences. For example, : and :: delimit the term , whereas :: and :, in that order, delimit the term .
In light of the usual notation for analogies A : B :: C : x, it is natural to consider the expected solution as the missing text that can be predicted using the left context. To learn sequential information, we explore three patterns of masking for answered prompts. We present some examples of masked sequences that result from masking the following sentence analogy to exemplify masking schemes.
he will come tomorrow. : he will come. :: i have no time tomorrow. : i have no time.
Arbitrary Masking (Mask = any span) Like the document corruption method introduced
in [
he will [mask]will come. :: i have no time tomorrow. : i have no time.
One-term Masking (Mask = any term) A masking span is a whole term in analogies, selected from the quadruplet (, , , ). Due to the binding meaning between the four terms in an analogy, each term can be derived from the other three. This scheme randomly masks one of the terms. It allows models to capture more comprehensively analogical regularities. The term is masked in the following sequence.
he will come tomorrow. : he will come. :: [mask] : i have no time.
Target-oriented Masking (Mask = term D) In this setting, we regard the target prediction as the masking span in analogy prompts. To follow standard notations in analogy, i.e., the format A : B :: C : x, we specifically mask the fourth term in answered prompts as shown below.
he will come tomorrow. : he will come. :: i have no time tomorrow. : [mask]
In general, the paradigm of prompt-based fine-tuning reformulates downstream tasks as masked
language modeling on prompts, with the goal of optimizing the prediction of the mask token
specified as target outputs [
Given a pair of sequences (, ) , where is the sequence of a masked prompt (including a single mask token) obtained by applying a masking scheme to the answered prompt, is the target sequence of the masking span. As in regular sequence-to-sequence learning, we tune the model parameters Θ = ( , ) to estimate the conditional probabilities of target tokens given masked prompts. The masked prompt is encoded bidirectionally. Each token in the target sequence is predicted by the autoregressive decoder by maximizing the conditional probability given the input sequence and its preceding sequence < : ︁∏ | |
Note that finding sentence analogies from a corpus will be dificult unless the corpus is dense. It is easier to find more analogies between small chunks from a corpus than entire sentences. Sentence constituents (i.e., phrases) are structural chunks that play grammatical roles in sentences. We focus on finding formal analogies between phrases, which can indirectly reflect the linguistic regularities contained in the given corpus.
We build analogies from the English part of a parallel corpus1 released for the news translation
task at the Workshop on Machine Translation (WMT20). The data we used is made up of 3,003
sentences with an average length of 25 words. In order to detect phrases, we use Berkeley
Neural Parser2[
Starting from the set of phrases, we apply some functions from the Nlg tool3 introduced in
[
In our settings, over 1.5 million analogical clusters are extracted, where each cluster contains two phrase ratios in average. By combining every two ratios in a cluster, we are able to enumerate
2https://github.com/nikitakit/self-attentive-parser 3http://lepage-lab.ips.waseda.ac.jp/en/projects/kakenhi-15k00317/ → Tools - Nlg Module 1,524,293 analogies that capture formal similarities between four distinct phrases. The analogy data includes 17,480 types of phrases with an average length of 3 words (20 characters). It can be roughly estimated that the average length of analogy prompts is 3 × 4 + 3 in words (resp. 20 × 4 + 3 in characters). For example, a collected phrase analogy to say : want to say :: to go out : want to go out indicates a verb phrase attachment with the modifier of the verb want, which consists of 15 words including three symbolic tokens in the prompt template.
We fine-tune LMs on prompts of phrase analogies. To avoid that phrase ratios in the test data appear also in the training set, we split the cluster data before making analogies. We take 1,000 clusters for building the analogies for testing and another 1,000 for validation, while the remaining ones are for training. For each cluster set, we assemble every two ratios in the same cluster into an analogy. The training, validation and test sets consist of approximately 1.5 million, 1,000 and 1,000 phrase analogies respectively.
In addition to the phrase analogy test set, we also explore the performance of Transformer
models on solving sentence analogies, which are unseen analogies and have diferent
distributions than the training data. We sample sentence analogies from the English part of bilingual
analogies4 used in an EBMT by analogy setting [
Even if we swap the ordering of four phrases, each analogy only appears once in all datasets, with no duplicates. The length statistics of analogies are shown in Table 1, including lengths of terms and answered prompts. Analogies in the test sets are processed as prefix prompts, in which the mask token is the last token in sequences. Each model is tested by infilling the unknown term in masked prompts with the default format A : B :: C : [mask].
In terms of the sequence-to-sequence Transformer architecture, we experiment with a
pretrained BART [
We made some modifications to the BART fine-tuning procedure provided by the Transformers
library [
As for the autoregressive baseline, we explore the performance of the distilled GPT-2 model consisting of 6 layers. The entire pre-trained GPT-2 is fine-tuned on answered prompts of phrase analogies for generating the last term given preceding context. To compare diferent methods, we employ the accuracy metric to measure the percentage of generations that exactly match the references. In addition, we also compute the Levenshtein distance (including spaces) to measure diferences at the character level. Table 3 shows the performance of diferent fine-tuned models on completing phrase analogies and sentence analogies.
Model Architecture Prompting together with fine-tuning, benefits PLMs to accomplish efective analogy completion of in-distribution cases (phrase analogies). As far as the model architecture is concerned, the autoregressive LM excels in inferring answers to phrase analogy questions where the last term (D) is missing. By learning on answered prompts of phrase analogies in a feed-forward fashion, GPT-2 achieves the best accuracy 99.7% on phrase analogies. However, the sequence-to-sequence model fine-tuned for infilling the term D performs noticeably worse, only reaching 50.9% in accuracy. Except for the setting of target-oriented masking (term D), BART fine-tuned with masked prompt learning have competitive performance with GPT-2.
Masking Scheme The masking deployment of prompts has a large impact on capturing analogical regularities in masked prompt learning for the sequence-to-sequence model. The target-oriented masking scheme (term D), like the mechanism of few-shot prompt-based finetuning in MLMs, performs the worst in phrase analogies. We posit that it makes BART overly imitate the features of the fourth terms of analogies rather than comprehend analogical relationships. The arbitrary masking (any span) and the one-term masking (any term) are optimal for modeling the sequential information of prompts for phrase analogies, which enables the model to perform well in generating the last phrase in analogies, with only a 0.05 character diference. Out-of-distribution Generalization As shown in the results of SA in Table 3, the autoregressive LM exhibits poor out-of-distribution generalization capability, although it achieves excellent performance on phrase analogies. GPT-2 can only correctly answer 4.2% unseen sentence analogies where sequences are longer than the training data. This suggests that the autoregressive LM may overfit to the narrow distribution of phrase analogies, which leads to a failure in solving analogies between longer sequences. In comparison, the sequence-to-sequence models exhibit better out-of-distribution generalization. Particularly, BART with similar finetuning procedure, predicting the term D conditional on previous tokens, is approximately three times superior to GPT-2.
It is noticeable that masked prompt learning on prompts with one-term masking (any term) has the best generalization on sentence analogies. It is 4 times more accurate than any span masking with competitive performance on phrase analogies. It enables BART to generate sequences that very closely match the reference sentences, difering by only 3 characters on average, about 3/20 of the reference length. Table 5 presents some example errors. The finetuned BART model profiting from the bidirectional learning on previous and future tokens, can accomplish efective adaptation to unseen sentence analogies, with the achievement of an order of magnitude greater accuracy than GPT-2.
To further explore analogical capabilities of LMs fine-tuned by masked prompt learning, we conduct a fine-grained probing on analogical questions with diferent formats. For each test analogy, we replace one of the four terms with the mask token to enumerate four analogy questions with diferent masking spans. We use an individual accuracy Acc (where ∈ {, , , }) to indicate the performance in solving analogies in a specific format. For example, Average accuracy results of fine-tuned BART models with diferent masking schemes in solving analogy questions in diferent formats. We report both individual accuracies and universal accuracy Accall introduced in Subsection 5.4 .
Acc (where ∈ {, , , } ) Data
Masking Scheme PA SA
Any span Any term Term D Any span Any term Term D
mask settings that take analogical terms into account, the masking scheme of any term achieves competitive performance in terms of superior accuracy on each individual question, while term-specific masking enables the model to answer only half of the questions where term is masked. For performance on sentence analogies, fine-tuning for any term masking outperforms substantially other strategies in terms of both individual accuracies and universal accuracy. Concretely, the fine-tuned model performs relatively well on completing sentence analogies where one of the last triplets is missing. It is not a surprise that BART with the target-oriented masking (term D) fails to predict terms other than the term D. The reason is that the mask token does not appear in any position during the fine-tuning procedure except for the last term.
The masking scheme of any term is more adapted to solving analogies in masked prompt learning. In this subsection , we use one-term masking (any term) and ablate fine-tuning strategies of tuned layers and prompt templates.
Input he is getting better. : tom is getting better. :: he doesn ’t like to lose. : [mask] Output tom doesn ’t like to lose Reference tom doesn ’t like to lose.
Input i know your name. : i believe you. :: i don ’t know your name. : [mask] Output don ’t believe you.
Reference i don ’t believe you.
Input Output Reference Input Output Reference what is going on here? : he is intelligent. :: what’s going on here? : [mask] he is intelligent. he’s intelligent. how did you do this? : what did you say? :: how do you do this? : [mask] what did you say? what do you say? Input he will come tomorrow. : he will come. :: i have no time tomorrow. : [mask] Output he will have no time tomorrow.
Reference i have no time. entire model helps model phrase analogies, with a significant gain of 94.2 points in predicting the last phrase in analogy questions.
However, it is imprudent to update the entire model. The full-scale fine-tuning makes the model specialized in the training distribution, achieving only 11.7% accuracy in completing sentence analogies. Freezing the entire encoder or decoder degrades the performance of solving phrase analogies, while it increases the performance by at least 12 points over the unfrozen BART for generalizing out-of-distribution analogies. In particular, only tuning the decoder and freeze the encoder is useful to learn phrase distributions, whereas fine-tuning the encoder and freeze the decoder performs relatively better for capturing analogical regularities.
By contrast, lightweight joint fine-tuning on both encoder and decoder performs well on two analogy test sets. Fine-tuning the bottleneck layers (the top two layers of encoder and the bottom two layers of decoder), which closely updates the encoder-decoder attention, achieves accuracy scores of 95.8% and 42.0% on phrase and sentence analogies respectively. Our strategy of fine-tuning only the bottom two layers of the encoder and the top two layers of the decoder, exhibits the best performance, yielding slight gains of about 2 points over fine-tuning the bottleneck layers of BART.
Prompt Templates In Table 7, we list results of GPT-2 and BART learned on prompts with
diferent manually written templates including symbolic prompt, textual prompt and null
prompt.5 As findings in [
Despite a subtle drop (1.4 points) in the phrase analogy test, our prompt contributes to better
adaptation on unseen analogies than the textual prompt used in [
Regardless of the template, GPT-2 models struggle in completing sentence analogies, although they excel in completing phrase analogies. In contrast, the BART model is over 10 times more accurate than GPT-2 in sentence analogies with the help of non-null prompts.
Our work demonstrated the potential of LMs to complete analogies beyond words. We introduced a simple but efective prompt-based fine-tuning paradigm for solving analogies beyond words by masked sequence-to-sequence learning on answered prompts with diferent masking schemes. To extract useful information about analogical regularities, we proposed three patterns of masking on answered prompts. We found that fine-tuning with the objective of language modeling on answered prompts, is efective for adaptation of generative analogy completion on phrase analogies, except the sequence-to-sequence framework with target-oriented masking which leads to overfit to narrow features in the training data. Compared to the autoregressive framework, masked prompt learning is beneficial for out-of-distribution generalization over sentence analogies. Lightweight fine-tuning in masked prompt learning with one-term masking has the best potential for learning robust analogical capabilities. In the future, we intend to refine the fine-tuning paradigm to enhance out-of-distribution performance in the few-shot scenario. We hope to apply to other languages and build a multilingual generator for analogies beyond words.
The work is supported by China Scholarship Council (CSC) under the CSC Grant No. 202008050136.