This paper introduces a novel translation technique, driven by modeling fuzzy analogies that capture approximate conformity to parallel transformations between fragments in sentences. We conduct preliminary experiments on English-Japanese translations with a data set of limited size. The results show the potential of using fuzzy analogies for translation, achieving an increase of about 6 BLEU points compared to NMT.
The proposed method is built on the indirect paradigm of example-based MT in [
= ′.
To retrieve sentence analogies, we first pre-compute candidate pools for terms , , and by collecting the nearest neighbors of
using cosine similarity between sentence embeddings.
Theoretically, there will be a cubic number of possible combinations of sentence quadruples (, , ,
). To reduce the computational cost, we prune candidate quadruples. We leave
out the quadruples with no lexical overlap between and , and between
and . Finally,
for each , we rank the quadruples by analogical score, and select the first ones. As in [
) considered as sequences of (sub-)words. We count the number of trivial analogies of the form a : a :: b : b or a : b :: a : b for every aligned (sub-)word quadruple. Figure 1 illustrates the computation of analogical scores.
Next, we train a sequence-to-sequence model to solve analogies, so as to derive translation answers. Suppose : :: :
and ′ : ′ :: ′ : ′ are a retrieved source analogy and its corresponding translation. We concatenate 7 sentences (excluding ′) in two monolingual analogies as input , to train the model to generate the solution ′ by optimizing cross-entropy (CE) between probability distributions conditional on the context of input and preceding target tokens: ℒ CE = − log ( ′| <′ , ) (1) To encourage the model to be more confident in reconstructing target fragments that are in analogical relationships, while being flexible to non-analogical relationships, we introduce a weighting scalar in (1). Formally, the aim is to minimize weighted CE (WCE): ︁∑ where takes the value of 1 for trivial analogies, and 0.5 else. For each target token, a weighted value is determined by its corresponding aligned token in .
We experiment with parallel sentences from the Japanese-English Subtitle Corpus1, with 50,000 pairs for training, 2,000 for validation, and 2,000 for test. In this work, we primarily investigate the translation quality from English to Japanese. The source sentences contain approximately nine words on average. For each data set, we take source sentences as queries and look for fuzzy analogies from the source part of translation memory (i.e., the training set). The strictness in analogies depends on how closely the queries match the examples in memory. We assess the closeness between the data sets and the memory by computing the similarity using the length of longest common subsequence between sentences at the word level. Specifically, we compare the query sentence to the twenty most similar examples in the memory, excluding itself in the case of the training set. Table 1 shows the statistics of three data sets. On average, the three data sets exhibit similar characteristics, where source sentences are found to have an overlap of four words with their corresponding similar sentences in the memory.
In order to retrieve analogies from the corpus, we first use a Sentence-BERT [
To learn from analogy, we fine-tune a pre-trained mBART [
We compare to an NMT system by fine-tuning the same pre-trained mBART model on the data sets of parallel sentences. The baseline NMT model is trained using the consistent settings as described above. On 50,000 parallel sentences, NMT obtains a BLEU score of only 2.9. Our system using (1) achieved an improvement of 5.6 and the use of (2) leads to a further gain of about 0.4 BLEU points. Even though fuzzy analogies relax the strictness, the inclusion of partial evidence in parallel transformations still helps in deducing possible translation.
In the retrieved analogies, query sentences are covered by examples under the analogy constraint to diferent extents with analogical scores ranging from 0 to 1. Figure 2 shows the number of fuzzy analogies constructed for the sentences in the three data sets, categorized by their respective scores. In general, three sets of analogy data demonstrate a comparable distribution in the extent of fuzzy matches between sentence transformations. The majority of analogies fall within the score range of 0.3 to 0.7. This indicates that approximately 30%-70% of tokens in query sentences are associated with examples in the analogy relationship.
Next, we examinate the model performance in inferring translation answers by solving fuzzy analogies with diferent scores. Figure 3 shows that our model is capable of reasoning analogies with lower scores, where less than half of a query sentence is linked to translation examples through analogical associations. This suggests that fuzzy analogies can capture relative knowledge of two languages, which can even assist in translating queries that are distant from memory. We also compare to an NMT baseline on translating each test sentence. In Figure 3, blue points (415 out of 2,000) indicate the cases where our model performs worse than NMT in BLEU. Relatively, there are fewer underperforming cases when analogies have higher scores (>0.7). In Table 2, we list examples of two methods in translating sentences that are either close 3To enable the learning model (e.g., mBART) to identify analogical transformations in quadruples, we use the SentencePiece model with the same tokenization as in mBART. 4http://www2.statmt.org/moses/ 5https://huggingface.co/facebook/mbart-large-50
×1,000 s e igo 10 l a n a g n i n i ra 5 t f o r e b m u N 0
×100 train valid test 2 to or distant from translation memory.
Do diferent source analogies constructed for the same query result in diverse translation outputs? We conduct additional experiments to address this question. For each test sentence, we retrieve five fuzzy analogies with the maximum scores and then employ the model trained specifically to handle one analogy per query to solve each of these analogies. Table 3 presents ifve distinct translations to a query sentence. As shown by the example, it is possible for the model to generate more idiomatic translations that closely convey the intended meaning, using Examples of translating sentences that are either close to or distant from the memory. For each generated Japanese answer, we also provide the corresponding English translation below, which has been translated using Google Translate. The underlines denote formal matchs to the references. For the upper instance, NMT achieves a higher BLEU score but it fails to specifically mention the content regarding worry, whereas our model efectivly captures the meaning of the original sentence. Regarding the translation of the distant query that involves specialized terms not present in the memory, NMT seems to draw upon knowledge from its pre-trained data. However, the translation is not accurate. Our model provides a translation for the word "called" by leveraging hints from the analogy, but does not convery an accurate translation for the term itself.
ah... i’m sorry i made you worry. あ... 心配かけてすみません。 あ... ごめんなさいごめんなさい。 (ah... i’m sorry, i’m sorry.) Ours analogical score: 0.82 ⇒ = あ... ごめんね心配かけちゃって。 (ah... i’m sorry i made you worry.) i’m sorry for bother- i... i’m sorry. i wasn’t ing you. thank you. す い ま せ ん で し た。失礼します! : : being careful enough. す・・すみません でした私が十分な 注意を払っていま せんでした :: i’m fine. i’m sorry for making you worry.
: ah... i’m sorry i made you worry.
大丈夫です。すい :: ませんご心配おか : けして
called alpha lipoxanthine glucoside
Ref. [スピーカ] 「アルファー・リボキサンチン・グルコシド」n略して NMT アルファリン酸オキシトリン酸グリシトリン酸 (alpha phosphate oxyphosphate glycitrate) Ours analogical score: 0.0 that’s the name. そう名付けること にしたよ。 : biotech company.
it is called alexon そ れ は ア レ ク ソ : ン・バイオテック という会社なんで すけど ⇒ = アルパルス・グローブという名で (under the name alpuls grove)
lincoln. :: the name is abraham : called alpha lipoxan
thine glucoside :: 名 は エ イ ブ ラ ハ ム・リンカーン : analogies with less evidence. We speculate that enlarging the number of fuzzy analogies will facilitate models in acquiring more potential associations in two languages.
4. Conclusion and Future Work
In this paper, we introduced a novel translation approach based on the mechanism of using
indirect analogies for translation. Unlike the work in [
In future work, we will conduct ablation studies to search for optimal configurations for modeling analogies. In addition, we will expand this work to diferent language pairs and directions, as well as investigate the influence of corpus size on performance.
The research reported in this paper was supported in part by a grant for Kakenhi (kiban C) from the Japanese Society for the Promotion of Science (JSPS), n° 21K12038 “Theoretically founded algorithms for the automatic production of analogy tests in NLP”. はい。それがいい : 確いかかもにな、それが良 :: るっていいことだ : かと思います。 好きなことを頑張 と思いますけど ⇒ = じゃあはっきり言うなら... 。 ( well, to put it bluntly...) ( to put it bluntly... sorry for being mean.) よ? これ。痛っ。 やまだ。ヘ俺ヘのッ方。がい何いだ : あぁ、最良だった :: い確かかもにな、それが良 :
と思うよ ⇒ = はっきり言うなら... 意地悪いのは悪いけど。 i think it’s better if 0.40 imgyuseisdse.it’s better on : aybeply, ifogrutehses bite’sstp.rob-:: i think that’s best. : you frankly say your 0.40 i think that’s best. : aybeply, ifogrutehses bite’sstp.rob-:: what you like.
i think it’s great to do : iyotuhifnrkaniktl’ys sbaeyttyerouirf 確かに、それが良 : あと思ぁう、よ最良だった :: 好るっきてないこといこを頑と張だ : いかもな ⇒ = はっきり言うなら... 意図は正直に言うべきよ ( to put it bluntly... you should be honest about your intentions)
と思いますけど ( to put it bluntly, your meanness is bad.) よ? これ。痛っ。 やまだ。ヘ俺ヘのッ方。がい何いだ : 確かに、それが良 :: とあ思ぁう、よ最良だった :
いかもな ⇒ = はっきり言うなら君の意地悪は悪いけどな。 0.40 imgyuseisdse.it’s better on : i think that’s best. :: aybeply, ifogrutehses bite’sstp.rob-: iyotuhifnrkaniktl’ys sbaeyttyerouirf 0.40 aybeply, ifogrutehses bite’sstp.rob-: i think that’s best. :: imgyuseisdse.it’s better on : iyotuhifnrkaniktl’ys sbaeyttyerouirf intentions.
by solving the first five fuzzy analogies with diferent analogical scores. The reference translation is "あの... ズバッと訴えたい事を伝えたほうがいいと思うんです。". We highlight the answer that closely matches the reference according to the human evaluation.
score Analogy and solution 0.47 iidtehai.nk that’s a good : i think that’s best. :: iwthhaintkyoitu’sligkree.at to do : iyotuhifnrkaniktl’ys sbaeyttyerouirf ( honestly... if you say mean) ⇒ = 正直なところ... 意地悪いって言ったら あぁ、最良だった : い確かかもにな、それが良 :: や。ヘヘッ。何だ : と思うよ