The past decade has seen an increasing interest in analogical reasoning (AR) and of analogical proportions (APs), which are statements that four elements 𝐴, 𝐵, 𝐶, 𝐷 are in analogy (usually written 𝐴 : 𝐵 :: 𝐶 : 𝐷). Indeed, AR and APs are useful not only in the study of the mechanisms of human cognition [1] but also for applications in artificial intelligence [2,3]. There are two basic tasks associated with AR: the first is analogy detection that corresponds to the task of deciding whether a quadruple 𝐴, 𝐵, 𝐶, 𝐷 constitutes a valid AP, and the second is analogy solving that corresponds to finding the solution of an analogical equation, i.e., an AP 𝐴 : 𝐵 :: 𝐶 : 𝑋 where 𝑋 is unknown.

Analogies between words and strings of symbols has long been studied [4,5,6,7,8,9, 10] and makes for an experimental setting in which a wide range of analogies appear, from simple (𝑎 : 𝑎𝑎 :: 𝑏 : 𝑏𝑏) to more complex (word:language::note:music). In this paper, we focus on the study of morphological analogies, i.e., analogies modeling changes of morphemes. In particular, we study how the deep learning approach for detecting morphological analogies proposed in [11,12] behaves when transferred across domains. We call this approach Analogy Neural

The analogy detection task corresponds to classifying quadruples 𝐴, 𝐵, 𝐶, 𝐷 into valid or invalid analogies. In other words, it can be seen as a binary classification task. Morphological analogy detection is used in the context of analogical grids [15], i.e., matrices of transformations of various words, similar to paradigm tables in linguistics [16]. To detect analogies and build the analogical grids, Fam and Lepage [15] use the number of characters occurrences and the length of the longest common subword.

In the context of semantic word analogies, Bayoudh et al. [4] use Kolmogorov complexity as a distance measure between words for analogy detection, and Lim et al. [17] implement a data-driven approach. Using a dataset of semantic analogies, Lim et al. learn a neural network to classify quadruples 𝐴, 𝐵, 𝐶, 𝐷 into valid or invalid analogies, using their embeddings 𝑒 𝐴 , 𝑒 𝐵 , 𝑒 𝐶 , and 𝑒 𝐷 . We adapt the latter approach to morphology by replacing the original GloVe [18] semantic embedding model by a character-level embedding model in our previous work [11], which significantly outperforms previous approaches.

Analogy detection and solving are closely related tasks. For instance, the morphological analogy solving approach of Langlais et al. [19] and the one of Murena et al. [14] are used as analogy classifiers in [11]: given a quadruple 𝐴, 𝐵, 𝐶, 𝐷, if 𝐷 is in the predicted solutions of 𝐴 : 𝐵 :: 𝐶 : 𝑋 then 𝐴 : 𝐵 :: 𝐶 : 𝐷 is a valid analogy, otherwise it is invalid. Other works on solving analogies on character strings can be found in the literature, including Copycat ICCBR'22 Workshop Proceedings by Hofstadter and Mitchel [5] but also works relying on embedding spaces [20,17,21,9,10]. For instance, the work of Lim et al. [17] also proposes a model for analogy solving in addition to ANNc. The former model was adapted to morphological word analogies in our previous work [21] and outperforms generative methods that do not rely on deep learning.

In this article, by transfer we mean applying a machine learning model on a target domain, different from the source domain used to train/learn the model. Different types of transfer are possible, from directly applying a model to the target model, to transferring the model and finetuning it on the target domain. The latter method is a type of model adaptation, which consists in altering a transferred model to fit the new data and which is a key step in the transfer methodology. It is also possible to transfer a part of a model and reuse it as a component of a larger model, as is usually done with large pretrained embedding models such as BERT [22], wav2vec2 [23], or vision transformers [24].

Transferring a model can serve two main purposes: achieving satisfying performance on the target domain while dealing with issues of the target domain or the model (lack of data or of labeled data, large training time, biases, etc.), or studying differences in the behavior of the model on different domains. In this work, we focus on the latter aspect.

In [12], we performed multiple transfer experiments with ANNc on analogies extracted from Sigmorphon2016 [25] and Japanese Bigger Analogy Test Set [26] (which are now available in Siganalogies [27]). We transfered between languages to explore how the analogy model could generalize between domains, and built models on a subset of representative languages to determine the feasibility of a more general model of analogy. In both settings we obtained encouraging performance, but we were not able to fully explain the difference in performance between the languages used. We first experimented with what we called full transfer, in which all the components of the approach (character encoder, morphological embedding model and ANNc) are trained on the source language and transferred to the target language, without finetuning. This approach produced good overall results except on some languages using nonroman characters, and is the approach we take for the present article. To solve this alphabet gap issue, we used partial transfer, i.e., the character encoder and morphological embedding model trained on the target language are reused instead of the ones trained on the source language. While this approach improved performance in case of alphabet gap, it was still far from the performance of models trained on the target language, probably due to a mismatch between the embedding model and the embedding space used by ANNc.

In this paper, we introduce general elements of our experimental setting in Sec. 2. We extend the results of [12] in several ways:

• in Sec. 3, we use transfer to determine the impact of the axiomatic setting on the performance of the model, as mentioned above, and confirm that different training procedures results in compliance to different sets of axioms; • in Sec. 4, we confirm that ANNc coupled with the morphological embedding of [11] generalizes to similar data, by transferring models of analogies in 8 languages between the Sigmorphon2016 and Sigmorphon2019 dataset; • in Sec. 5, we leverage 42 high resource languages of Sigmorphon2019 [28] to extend previous results on inter-language transferability, and confirm previous hypotheses on the alphabet gap issue and the transferability of morphological analogies between related languages.

In this section we first present the analogical data. Then, we detail the default axiomatic setting 𝐶𝑃 (accepting central permutation as an axiom) and two variants 𝐶𝑃 (explicitly refusing central permutation) and ¬𝐶𝑃 (not taking central permutation into account). Finally, we specify our training, evaluation and transfer protocol.

In our experiments we use the analogies available in Siganalogies [27], which are extracted from three datasets: Sigmorphon2016 [25], Japanese Bigger Analogy Test Set [26], and Sigmor-phon2019 [28]. In Siganalogies the analogies are obtained by associating four words 𝐴, 𝐵, 𝐶, 𝐷, with 𝐵 is a morphological transformation of 𝐴 (i.e., 𝐵 = 𝑓 (𝐴)) and similarly 𝐷 = 𝑓 (𝐷), such that the morphological transformations are identical.

In previous works on ANNc, the model was trained using training examples obtained by permuting the four words of each analogy in the dataset. For the positive class, permutations of four words resulting in valid analogies (𝑃 + ) are generated from each analogy 𝐴 : 𝐵 :: 𝐶 : 𝐷 in the dataset. For the negative class, permutations resulting in invalid analogies (𝑃 − ) are generated from each analogy of 𝑃 + , as they all are valid analogies.

In introduction, we mention the possibility of using different axiomatic settings, which leads us to experiment with central permutation among the most discussed axioms of analogical proportions [13] in Sec. 3. For this purpose, we consider three axiomatic settings (𝐶𝑃 , ¬𝐶𝑃 , and 𝐶𝑃 ) described below.

In the setting of [11,12,17], that we call 𝐶𝑃 , the axioms of [7] are used and central permutation is considered as an axiom for APs. Central permutation is thus used to generate the permutations in 𝑃 + 𝐶𝑃 . In particular, the permutations in 𝑃 + 𝐶𝑃 (Eq. ( 1)) can be obtained by applying successively central permutation and symmetry. Permutations that contradict these two axioms or strong inner reflexivity are used to obtain 𝑃 − 𝐶𝑃 . Given a base form 𝐴 : 𝐵 :: 𝐶 : 𝐷, 𝑃

We consider two settings in which central permutation is not an axiom: ¬𝐶𝑃 in which we discard the central permutation axiom, and 𝐶𝑃 in which we explicitly consider that applications of central permutation are invalid analogies. Discarding central permutation to obtain ¬𝐶𝑃 is the simplest way to refute the central permutation axiom, and results in the following sets of permutations:

¬𝐶𝑃 ={⟨𝐴, 𝐵, 𝐶, 𝐷⟩, ⟨𝐶, 𝐷, 𝐴, 𝐵⟩, ⟨𝐵, 𝐴, 𝐷, 𝐶⟩, ⟨𝐷, 𝐶, 𝐵, 𝐴⟩}

For 𝐶𝑃 we go one step further in refuting the central permutation axiom by considering that applications of central permutation are invalid analogies, as mentioned above. To do so, central permutation is removed from the valid permutations (as in 𝑃 + ¬𝐶𝑃 ) and added to the permutations of the invalid class:

In [11,12] a subset of 8 permutations from 𝑃 − 𝐶𝑃 is randomly sampled during training to obtain balanced classes (8 permutations of 𝑃 − 𝐶𝑃 for 8 permutations of 𝑃 + 𝐶𝑃 ). For 𝐶𝑃 and ¬𝐶𝑃 , to obtain balanced classes with a number of permutations comparable to 𝐶𝑃 , 8 permutations from each class are randomly sampled 1 during training.

The training of ANNc on the source domain is done in the same setting as [11], i.e., using 5 × 10 4 analogies for the dataset and, for each, generating permutations as mentioned above to obtain 8 samples of the valid and 8 of the invalid class. Similarly, the testing is done on 5 × 10 4 base analogies and all the corresponding permutations. To evaluate the model, we use the balanced accuracy, i.e., the average of the accuracy of the valid class and the accuracy of the invalid class, thus ignoring the number of permutations seen in each class. For the experiment on the axiomatic setting (Sec. 3) and on the transfer between datasets (Sec. 4) we use 10 random seeds to ensure stability across random initialization and random selection of the base analogies. For the experiment on the transfer between languages (Sec. 5), due to the large number experiments to perform, a single random seed is used.

In our experiments, the classifier and the embedding model with the character vocabulary of the source domain are transferred to the target domain, and the evaluation is performed without finetuning the models on the target domain. For simplicity, we write 𝑠𝑜𝑢𝑟𝑐𝑒 → 𝑡𝑎𝑟𝑔𝑒𝑡 to denote the transfer from the source domain to the target domain.

The purpose of our first experiment is to study the impact of the training procedure of ANNc on the permutations it considers valid or invalid. To confirm that using different training procedures results in models that fit different axiomatic settings, we compare how models trained in the three settings described in Subsec. 2.2 (𝐶𝑃 , ¬𝐶𝑃 , and 𝐶𝑃 ) behave when transferred to the other settings.

We use the 11 languages of [11], i.e., Sigmorphon2016 and Japanese Bigger Analogy Test Set, and transfer between the three axiomatic settings described in Subsec. 2.2: 𝐶𝑃 (using 𝑃 + 𝐶𝑃 and 𝑃 − 𝐶𝑃 ), ¬𝐶𝑃 (using 𝑃 + ¬𝐶𝑃 and 𝑃 − ¬𝐶𝑃 ), and 𝐶𝑃 (using 𝑃 + 𝐶𝑃 and 𝑃 − 𝐶𝑃 ). Intuitively, the expected behavior is the following, also represented in the top left corner of Fig. 1: • each model is expected to perform best on when the source setting is the same as the target setting (𝐶𝑃 → 𝐶𝑃 , ¬𝐶𝑃 → ¬𝐶𝑃 , and 𝐶𝑃 → 𝐶𝑃 ); • both 𝐶𝑃 → 𝐶𝑃 and 𝐶𝑃 → 𝐶𝑃 are expected to perform poorly, as the source and target settings are incompatible w.r.t CP; • both 𝐶𝑃 and 𝐶𝑃 are expected to perform well on ¬𝐶𝑃 , as the permutations in ¬𝐶𝑃 are common to both 𝐶𝑃 and 𝐶𝑃 ; • the performances of ¬𝐶𝑃 → 𝐶𝑃 and ¬𝐶𝑃 → 𝐶𝑃 are hard to predict, as ¬𝐶𝑃 has no constraints w.r.t CP.

The results of the experiment are reported in Fig. 1, and for all languages we observe the expected results with minor variations. First, on the target ¬𝐶𝑃 setting, all the models perform equally, instead of ¬𝐶𝑃 → ¬𝐶𝑃 performing slightly better. Second, the performance of 𝐶𝑃 → 𝐶𝑃 and 𝐶𝑃 → 𝐶𝑃 are not as low as expected, with the peculiarity that 𝐶𝑃 → 𝐶𝑃 always outperforms 𝐶𝑃 → 𝐶𝑃 by roughly 10%.

On the one hand, these results confirm that the training procedure does have an impact on which permutations will be considered valid or invalid by the model. On the other hand, the observed results match the expected Z shape for all languages, which supports the intuitions used to construct the expected results. As these intuitions rely on the differences between the axiomatic settings, this experiment confirm that models of a specific axiomatic setting can be obtained with the corresponding training procedure.

The purpose of our second experiment is to check whether the model is able to generalize to a closely related domain with a slightly different distribution of morphological transformations. Indeed, for each language the morphological transformations in Sigmorphon2016 [25] and Sigmorphon2019 [28] are not exactly the same but the morphology of the language does not change.

For each language present in both Sigmorphon2016 and the high resource languages of Sigmor-phon2019, we consider two transfer directions and two baselines:

As shown in Fig. 2, the performance of the transferred model is comparable to or slightly lower than the non-transferred model. A significant correlation is noticeable between the performance of the transferred model and the character coverage of the target domains by the source domains (i.e., the number of characters present in both domains divided by the number of characters present in the target domain), with a Pearson correlation coefficient of 𝑟 = 0.9639 for 19 → 16 and 𝑟 = 0.7595 for 16 → 19. When normalizing the transfer performance by the performance trained on the target domain, the correlation goes up to 𝑟 = 0.9739 for 19→16 16 and 𝑟 = 0.8639 for 16→19 19 . A critical case of this correlation can be seen for Arabic, which is romanized in Sigmorphon2016 and not Sigmorphon2019, leading to a coverage close to 0%.

These results identify character coverage as a key factor in the transfer performance between strongly related domains. In this setup, the embedding model is the main source of performance loss.

To go beyond the limitations of character coverage, our third experiment leverages the large amount of multilingual data available in Sigmorphon2019 [28] to experiment following a similar intuition as in [12] but only between languages with similar alphabets.

We experiment with transfer between the high resource languages of Sigmorphon2019. We exclude Basque and Uzbek as they have less than 5 × 104 analogies. From the results of Sec. 4, we know that the amount of characters in common between the source and target domains strongly impacts the transfer performance. We extract clusters of languages sharing a significant part of their alphabets, and we transfer only within each cluster. This allows us to omit transfers likely to perform poorly due to the alphabet gap. We perform a total of 740 transfers, excluding cases where the source and target are the same. Compared to our other experiments, we reduce the number of test analogies from 5 × 10 4 to 5 × 103 and use a single random seed.

We use hierarchical clustering with the nearest point algorithm to get the clusters. Instead of the coverage between the source and the target language which is asymmetric, we use the Jaccard index2 between the alphabets of the languages as a similarity measure. Using a threshold of 40% on the Jaccard and excluding singletons, we extract four clusters of at least two elements 3 . Based on our observations on the coverage 4 , we consider relevant to include Romanian in both the Roman and Cyrilic clusters, and obtain the following language clusters named after the dominant alphabetic setting:

Once the languages with an overlap lower than 40% are eliminated, the Pearson correlation between the performance and the character coverage drops to 𝑟 = 0.6565 for clusters 2, 3 and 4. For cluster 1, the largest cluster, coverage and performance appear uncorrelated with 𝑟 = 0.0379. Similar values are observed when normalizing the transfer performance by the performance trained on the target domain. We report transfer performance for cluster 1 in Fig. 3.

We do not exclude that these correlations are influenced by the smaller amount of data used (only one seed, fewer testing analogies than usual). However, such a significant drop in correlation is unlikely if only this bias is involved. In fact, the tendencies we observe in the performance matrix indicate that the performance is linked to the language being used as a source language (e.g., horizontal bar for English), and to the one being used as a target language (e.g., vertical bars for Asturian and German). This behavior is likely due to either the quality of the learned model (how well it performs in general) or to the morphological similarities of some languages (at least within Sigmorphon2019). We exclude the former hypothesis, as only tendencies in the behavior as a source language (i.e., horizontal bars) would be observed, while we mostly observe tendencies in the behavior as a target language (i.e., vertical bars).

To confirm the influence of language similarities on performance, we explore hierarchical clustering within cluster 1 to study which key groups appear. When considering the behavior of the language as a target domain (i.e., using performance from different source languages as a features for the clustering) rather than as a source domain, the clusters are more distinct. We focus on clusters extracted from the former, which can be seen in the dendrogram in Fig. 4. As a first analysis, we compare the corresponding clusters with language families as defined in Wikipedia. The Wikipedia page of each language contains a box with key information, the infobox. We use the "Language family" field of the infobox in the page of each language to determine how closely related they are, after minor corrections. The tree structure in Appendix Fig. 3, summarizes this information, with the leaves colored to match the colors of the clusters on Fig. 4. We find that the small clusters, which are the most easily distinguishable by the clustering algorithm, correspond to closely related groups of languages. More precisely, the orange cluster contains Western Romance languages (Asturian, Portuguese, Spanish, and French), the purple cluster contains all the Bantu languages (Zulu and Swahili), and the green cluster contains Slavic languages: West Slavic languages (Slovak, Polish, and Czech) and slightly further the South Slavic language (Slovene). Finally, Irish is isolated and the red cluster contains all the remaining languages, even if distinct sub-clusters can be found: the (Finnish and Estonian) sub-cluster corresponds to Finnic languages and the (Romanian and Italian) sub-cluster contains the non-Western Romance languages. Other sub-clusters of the red cluster do not correspond to specific language families, like the (Kurmanji and Dutch) and the (Welsh and Dutch) sub-clusters.

From these results, we confirm that the morphological similarities of the languages are reflected in the model behavior during transfer. This indicates that our approach models morphological rules that can be transferred to related languages. However, it is clear that transfer in some clusters performs better than in others, though we are not yet able to provide explanations. Also, the performance is most likely influenced by the fact that the data in Sigmorphon2019 do not represent the full morphology of each language.

In this work, we use transfer to study the behavior of the ANNc analogy model when changing the axiomatic setting, the dataset, or the language of the analogies.

With results in 11 languages, we empirically confirmed that it is possible to model different axiomatic settings of analogy by changing the sets of permutations used when training ANNc. This highlights the importance of careful consideration on the axiomatic setting to use for data augmentation depending on the application, as it can significantly change model behavior. These results suggest that it is possible to determine the axiomatic setting matching a domain from data. Indeed, if domain data containing valid and invalid analogies is available, an ANNc model can be learned and matched against multiple axiomatic settings to find the one fitting the domain. This kind of method could provide empirical arguments to define the notion of analogy in specific domains.

We also extended previous results on tranferability between languages and complemented it with transferability between datasets. Empirical results confirm previous hypotheses on the alphabet gap issue. We found that in many cases it is possible to use the proximity in the Wikipedia language families to predict the performance of transferred models, which confirm the transferability of morphological analogies between languages. These results suggest that analogies and transfer could be used to empirically study morphological similarities between languages. Such similarities can be useful in language learning, by selecting languages known by a learner and having similar morphology to a language to learn. They could also be used to automatically create a data-driven language classification. Slovak, Slovene, Sorani, Spanish, Swahili, Turkish, Welsh, and Zulu). The language families are extracted from the "Language family" field of the infobox in the Wikipedia page of each language.