The traditional approach to morphological inflection (the task of modifying a base word (lemma) to express grammatical categories) has been, for decades, to consider lexical entries of lemma-tag-form triples uniformly, lacking any information about their frequency distribution. However, in production deployment, one might expect the user inputs to reflect a real-world distribution of frequencies in natural texts. With future deployment in mind, we explore the incorporation of corpus frequency information into the task of morphological inflection along three key dimensions during system development: (i) for train-dev-test split, we combine a lemmadisjoint approach, which evaluates the model's generalization capabilities, with a frequency-weighted strategy to better reflect the realistic distribution of items across diferent frequency bands in training and test sets; (ii) for evaluation, we complement the standatrydpe accuracy (often referred to simply as accuracy), which treats all items equally regardless of frequency, wit htoken accuracy, which assigns greater weight to frequent words and better approximates performance on running text; (iii) for training data sampling, we introduce a method novel in the context of inflectionfr,equency-aware training, which explicitly incorporates word frequency into the sampling process. We show that frequency-aware training outperforms uniform sampling in 26 out of 43 languages.
UPOS=ADV
Degree=Sup lemma morphological tags Inflection system b e s
t form superlative, inflected form “best” (figure adopted from Sourada and Straková1[]).
Morphological inflection, the task of producing the inflected form from a base word form (lemma) to
express grammatical categories (see Figur1e), has become a highly active research area in recent years,
largely driven by the SIGMORPHON shared tasks held from 2016 to 20232,[
However, in production deployment of a morphological inflection system, we may expect the user input to reflect a real-world distribution of frequencies: users will probably enter generally frequent
CEUR Workshop
ISSN1613-0073
words more commonly than rare words. This fact has been ignored for decades until quite recently
[
With a focus on future deployment, we address this gap by exploring the incorporation of corpus frequency information into the task of morphological inflection along three key axes: train-dev-test split, data sampling during training, and evaluation, experimenting with Universal Dependencies (UD) [11] corpora in 43 languages.
Our contributions are as follows:
1. We experiment with a new train-dev-test split technique that satisfies both recent methodological
standards (being lemma-disjoint1[
We make all our work open-source by publicly releasing our code at GitHuhb:ttps://github.com/ tomsouri/corpus-frequencies-in-inflecti o.n
In the field of morphological inflection, there has been a lot of recent work, mostly pushed forward by
the SIGMORPHON shared tasks2[
Until 2021, common practice was to split the lexicon of lemma-tag-form triplets into train, dev, and test uniformly randomly, without any additional control over the lemma overlap or feature overlap between train and test. Liu and Hulde[n16] showed that lemma overlap artificially inflates the model’s performance. Goldman et al.[12] investigated this problem more and suggested a lemma-disjoint split technique, which was then adopted by the next installation of the SIGMORPHON shared t8]a.sk [
Finally, Kodner et al.[10] have focused on diferent split practices and thoroughly evaluated and discussed their properties. In addition to the issues with overlap control, he also revealed drawbacks of standard uniform sampling. Uniform sampling over a set of triplets lemma-tag-form means that we sample them uniformly randomly into train, dev, and test, regardless of their corpus occurrence frequencies.1 According to Kodner et a[l.10], this leads to an unnatural train-test split, with a bias towards low-frequency and thus more regular and reliable training items. To mitigate this problem, they suggest a new,frequency-weighted split strategy: the types (lemma-tag-form triples) are partitioned at random weighted by their frequency in a corpus. The train set is sampled first, then the dev+test is sampled and then uniformly separated. This sampling biases the train set towards items with high corpus frequency, and dev and test towards low-frequency items, leading to a more realistic traintest distribution. Sourada and Strako[v1á] have built on this and proposed a new split technique 1This is the only approach one may use if working with a morphological lexicon that only contains the lemma-tag-form triplets and lacks the information about their frequency in natural text. that combines the frequency-weighted split with the lemma-disjoint approach. We employ this new technique, as the realistic train-test frequency distribution fits better our needs, and lemma-disjoint split allows evaluating generalization abilities.
When evaluating morphological inflection, exact-match accuracy is used, as the ratio of correctly predicted forms (also calledtype accuracy or simply accuracy, see Equation 1()), on a lexical test set, not considering the corpus frequencies of the items.
type acc= 1 ∑ 1 (1) total item⋅scorrect items
Nicolai et al.[13] proposed to use the so calletdoken accuracy (see Equation 2()), where the items are weighted by their corpus frequency.
token acc= 1 ∑ occurrences(item) (2) total occurrencecsorrect
items
They argue that although type accuracy is easy to compute from the morphological lexiconms,aiyt “ over-represent rare forms, which are often regularly inflected and thus simpler to produce ” [13, Section 7]. However, they conclude that computing token accuracy is infeasible because it would require a corpus of running text annotated for morphological inflection, and they themselves do not evaluate using that metric, but only its approximation. To our knowledge, no further work on morphological inflection used the token accuracy for the evaluation.
To meet the standards, we report traditional type accuracy and also include the proposed token accuracy, which better reflects real-world inflection system usage.
To the best of our knowledge, we are not aware of previous relevant work in morphological inflection that incorporates corpus frequencies in (supervised) machine learning context, especially a neuralnetwork-based one.
With the focus on future deployment, we decide to perform the frequency-weighted train-dev-test split (for a realistic train-test distribution), evaluate with token accuracy, which corresponds to evaluating on a running text and therefore reflects a real-world usage, and experiment with frequency-aware training. To be able to experiment with the techniques using corpus frequency information, we need to first have that information. The commonly used data source, UniMorph (UM), is a lexicon and completely lacks information on frequency.
We consider several possible approaches to obtain morphological data (lemma-tag-form triples) along
with their corpus frequencies:
1. Use a manually, morphologically annotated corpus of running text, such as Universal
Dependencies (UD) [17] directly as the data source. This approach has the advantage that it is simple and
does not require any matching or tag conversion. Furthermore, if UD is used as the annotated
corpus, the morphological tags are already in the CoNLL-U format, which is suitable for
connecting the system into pipelines with other systems that work with UD (for example, UDPipe
[18]). However, using UD as the only source of morphologically annotated data could lead to
lower coverage of lemmas/forms, compared to using UniMorph. This approach was used by
Sourada and Straková[
Nevertheless, none of these techniques to resolve the ambiguities removes the problem of still having a high number of UM items with zero corpus occurrences and thus throwing away a lot of data. 3. Use option 2) and do not drop UM items with 0 count: Since we work with corpus counts as numbers, we could assign some constant (less than 1) as the corpus count to all theoretically possible forms (those present in UM) that are not found in the corpus of running text.
After carefully considering the advantages and drawbacks of each method, we select the first method, using the annotated corpus of running text (UD) directly as the data source, as it is the most straightforward approach, while keeping in mind that it leads to lower coverage of lemmas/forms, and we leave the other approaches for future work.
From Universal Dependencies, we select corpora for 43 language2s,specifically, Indo-European languages written in the Latin script and languages spoken in Europe. We lexicalize them by extracting unique lemma-tag-form triples along with their occurrence counts for training and evaluation. For hyperparameter tuning, we select a subset of five development languages: Czech, English, Spanish, Breton, and Basque.
The canonical split of UD is not suitable for the task of morphological inflection, because a naive extraction of lemma-tag-form triples from the canonical UD split naturally produces a train-test overlap, both on the item level (lemma-tag-form triples), and on the lemma level. Therefore, we need to resplit the data. We aim for potential future deployment, therefore, we need a frequency-weighted split, as advocated by Kodner et al[.10], for a realistic train-test frequency distribution. However, their proposed split is not lemma-disjoint and thus breaks recent methodological requireme1n2t,s8[] by obscuring an evaluation of the model’s generalization abilities. To meet both requirements, we employ a new split technique proposed by Sourada and Strakov[á1], which combines the lemma-disjoint approach with the frequency-weighted approach. It consists of the following steps:
1. obtain the total occurrence count for each lemma in the corpus, 2See Table 4 for the selected languages and corpora.
hyperparam
value attn drop 0.1 drop 0.15 activation drop 0.35 layer drop 0.2 grad-clip-norm 1.0 hyperparam value 2. sample lemmas first into the train set, randomly, weighted by the occurrence counts (until we obtain a desired amount of data in the train set), 3. sample lemmas from the rest uniformly randomly (no weighted sampling) to the dev set (until a desired amount of data in the dev set is achieved), 4. put the rest of lemmas into the test set.
We use a train:dev:test ratio 8:1:1 in terms of total occurrence counts.
For evaluating the inflection systems, we use the standatrydpe accuracy (also called simply accuracy, see Equation (1)), working at the level of a lexicon, and complement it wittokhen accuracy (see Equation 2()) [13], which gives more weight to frequent words.
In its natural interpretation, type accuracy corresponds to evaluation on a lexicon of unique lemmatag-form triples (ignoring frequencies), while token accuracy corresponds to evaluation over a running text annotated with lemmas and tags (naturally reflecting frequencies). Therefore, token accuracy better reflects real-world usage of an inflection system. Clearly, token accuracy is better suited for distinguishing between two models when one is superior in predicting more frequent words — a situation that type accuracy cannot detect. 3.4. Model For model training, we use a state-of-the-art, small-capacity, encoder-decoder, sequence-to-sequence Transformer architecture, trained from scratch on the inflection data, with the lemma-tag pair as input and the inflected form as output (see Figur1e), as standard in the field. We tune the model’s hyperparameters on the five development languages (see Table1 for the final values). The tuned model (without frequency-aware training) outperforms both SIGMORPHON neural baselin1e9s][in all five development languages.
As we give more weight to frequent words in evaluation with token accuracy, one might ask whether it could be beneficial to give more weight to such words also in training. To support this, we introduce sample weights and use weighted random sampling to sample training data items into batches.
If we wanted to mimic training on a running corpus, the direct approach would be to use raw
frequency counts of the data items as sample weights. Another specific case is setting all sample weights
to 1, which would lead to uniform sampling, which is a standard approach in morphological inflection.
For a continuous scale between these two extremes, we introduccoerpaus-frequency temperature,
denoted as , a real-valued hyperparameter, usually ∈ [
The probability that the training data itemwill be sampled in an individual sampling step is given by the sample weight with normalization:
∑ ( ()) The ratio of the probabilities of two data it emasnd being sampled in a single sampling step is: () = () ()
( () ) = ( ()
) () =1 () =
() ∑ () = ()
=0 () = 1 3.5.1. Specific Values of Corpus-Frequency Temperature and Their Interpretation With = 1 , sample weights are equal to the raw corpus frequencies, and the probability that the training data item will be sampled in an individual sampling step is then (3) (4) (5) (6) (7) (8)
The value = 0 corresponds to the uniform sample weights, and the probability that any specific data item will be sampled in an individual sampling step is then
With = 1/2 , sample weights are equal to the square root of raw corpus frequencies. Then i tem with raw occurrence count 400 would be sampled wit√h400 = 20 times higher probability than item with raw occurrence count 1 (with sample weigh√t1 = 1).
Corpus-frequency temperature values from the interv[a0l, 1] correspond to the continuous scale between the two extremes (uniform sample weights ignoring the corpus frequencies w=it0h, and sample weights equal to the raw corpus frequencie=s 1 ). Nevertheless, it is technically possible to set it also to values outside the range, and it also has a natural meaning.
The value = −1 leads to sample weights being the inverse of the raw corpus counts. Then the ratio of the probabilities of two data item,s being sampled in a single sampling step is =−1 () =−1 () = () () That is, less frequent data items are sampled more frequently during training. The va l=ue2 leads to extremely emphasizing the frequent data items. In the example o(f) = 400 , () = 1 , with = 2 , would be sampled 160,000 times more frequently tha n during training.
To measure the efect of frequency-aware training, we experiment with the tuned Transformer model (as described in Section3.4), trained for 960 epochs with checkpoint selection based on token accuracy on the dev set, with diferent values of corpus-frequency temperature. Our hypothesis is that giving more importance to frequent words during training could improve the performance of the system when evaluating with token accuracy, which itself gives more importance to frequent words in evaluation.
We experiment with diferent values of corpus-frequency temperature)(from the range[
We observe tha t= 2.0 leads to a complete failur3e,probably because of the extreme emphasis on
the frequent words. When we focus on∈ [
Quite interestingly, there are also languages in which negative values of corpus-frequency temperature (which give more weight to rare forms during training) lead to the best performance, such as English, Galician, and Slovak. In other languages, such as Breton, Lithuanian, Low Saxon, or Ukrainian, the performance at diferent temperature levels is quite noisy and does not provide any insight.
On average in all languages (rightmost column in the bottom part of Ta2b),lethere is a continuous improvement when the temperature increases to 0.5, and then a continuous decline, indicating that if we shall select a single temperature value to be used in all languages, it would =be0.5 (sample weights equal to the square root of the raw corpus occurrence counts).
To sum up, for almost half of the languages the performance corresponds to our hypothesis that giving more importance to frequent words during training should increase the performance of the system in terms of token accuracy: either steadily with the best results achieve d=w1it,hweights being the raw corpus occurrence counts (Basque, Old French, Turkish), or rather up to the square root of occurrence counts =( 0.5 ). On the other hand, there are languages where frequency-aware training does not seem to have a major efect, and even some with opposite results (English, Galician, Slovak). 4.2. Efect of Corpus-Frequency Temperature Value on Type Accuracy In Table 3, we report a similar comparison, now evaluated with type accuracy (which disregards the corpus occurrence counts of the evaluation items, treating all items uniformly). Although the results appear to be slightly more noisy than with token accuracy, we observe similar trends in most of the languages: the continuous improvement in Basque and Turkish when increasing the temperature value to 1.0, and the languages with a peak close t=o 0.5 (Belarusian, Dutch, German, Irish, Italian, Latin, Low Saxon, Portuguese, Scottish Gaelic, and Welsh). This is quite surprising: why should we benefit from giving more importance to frequent words in training if we disregard the frequencies in evaluation? We discuss it further in Sectio5.n
Another interesting observation is that although token accuracy and type accuracy difer in the absolute value (compare English with = 0.0 with 96.22% in token accuracy and 93.37% in type accuracy), they seem to be similar in terms of model ranking, at least when comparing systems trained with the same hyperparameters with the only diference in temperature value (and it is a trend we observed also during the development experiments with hyperparameter tuning).
To further investigate whether using type accuracy as the main objective during the experiments could actually afect the final performance in token accuracy (our primary objective for deployment of the system), we conduct the following experiment: we first train a system, selecting the checkpoint based on token accuracy, and evaluate using token accuracy. Then, we train another system, selecting 3With the only exception of Old French, wher=e 2.0 leads to the best performance. the checkpoint based on type accuracy, and again evaluate using token accuracy. If a significant drop in performance is observed, it would suggest that relying on type accuracy for design decisions during experimentation might lead to a worse final performance in token accuracy.
For diferent maximum numbers of training epochs (60, 120, 240, 480, and 960), we run five instances4 of the proposed experiment on the five development languages. We find that performance drops are negligible: in Basque, the largest absolute drop is 0.7%; in Spanish and English, it is 0.3%; in Czech, 0.2%; and in Breton, 0.9%.
The fact that the diferences are minor may be attributed to the nature of our splits — data with high corpus-frequency counts tend to appear in the training set, resulting in a limited variance in the development set. The diference between token accuracy and type accuracy should be explored further, as we have not shown a substantial diference between them in terms of the relative ordering of models under our split and evaluation conditions.
For the evaluation on the test set, we use four diferent systems: a naciovpey baseline, which copies the input lemma to output, a model without frequency-aware training (with uniform sampling to batches, ignoring the corpus frequencies, denoted b y= 0.0 ), a model with frequency-aware training (weighted random sampling into batches during training) wi t=h 0.5 (the best value in average over all languages, weights being the square root of the raw corpus occurrence count), and finallbye,st, a model that for each language separately selects the best temperature value based on the performance on the dev set (token accuracy) and uses frequency-aware training with the selected temperature value.
In terms of token accuracy (see Tabl4e, left part), frequency-aware training with best outperforms uniform sampling in 26 languages and achieves the overall best result in 23 languages out of 43 languages. It is outperformed b y= 0.5 in 12 languages and by uniform sampling in additional 5 languages. That is relatively disappointing, and it probably shows overfitting to the dev set (by hyperparameter tuning, checkpoint selection and temperature value selection), because otherwise we would expe ctbetsthe model to outperform the rest of models. Compared to uniform sampl in=g,0.5 brings improvement in 23 languages and outperforms all other models in 13 languages. In two languages (Low Saxon and Old French), none of our models outperformed thceopy baseline.
In terms of type accuracy (see Tabl4e, right part), the results are more straightforwa rbde:st improves over uniform sampling in 41 out of 43 languages, achieving the overall best result in 35 languages. It is outperformed by = 0.5 in only 6 languages. As in token accuracy, it is outperformed bcyopy baseline in two languages. It is surprising that when choosing the temperature value based on token accuracy performance on the dev set, on the test set it helps more the type accuracy than the token accuracy. It would be beneficial to explore this more in future work and to seek an explanation.
Regarding the comparison of token accuracy and type accuracy, note that at least in the test results we can observe some diferences ( best system clearly better than the rest according to type accuracy, yet not so clearly according to token accuracy).
We found that, in at least some languages, frequency-aware training provides a clear benefit for both token accuracy and type accuracy.
If the data split was not lemma-disjoint, the reason for its positive efect on token accuracy would be fairly straightforward: Suppose a frequent lemma A appears in the training set. Under frequency-aware training, it receives a higher weight, allowing the model to learn it more efectively. If the same lemma A also occurs in the development or test set, the model will (likely) inflect it correctly. Since lemma A is frequent, its correct inflection contributes disproportionately to the overall token accuracy, due to the higher evaluation weight it receives. 4Difering by the random seed used for initialization.
Nevertheless, the split is lemma-disjoint, which raises a question: Why does frequency-aware training help? Furthermore, why should it help even in terms of type accuracy? Why does it help only in some languages, and why we even achieve an opposite efect some other languages?
One possible explanation for why frequency-aware training is beneficial, even with the lemmadisjoint split, is that frequent words in the dev/test sets tend to inflect in a similar way to frequent words in the training set. If this is the case, assigning greater weight to frequent words during training should help the model inflect frequent dev/test words more accurately, thereby increasing token accuracy. A more straightforward explanation is that frequency-aware training helps due to a derivational leak. For instance, a frequent lemma in the training set, such as the Czech vejírtb” “(“to go”), may have derived lemmas in the dev/test sets, e.gp.,ř“ijít” (“to come”), “projít” (“to go through”). These derived forms are also frequent and thus receive a higher weight during evaluation. Since derived verbs in Czech inflect in very similar ways, giving more weight toj“ít” during training improves performance on p“řijít” and “projít” during evaluation.
As for why frequency-aware training might improve also type accuracy, the gain may come from the fact that frequent lemmas occur in corpus not only with a higher occurrence count (reflected in token accuracy) but also with a greater variety of distinct forms. Since type accuracy treats each lemma-tag-form triple equally, this increased form diversity could contribute to the improvement.
Regarding why frequency-aware training helps in some languages but has the opposite efect in others, its success appears to depend on factors that are not yet well understood. Investigating these in future work would be worthwhile. Possible explanations include linguistic properties such as morphological richness and regularity, or corpus-related factors such as data size and variability.
In addition, other ways of obtaining corpus frequencies for morphological data could be explored, as described in Section3.1, especially using large raw data to extract the frequencies with some tag-based back-of for disambiguation, not dropping the UniMorph items with zero actual corpus occurrences.
Kodner et al.[10] argues that frequency-weighted split better reflects real-world frequency distribution of train-test, and so we use it. However, it would be beneficial to explore more the split compared to uniform split, to experimentally answer whether it would hurt the real-world performance, if we developed a system on a uniform train-dev-test split.
Also, it would be worth experimenting with the canonical UD train-dev-test split (the original split of UD, not lemma-disjoint).
In terms of metrics (token accuracy and type accuracy), it would be worth further exploring whether there is a diference in terms of what metric we use during the development of a model, in contrast to the metric used in the final evaluation (that is, tuning hyperparameters for two systems, each according to one of the metrics, and then evaluate both systems according to token accuracy and see if the performance difers). Furthermore, it would be particularly interesting to measure the diferences between them on a uniform (not frequency-weighted) split of UD. Moreover, enriching a standard benchmark like SIGMORPHON with corpus frequency information and re-evaluating the shared task using our token accuracy metric could potentially yield valuable insights.
We explored the incorporation of corpus frequencies in the task of morphological inflection along three key dimensions: train-dev-test split of data (frequency-weighted lemma-disjoint split), training data sampling (frequency-aware training), and evaluation. We designed and consistently applied a previously suggested but never before used evaluation metric that assigns greater importance to frequent word forms (token accuracy). We demonstrated that in our setting (lemma-disjoint, frequency-weighted split), the diference between the original metric (type accuracy or just accuracy) and token accuracy is minimal with respect to the relative ranking of diferent systems. Our pioneering experiments with frequency-aware training showed promising results, improving over uniform sampling in 26 out of 43 languages. The overall best weighting scheme seems to be with corpus frequency temperature 0.5 (each sample is given a weight corresponding to the square root of its corpus frequency).
All three dimensions (frequency-weighted split, frequency-aware training, and token accuracy) are worth further investigation.
This research was supported by the Johannes Amos Comenius Programme (P JAC) project No. CZ.02.01.01/00/22_008/0004605, Natural and anthropogenic georisks.
Computational resources for this work were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
The work described herein uses resources hosted by the LINDAT/CLARIAH-CZ Research Infrastructure (projects LM2018101 and LM2023062, supported by the Ministry of Education, Youth and Sports of the Czech Republic).
We thank the anonymous reviewers for their valuable comments.
During the preparation of this work, the authors used GPT-4 and Writefull to check grammar and spelling. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [6] T. Pimentel, M. Ryskina, S. J. Mielke, S. Wu, E. Chodrof, B. Leonard, G. Nicolai, Y. Ghanggo Ate, S. Khalifa, N. Habash, C. El-Khaissi, O. Goldman, M. Gasser, W. Lane, M. Coler, A. Oncevay, J. R. Montoya Samame, G. C. Silva Villegas, A. Ek, ..., E. Vylomova, SIGMORPHON 2021 shared task on morphological reinflection: Generalization across languages, in: Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, Association for Computational Linguistics, Online, 2021, pp. 229–259. URhLt: tps://aclanthology. org/2021.sigmorphon-1.25. doi:10.18653/v1/2021.sigmorphon-1.25. [7] J. Kodner, S. Khalifa, K. Batsuren, H. Dolatian, R. Cotterell, F. Akkus, A. Anastasopoulos, T. Andrushko, A. Arora, N. Atanalov, G. Bella, E. Budianskaya, Y. Ghanggo Ate, O. Goldman, D. Guriel, S. Guriel, S. Guriel-Agiashvili, W. Kieraś, A. Krizhanovsky, ..., E. Vylomova, SIGMORPHON–UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection, in: Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, Association for Computational Linguistics, Seattle, Washington, 2022, pp. 176–203. URLh:ttps://aclanthology.org/2022.sigmorphon-1.1.9 doi:10.18653/v1/2022.sigmorphon-1.19. [8] O. Goldman, K. Batsuren, S. Khalifa, A. Arora, G. Nicolai, R. Tsarfaty, E. Vylomova, SIGMORPHON– UniMorph 2023 shared task 0: Typologically diverse morphological inflection, in: Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 117–125. URL: https://aclanthology.org/2023.sigmorphon-1.1.3doi:10.18653/v1/2023.sigmorphon-1.13. [9] K. Batsuren, O. Goldman, S. Khalifa, N. Habash, W. Kieraś, G. Bella, B. Leonard, G. Nicolai, K. Gorman, Y. G. Ate, M. Ryskina, S. Mielke, E. Budianskaya, C. El-Khaissi, T. Pimentel, M. Gasser, W. A. Lane, M. Raj, M. Coler, ..., E. Vylomova, UniMorph 4.0: Universal Morphology, in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, S. Piperidis (Eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 840–855. URL: https://aclanthology.org/2022.lrec-1.8.9/ [10] J. Kodner, S. Payne, S. Khalifa, Z. Liu, Morphological inflection: A reality check, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 6082–6101. URL:https://aclanthology.org/2023.acl-long.33.5d/oi:10. 18653/v1/2023.acl-long.335. [11] D. Zeman, J. Nivre, M. Abrams, E. Ackermann, N. Aepli, H. Aghaei, Ž. Agić, A. Ahmadi, L. Ahrenberg, C. K. Ajede, S. F. Akkurt, G. Aleksandravičiūtė, I. Alfina, A. Algom, K. Alnajjar, C. Alzetta, E. Andersen, L. Antonsen, T. Aoyama, ..., R. Ziane, Universal dependencies 2.14, 2024. URL: http://hdl.handle.net/11234/1-550,2LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. [12] O. Goldman, D. Guriel, R. Tsarfaty, (un)solving morphological inflection: Lemma overlap artificially inflates models’ performance, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 864–870. URL: https://aclanthology.org/2022.acl-short.9.6d/oi:10.18653/v1/2022.acl-short.96. [13] G. Nicolai, D. Lewis, A. D. McCarthy, A. Mueller, W. Wu, D. Yarowsky, Fine-grained morphosyntactic analysis and generation tools for more than one thousand languages, in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 3963–3972. URL: https://aclanthology.org/2020.lrec-1.48.8/ [14] C. Kirov, R. Cotterell, J. Sylak-Glassman, G. Walther, E. Vylomova, P. Xia, M. Faruqui, S. J. Mielke, A. McCarthy, S. Kübler, D. Yarowsky, J. Eisner, M. Hulden, UniMorph 2.0: Universal Morphology, in: N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, T. Tokunaga (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan, 2018. URLh:ttps://aclanthology.org/L18-129.3/ [15] A. D. McCarthy, C. Kirov, M. Grella, A. Nidhi, P. Xia, K. Gorman, E. Vylomova, S. J. Mielke, G. Nicolai, M. Silfverberg, T. Arkhangelskiy, N. Krizhanovsky, A. Krizhanovsky, E. Klyachko, A. Sorokin, J. Mansfield, V. Ernštreits, Y. Pinter, C. L. Jacobs, R. Cotterell, M. Hulden, D. Yarowsky, UniMorph 3.0: Universal Morphology, in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 3922–3931. URLh:ttps://aclanthology. org/2020.lrec-1.483/. [16] L. Liu, M. Hulden, Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 739–749. URL: https://aclanthology.org/2022.acl-short.8.4d/oi:10.18653/v1/2022.acl-short.84. [17] J. Nivre, M.-C. de Marnefe, F. Ginter, J. Hajič, C. D. Manning, S. Pyysalo, S. Schuster, F. Tyers, D. Zeman, Universal Dependencies v2: An evergrowing multilingual treebank collection, in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 4034–4043. URL: https://aclanthology.org/2020.lrec-1.49.7/ [18] M. Straka, J. Straková, Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe, in: Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 88–99.
URL: http://www.aclweb.org/anthology/K/K17/K17-3009.pd.f [19] S. Wu, R. Cotterell, M. Hulden, Applying the transformer to character-level transduction, in: P. Merlo, J. Tiedemann, R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online, 2021, pp. 1901–1907. URLh:ttps://aclanthology.org/2021.eacl-main.16.3/ doi:10.18653/v1/2021.eacl-main.163.