English. Machine learning offers two basic strategies for morphology induction: lexical segmentation and surface word relation. The first one assumes that words can be segmented into morphemes. Inducing a novel inflected form requires identification of morphemic constituents and a strategy for their recombination. The second approach dispenses with segmentation: lexical representations form part of a network of associatively related inflected forms. Production of a novel form consists in filling in one empty node in the network. Here, we present the results of a recurrent LSTM network that learns to fill in paradigm cells of incomplete verb paradigms. Although the process is not based on morpheme segmentation, the model shows sensitivity to stem selection and stem-ending boundaries.
Italiano. La letteratura offre due strategie di base per l’induzione morfologica. La prima presuppone la segmentazione delle forme lessicali in morfemi e genera parole nuove ricombinando morfemi conosciuti; la seconda si basa sulle relazioni di una forma con le altre forme del suo paradigma, e genera una parola sconosciuta riempiendo una cella vuota del paradigma. In questo articolo, presentiamo i risultati di una rete LSTM ricorrente, capace di imparare a generare nuove forme verbali a partire da forme giï¿oe note non segmentate. Ciononostante, la rete acquisisce una conoscenza implicita del tema verbale e del confine con la terminazione flessionale.
Morphological induction can be defined as the task of singling out morphological formatives from fully inflected word forms. These formatives are understood to be part of the morphological lexicon, where they are accessed and retrieved, to be recombined and spelled out in word production. The view requires that a word form be segmented into meaningful morphemes, each contributing a separable piece of morpho-lexical content. Typically, this holds for regularly inflected forms, as with Italian cred-ut-o ’believed’ (past participle, from CREDERE), where cred- conveys the lexical meaning, and -ut-o is associated with morphosyntactic features. A further assumption is that there always exists an underlying base form upon which all other forms are spelled out. In an irregular verb form like Italian appes-o ’hung’ (from APPENDERE), however, it soon becomes difficult to separate morpholexical information (the verb stem) from morpho-syntactic information.
A different formulation of the same task
assumes that the lexicon consists of fully-inflected
word forms and that morphology induction is
the result of finding out implicative relations
between them. Unknown forms are generated by
redundant analogy-based patterns between known
forms, along the lines of an analogical
proportion such as: rendere ‘make’ :: reso ‘made’ =
appendere ‘hang’ :: appeso ‘hung’. Support
to this view comes from developmental
psychology, where words are understood as the
foundational elements of language acquisition, from
which early grammar rules emerge
epiphenomally
To understand how word inflection can be
conceptualised as a word relation task, it is useful to
think of this task as a cell-filling problem
The task can be simulated by training a learning model on a number of partial paradigms, to then complete them by generating missing forms. Training consists of <lemma_paradigm cell, inflected form> pairs. A lemma is not a form (e.g. credere), but a symbolic proxy of its lexical content (e.g. CREDERE). Word inflection consists of producing a fully inflected form given a known lemma and an empty paradigm cell. 2.1
Methods and materials Following Malouf (in press), our LSTM network (Figure 1) is designed to take as input a lemma (e.g. CREDERE), a set of morphosyntactic features (e.g. PRES_IND, 3, S) and a sequence of symbols (<crede>)1 one symbol st at a time, to output a probability distribution 1‘<’ and ‘>’ are respectively the start-of-word and the end-of-word symbols lexeme (50)
… symbol (t) (33) tense mood (5) person
(4) number (3) z(t)
LSTM(t)
1:n
over the upcoming symbol st+1 in the sequence:
p(st+1jst;CREDERE, PRES_IND, 3, S). To
produce the form <crede>, we take the start symbol
‘<’ as s1, use s1 to predict s2, then use the
predicted symbol to predict s3 and so on, until ‘>’ is
predicted. Input symbols are encoded as mutually
orthogonal one-hot vectors with as many
dimensions as the overall number of different symbols
used to encode all inflected forms. The
morphosyntactic features of tense, person and number are
given different one-hot vectors, whose dimensions
equal the number of different values each
feature can take.2 All input vectors are encoded by
trainable dense matrices whose outputs are
concatenated into the projection layer z(t), which is
in turn input to a layer of LSTM blocks (Figure
1). The layer takes as input both the
information of z(t), and its own output at t–1.
Recurrent LSTM blocks are known to be able to capture
long-distance relations in time series of symbols
We tested our model on two comparable sets
of Italian and German inflected verb forms
(Table 1), where paradigms are selected by sampling
the highest-frequency fifty paradigms in two
reference corpora
To meaningfully assess the relative computational difficulty of the cell-filling task, we calculated a simple baseline performance, with 695 forms of our original datasets selected for training, and 55 for testing.4For this purpose, we used the baseline system for Task 1 of the CoNLLSIGMORPHON-2017 Universal Morphological Reinflection shared task.5The model changes the infinitive into its inflected forms through rewrite rules of increasing specificity: e.g. two Italian forms such as badare ‘to look after’ and bado ‘I look after’ stand in a BASE :: PRES_IND_3S relation. The most general rule changing the former into the latter is -are -> -o, but more specific rewrite rules can be extracted from the same pair: 3The full data set is available at http://www. comphyslab.it/redirect/?id=clic2017_data. Each training form is administered once per epoch, and the number of epochs is a function of a “patience” threshold. Although a uniform distribution is admittedly not realistic, it increases the entropy of the cell-filling problem, to define some sort of upper bound on the complexity of the task.
4Test forms were selected to constitute a benchmark for evaluation. We made it sure that a representative sample of German and Italian irregulars were included for evaluation, provided that they could be generalised on the basis of the training data available.
5https://github.com/sigmorphon/ conll2017(written by Mans Hulden).
German test CoNLL baseline 128-blocks 256-blocks 512-blocks Italian test baseline 128-blocks 256-blocks 512-blocks all -dare -> -do, -adare -> -ado, -badare -> -bado. The algorithm then generates the PRES_IND_3S of - say - diradare ’thin out’, by using the rewrite rule with the longest left-hand side matching diradare (namely -adare > -ado). If there is no matching rule, the base is used as a default output.
The algorithm proves to be effective for regular forms in both languages (Table 2). However, per-word accuracy drops dramatically on German irregulars (0.23), and Italian irregulars (0.5). The same table shows accuracy scores on test data obtained by running 128, 256 and 512 LSTM blocks. Each model instance was run 10 times, and overall per-word scores are averaged across repetitions.6
The CoNLL baseline is reminiscent of
Albright and Hayes’ (2003) Minimal Generalization
Learner, inferring Italian infinitives from first
singular present indicative forms
Italian training (256−cell LSTM)
Italian training (512−cell LSTM) 1 is less criterial for stem allomorphy than the Italian one. LSTMs are considerably more robust in this respect. Memory resources allowing, they can keep track of local syntagmatic constraints as well as more global, paradigmatic constraints, whereby all paradigmatically-related forms contribute to fill in gaps in the same paradigm. For example, knowledge that a paradigm contains a few stem allomorphs is good reason for an LSTM to produce a stem allomorph in other (empty) cells. The more systematic the distribution of stem alternants is across the paradigm, the easier for the learner to fill in empty cells. German conjugation proves to be paradigmatically well-behaved.
An LSTM recurrent network has no information about the morphological structure of input forms. Due to the predictive nature of the production task and the LSTM re-entrant layer, however, the network develops a left-to-right sensitivity to upcoming symbols, with per-symbol accuracy being a function of the network confidence about the next output symbol. To assess the correlation between per-symbol accuracy and “perception” of the morphological structure, we used a Linear Mixed Effects (LME) model of how well structural features of German and Italian verb forms interpolate the “average” network accuracy in producing an upcoming symbol (1 for a hit, 0 for a miss) in both training and test. The marginal plots of Figure 2 show that there is a clear structural effect of the distance to the stem-ending boundary of the symbol currently being produced, over and above the length of the input string. Besides, stems and suffixes of regulars exhibit different accuracy slopes compared with stems and suffixes of irregulars. Intuitively, production of an inflected form by a LSTM network is fairly easy at the beginning of the stem, but it soon gets more difficult when approaching the morpheme boundary, particularly with irregulars. Accuracy reaches the minimum value on the first symbol of the inflectional ending, which marks a point of structural discontinuity in an inflected verb form. From that position, accuracy starts increasing again, showing a characteristically V-shaped trend. Clearly, this trend is more apparent with test words (Figure 2, bottom), where stems and endings are recombined in novel ways. The same results hold for German. On the other hand, no evidence of structure sensitivity was found in a LME model of the baseline output for both German and Italian.
The cell-filling problem is an ecological,
developmentally motivated task, based on evidence of
fully inflected forms. Although other (simpler)
models have been proposed to account for
formmeaning mapping in Morphology
Adele E Goldberg. 2003. Constructions: a new theoretical approach to language. Trends in cognitive sciences, 7(5):219–224.
Robert Malouf. in press. Generating morphological paradigms with a recurrent neural network. Morphology.
Appendix A. Comparative test results base form
target bleiben dï¿oerfen
sein mï¿oessen bestehen sprechen geben sehen
tun stehen fahren finden dï¿oerfen
fahren beginnen kommen liegen sehen bringen fragen gehen haben nehmen nennen sagen tragen bitten denken geben scheinen
setzen sprechen werden bliebt gedurft seiend gemusst bestandet spricht gibt siehst tatet standet fï¿oehrst fandet darf fuhrst begannt kamst lagt saht brachtet fragtet gingt hattet nahmt nanntet sagtet trï¿oegst
baten dachtest
gabst schienst setztet sprachst wurdet
CoNNL baseline
blieb gedï¿oerfen
seind gemï¿oessen bestehtet sprecht gebt sehst tut stehtet fahrst findet dï¿oerfe fahrtest beginntet kommst liechtet sehtet brinchtet frugt gehtet habt neht nenntet sugt tragst bitten denkest
gebst scheintest
setzet sprechtest werdet