Morphemic structure of words is useful for various NLP problems, in particular, for deriving a meaning of unknown words in languages with rich morphology, such as Russian. For Russian, several machine learning models for automatic morpheme segmentation of words were built, but only for parsing their lemmas. Meanwhile, signi cantly varying word forms are present in texts, among them unknown words are often encountered, and their lemmas are unknown. The paper reports on experiments for comparing two ways to automatically segment Russian word forms, both ways involve splitting into morphs and classi cation of resulted morphs. The former is based on a neural model trained on a data set automatically augmented with segmented word forms, the latter produces segmentation through predicted lemma and a pre-trained neural morpheme segmentation model for lemmas. It was shown that the models have comparable quality in morpheme segmentation with classi cation, and the model based on the augmented dataset slightly outperforms in word-level classi cation accuracy.
Morpheme segmentation as a kind of morphological analysis implies splitting words into constituent morphs, which are the surface forms of morphemes (roots and a xes), for example: без-вкус-н-ый, taste-less. Though the task of automatic morpheme segmentation was studied in early years of natural language processing (NLP), signi cant progress in its solution has appeared in recent years, when various machine learning techniques began to be applied.
Since morphemes are the smallest meaningful language units, information
about morphemic structure of words is already in use in various NLP
applications and auxiliary tasks, including machine translation [
Morpheme segmentation is especially topical and at the same time more difcult for languages with rich morphologies (such as Russian or Finnish). For morphologically rich languages with many a xes of various types and meanings, a more complicated task is relevant, which involves besides segmentation classi cation of segmented morphs. The main types of morphemes are Pre x, Root, Su x, Ending, for example: без:PREF/вкус:ROOT/н:SUFF/ый:END, taste:ROOT/less:SUFF.
The rst works on morpheme segmentation were pure statistical and
dictionarybased [
The task of morpheme segmentation with classi cation of segmented morphs
remained almost unexplored until recent works [
The implemented methods consider the task of morpheme segmentation with
classi cation as sequence labeling [
However, these SOTA models for Russian were developed only for morpheme segmentation of lemmas (normalized forms of words), so far as only lemmas are present in the existing labeled datasets. Meanwhile, for morphologically rich and highly-in ecting Russian language, signi cantly varying word forms are present in texts, in particular, for verb успеть (to be in time) more than 15 its forms may be used: успеют, успел, успели and so on. Among various word forms, unknown ones are often encountered, and their lemmas are unknown. Since it turned out that the developed SOTA models work poorly for word forms, giving
4 https://github.com/alesapin/GBDTMorphParsing 5 https://github.com/alesapin/RussianMorphParsing only about 30% for classi cation accuracy, we aimed to research segmentation methods applicable for word forms.
In the paper we describe and experimentally compare two ways to
automatically segment Russian word forms, both ways involve splitting into morphs and
classi cation of resulted morphs. The former is based on a neural model trained
on a dataset automatically augmented with segmented and labeled word forms,
the latter produces segmentation through predicted lemma and a pre-trained
segmentation model for lemmas. It is unclear a priory, which of the ways is
preferable, and to evaluate them, we have chosen CNN model as a core for both
ways and have exploited an available dataset containing about 90,000 segmented
words (lemmas) from Tikhonov's dictionary [
Experimental evaluation has shown that the model trained on the augmented dataset (hereafter, model on word forms) and the model trained on lemmas and supplemented by the rules for segmenting the word form based on its segmented lemma (hereafter, hybrid model) have comparable quality in morpheme segmentation with classi cation (as well as comparable with quality of SOTA methods), while the model on word forms slightly wins in word-level classi cation accuracy with score 88%.
The paper starts with an overview of the main works on the morpheme segmentation, followed by explanation of our augmentation procedure and the resulted augmented dataset. Then our CNN model architecture and key issues of training the model on word forms are described, and the results of experiments with the compared models are reported and discussed. Finally, we present some conclusion. 2
The earliest method of morpheme segmentation was proposed by Z. Harris in [
In the next years, the most known solutions for morpheme segmentation were
implemented in Morfessor system [
Another kind of semi-supervised machine learning for morpheme
segmentation [
A pure supervised method with signi cantly better quality for the twofold
task of morpheme segmentation with classi cation was proposed in [
Two more supervised machine learning models for morpheme segmentation
with classi cation were developed for Russian words in recent works [
Evaluation of these CNN, GBDT and Bi-LSTM models trained on the same
Russian datasets has showed their comparable quality, about 98{99% of
Fmeasure on morpheme boundaries and 96{98% of classi cation accuracy for
letters and about 87{89% on words [
In order to build a dataset augmented with segmented word forms and thus
suitable for training, we have developed a procedure that produces necessary
segmentation of word forms based on known segmentation of the corresponding
lemmas along with grammatical information about Russian word formation
sufxes and about speci c features of Russian in ection for words of various part
of speech [
The dataset6 based on Tikhonov's dictionary was the source of segmented and
labeled lemmas, and various word forms for a particular lemma were taken from
Open Corpora dictionary7 [
While applying our augmentation procedure, all lemmas from Tikhonov's dataset were considered and their corresponding word forms from OpenCorpora were processed, but those dataset elements that are absent in Open Corpora dictionary were discarded (approximately, 5 thous. words, the most of them are very rare, such as гофмейстерский, яспис, спассеровать).
For a given word form to be segmented and its segmented lemma, the procedure applies segmenting rules depending on the part of speech and its subclass. For most nouns, adjective, and participles the rules are quite simple: in the general case, the given word form and lemma have some common beginning, and if the rest part of the lemma is labeled as ending, the rest part of the word form is also annotated as ending, whereas its common part copies segmentation and labels of the lemma. The following word pair illustrates the rule: Lemma: разрумяненный раз:PREF/румян:ROOT/енн:SUFF/ый:END Word form: разрумяненному раз:PREF/румян:ROOT/енн:SUFF/ому:END
However, for some subclasses of nouns and adjectives (words with a nal yota: ковбой { ковбоя, соболий { собольего), short adjectives (послушный { послушен) nouns (words with fugitive vowels: день { дня, замочек { замочка), as well as for verbs, more di cult segmenting rules were elaborated.
Speci cally, to segment personal verbal forms and gerund (e.g., увидевши { у:PREF/вид :ROOT/е:SUFF/вши:SUFF), after detection of the common part with in nitive form, the segmenting rules sequentially try to recognize and to label word-formative su xes (ова, ева, ыва, ива, вши, ев, ен, в, л, and so on) and post x (ся, сь) in the mismatching part of the given word form, and its rest part (if any) is classi ed as ending. Here is an example:
Word form выходила вы:PREF/ход :ROOT/и:SUFF/л:SUFF/а:END In such a way, our augmentation procedure has processed about 92% of wordforms. Some rare di cult cases were discarded, in particular, consonant alternation, but such discarding does not impact on the result of comparing. The resulted dataset augmented with segmented and classi ed word forms has a total size of 1,130,359 elements: 34% nouns, 32.35% adjectives and participles, 33.56% verbal forms, and 0.07% words of other POS. 6 https://github.com/AlexeySorokin/NeuralMorphemeSegmentation/tree/master/data 7 http://opencorpora.org
The augmented dataset consists of in ectional paradigms for the processed lemmas (hereafter, in ectional groups), each group encompasses word forms for a particular lemma. Groups for nouns and adjectives are relatively small, while for verbs, a group includes all forms of present, future, and past tense, gerund forms, up to 31 elements. Here is a fragment of in ectional group for verb обсыпать (to strew ): обсыпать об:PREF/сып:ROOT/а:SUFF/ть:SUFF обсыпал об:PREF/сып:ROOT/а:SUFF/л:SUFF обсыпала об:PREF/сып:ROOT/а:SUFF/л:SUFF/а:END обсыпало об:PREF/сып:ROOT/а:SUFF/л:SUFF/о:END обсыпали об:PREF/сып:ROOT/а:SUFF/л:SUFF/и:END обсыплю об:PREF/сып:ROOT/л:SUFF/ю:END обсыпем об:PREF/сып:ROOT/ем:END обсыплем об:PREF/сып:ROOT/л:SUFF/ем:END обсыпешь об:PREF/сып:ROOT/ешь:END обсыплешь об:PREF/сып:ROOT/л:SUFF/ешь:END обсыпете об:PREF/сып:ROOT/ете:END 4
For our study of segmenting word forms and building morpheme segmentation
models, among three SOTA models for morpheme segmentation, namely CNN,
GBDT, and Bi-LSTM we have chosen convolutional neural network (CNN),
because CNN is training much faster than others, and at the same time does not
lose in quality. For simpli cation of experiments, in all our segmentation models
we did not use the auxiliary correction procedure proposed for the original CNN
model, as well as ensembles of several models [
All our trained CNN models for segmenting words (word forms) were
implemented with Keras library [
The model has several layers, the last layer is fully connected and completed
with a softmax activation function, which outputs a probability distribution
over all possible letter classes. The resulted classes of letters are obtained from
probability distribution with argmax function. Similar to works [
Various hyperparameters of our CNN model were experimentally tested in preliminary experiments. The resulted model has four layers with 512 lters in each layer, dropout of 40%, ReLU activation function and kernel size of 5. More lters in a layer slightly improve the quality (less than 0.5%), but the model became too heavy both for training and for evaluation. As for additional layers, they also do not signi cantly improve quality: the model with three layers gives su cient results, losing to four-layer network only about 1{2%. Among the gradient descent algorithms (Adam, RMSprop, SGD), the better results were shown by Adam. 5
For all our experiments, the data sets (original Tikhonov's dataset and the augmented one) were randomly divided in proportion 70:10:20 for training, validation, and testing, respectively; the training subset of the augmented dataset includes 791 thous. word forms. After tuning the model with random splits, for correct evaluation of the models, we have xed our training, testing and validation sets for reproducibility. All trained and evaluated models are freely available8.
In experiments with training our CNN model on the augmented dataset, two di erent variants of random dividing the dataset were studied: { Random mixing of labeled word forms and then splitting them to training and testing subsets; { Random mixing of in ectional groups (each group consists of all word forms corresponding to the same lemma); and after that splitting to training and testing subsets is performed (thus, splitting does not divide the groups).
Thereby we have obtained two trained models, namely, the model on word forms with simple mixing and the model on word forms with group mixing, the results of their evaluation are presented in Tables 1, 2. Table 1 shows quality of only segmentation measured in precision, recall, and F-measure (computed as mean harmonic of the recall and precision).
recognized classes of letters to the number of all letters, the latter estimates the ratio of completely correctly segmented words with true classes of all their letters.
For comparison, in the last lines of the Tables we have added scores of the CNN model trained only on lemmas taken from the augmented dataset (more precise, from its training subset). The scores show that this model signi cantly loses when applied to word forms: much worse F-measure on morpheme boundaries (89.52%) and even worse classi cation accuracy (81.19% and 34.30% for letters and word, respectively). At the same time, almost similar scores for lemmas con rm consistency of experimental settings. As for our models on word forms, the model with simple mixing outperforms its counterpart in all the scores (slightly on morphs boundaries and signi cantly in classi cation accuracy for words). The explanation is simple: since in ectional groups may be divided while mixing and splitting to training and testing subsets for the model with simple mixing, the testing subset can contain some word forms of the groups, whose elements are present in the training subset, and this improves evaluation results. At the same time, the quality of the model with group mixing is comparable with SOTA morpheme segmentation models built on lemmas. Therefore, it is not quite correct to compare the model with simple mixing with our hybrid model for segmenting word forms, and we have compared only the model with group mixing. 6
The hybrid model implements another way to segment word forms, which implies the following steps:
2. segmenting the latter by the model trained on lemmas (in our experiments, by the model already learned and indicated in the last lines of Tables 1, 2); 3. transforming the resulted segmented lemma into a segmented word with the aid of the procedure and segmenting rules described in section 3.
Using the model already trained on lemmas, we have evaluated the proposed hybrid model, with precision, recall and F-measure on morph boundaries (for segmentation, the scores are given in Table 3), and also accuracy both for letters and whole words (see Table 4). For comparison, in these Tables we repeat scores of the model on word forms (trained with group mixing). It is important, that CNN network of the hybrid model was trained on the lemmas taken from the training dataset for the model on word forms, and it was evaluated on the same testing set. One can notice that two our evaluated models for segmenting Russian word forms have highly close scores for morpheme segmentation, while for classi cation (Table 4), the model on word forms (group mixing) slightly wins both for letters and words. Additionally, we have evaluated ratio of various errors in morpheme segmentation, depending on wrong boundaries between morphemes of various types, the results are presented in Table 5. In both models under comparison, the most frequent errors are related with wrong boundaries between roots and sufxes, almost half of the errors (column ROOT-SUFF in Table 5). Another types of frequent errors are wrong recognition of boundary between pre x and root (PREF-ROOT) and erroneous segmentation of successive roots (ROOT-ROOT) or su xes (SUFF-SUFF). Below we present some examples of these types. In general, the presented statistics of errors are about the same, with rare errors of segmenting word endings.
{ Root and su x(ROOT-SUFF) { for verb перетлевать, the incorrectly segmented word form пере:PREF/тл:ROOT/е:SUFF/ва:SUFF/ешь:END instead of correct пере:PREF/тле:ROOT/ва:SUFF/ешь:END; { Pre x and root (PREF-ROOT) { for adjective подоблачный, erroneous под :ROOT/о:PREF/блач:ROOT/н:SUFF/ою:END instead of the correct segmentation под :PREF/облач:ROOT/н:SUFF/ою:END; { Successive roots and su xes (ROOT-ROOT, SUFF-SUFF) { for adjective трегубный, instead of correct тр:ROOT/е:LINK/губ :ROOT/н:SUFF/ого:END, wrong segmentation variant: тре:ROOT/губ :ROOT/н:SUFF/ого:END. We have developed and evaluated two models of morpheme segmentation with classi cation, which were proposed speci cally for word forms and are important for morphologically rich and highly-in ective languages, such as Russian. The rst model is purely supervised and built on the augmented dataset with labeling of constituent morphs, the second is the hybrid one combining both the supervised model based on lemmas and rules for segmenting word forms. For augmentation of existing dataset with labeled Russian lemmas we have created the rule-based procedure generating segmented word forms.
The quality of the developed models turned out to be comparable, and the model based on the augmented dataset is slightly better in word-level accuracy. This means, that both models can be used in various NLP experiments with Russian text. At the same time, the choice of the model may depend on its computational complexity important in particular applications. For some applied tasks, a three-layer CNN model instead of our four-layers CNN (as a core of the hybrid model) is more preferred, as it is faster to train and takes less memory.
Our future work implies: { To resolve some inconsistencies and errors in the original Tikhonov's dataset, which have been observed while experimenting with it, in order to increase the quality of the built models for word forms; { To elaborate additional segmenting rules for some unconsidered cases of word forms, such improvement of our augmentation procedure may be useful not only for improving the morpheme segmentation models, but also for other tasks.