Pymorphy2 is a morphological analyzer implemented in Python for Russian. The parser takes a word and, based on its morphology, produces a series of classification hypotheses regarding class, gender, number, case, etc. However, the analysis of the isolated word rarely occurs without any ambiguity. This article presents an implementation of a trigram tag model that works on top of the morphological parsing performed by pymorphy2 and uses the sequence of words in the sentence to choose the most probable morphological interpretation for each word.
Taken separately, Russian word forms often allow for more than one grammatical interpretation.2
probability. When working with bag-of-words models, a common practice is to simply select for each word form the parse with the highest probability. In this case, assuming only the dictionary form of the word is required, pymorphy2 will get the correct part-of-speech about 92% of the time. However, if the grammatical case of the word is taken in consideration, the precision will drop to 82% (see the experiment’s results in section 6) and the parsing will often produce ungrammatical sequences.
article is to implement a trigram part-of-speech model to disambiguate the morphological analysis that pymorphy2 performs on the isolated word. The part-of-speech tags of those word classes that have declensions are augmented with the grammatical case, while other features such as number or gender are discarded in order to keep the trigram model at a reasonable size and prevent data sparseness when training it on a rather small corpus.
exclusively on this morphological analyzer and measure its performance before and after the suggested extension. It does not intend to be a superior solution to other morphological analyzers available for
between the performance of pymorphy2 and other available options (mystem3, TreeTagger, FreeLing, etc.), the reader is referred to the work of Kuzmenko [4] or Kotelnikov et al. [5].
[
Parse(word='мама', tag=OpencorporaTag('NOUN,anim,femn sing,nomn'), normal_form='мама', score=1.0, methods_stack=((<DictionaryAnalyzer>, 'мама', 1907, 0),)) ] [
Parse(word='мыла', tag=OpencorporaTag('NOUN,inan,neut sing,gent'), normal_form='мыло', score=0.333333, methods_stack=((<DictionaryAnalyzer>, 'мыла', 54, 1),)),
Parse(word='мыла', tag=OpencorporaTag('VERB,impf,tran femn,sing,past,indc'), normal_form='мыть', score=0.333333, methods_stack=((<DictionaryAnalyzer>, 'мыла', 1813, 8),)),
Parse(word='мыла', tag=OpencorporaTag('NOUN,inan,neut plur,nomn'), normal_form='мыло', score=0.166666, methods_stack=((<DictionaryAnalyzer>, 'мыла', 54, 6),)),
Parse(word='мыла', tag=OpencorporaTag('NOUN,inan,neut plur,accs'), normal_form='мыло', score=0.166666, methods_stack=((<DictionaryAnalyzer>, 'мыла', 54, 9),)) ] [
Parse(word='раму', tag=OpencorporaTag('NOUN,inan,masc,Geox sing,datv'), normal_form='рам', score=0.5, methods_stack=((<DictionaryAnalyzer>, 'раму', 32, 2),)),
Parse(word='раму', tag=OpencorporaTag('NOUN,inan,femn sing,accs'), normal_form='рама', score=0.5,
methods_stack=((<DictionaryAnalyzer>, 'раму', 55, 3),))
]
(
the case of the noun "раму" the ambiguity arises in gender and case, while "мыла" can be analysed as
2 For example, the nominal and accusative plural cases endings for nouns like "сталь" (declension type 8a according to A. A. Zaliznyak's
classification) are identical for the singular cases of the genitive, dative and locative; most of the singular feminine adjectives in the genitive,
dative, locative and instrumental cases share the same inflection; a word form can even belong to different word classes, like "мыла", that can
be analyzed as a noun or a verb.
a verb or noun. Every parse object inside the list has a parameter "score" with an associated probability.
Korobov [3] states that the score corresponds to the conditional probability p (analysis | word) estimated
on the basis of the OpenCorpora corpus. This is obtained by counting how many times a certain analysis
has been associated with a given word form and, based on these frequencies, its conditional probability
is calculated using Laplace smoothing. The parse objects within the list are sorted according to this
probability in descending order, therefore picking the first item in the list is equivalent to selecting the
parse object with the most probable interpretation for the given word form. For example, the parse
objects with the highest score for each of the word forms analysed in (
Parse(word='мама', tag=OpencorporaTag('NOUN,anim,femn sing,nomn'), normal_form='мама', score=1.0, methods_stack=((<DictionaryAnalyzer>, 'мама', 1907, 0),))
Parse(word='мыла', tag=OpencorporaTag('NOUN,inan,neut sing,gent'), normal_form='мыло', score=0.333333, methods_stack=((<DictionaryAnalyzer>, 'мыла', 54, 1),))
Parse(word='раму', tag=OpencorporaTag('NOUN,inan,masc,Geox sing,datv'), normal_form='рам', score=0.5,
methods_stack=((<DictionaryAnalyzer>, 'раму', 32, 2),))
(
If "мама", "мыла", "раму" are no longer treated as separate tokens, but as words in the sentence "мама мыла раму", the parse object with the highest score incorrectly classifies the last two. When working with bag-of-words models and lemmas, the misclassification in case is usually not harmful (most of the time it will still provide the right dictionary form)3, but a wrong part-of-speech attribution will produce a different interpretation of the lemma. The next section suggests how the score values can be combined with the trigram tag model to obtain better results.
NLP tasks like part-of-speech tagging and word class disambiguation. The task of the model is to find for any string of word forms of the set Ψ (the observable states) the most probable sequence of part-ofspeech tags of the set Ω (the hidden states). For a better understanding of HMM the reader is referred to Jurafsky [6] or Bocharov et al. [7]. Although, nowadays POS-taggers are build using more advanced techniques, for example those based on neural networks, for the purpose envisaged here of eliminating the ambiguity of the analysis previously carried out by pymorphy2, a modified HMM model would be an easy solution to implement. The HMM is briefly described below along with the intended modification to disambiguate the analysis from pymorphy2.
∨ (
∨ , (
The approach taken here is to implement the tagging algorithm on top of the pymorphy2 parsing
results and use the score values from the Parse object as the emission probabilities.
∨
∨
,
,
∨
∨
(
Pymorphy2 not only implements the OpenCorpora dictionary, it also adopts the same set of tags. To take advantage of this fact, the model was trained on the OpenCorpora labeled subcorpus with homonyms removed [8] (26011 sentences, 256311 tokens, 188319 words), so no modifications on the tags were required. As mentioned in the previous section, the emission probabilities are directly replaced by the score values from the parse objects. Therefore, only the transition probabilities in the corpus were calculated on the basis of the following 49 tags, which were obtained by combining the 20 basic part-of-speech tags from pymorphy2/OpenCorpora and the grammatical case (where applicable):
As table 2 shows, those part-of-speech that have declensions, were augmented with the grammatical case. Gender and number were not taken into account to keep the tag set within reasonable limits. It was assumed that case is a good predictor to be included in the transition probabilities (although this hypothesis remains to be proven). Table 4 presents a fragment of the resulting matrix with the transition probabilities. The cells contain the conditional probability for the tag in the row when it is preceded by the two tags from the columns. For example, the probability that an adjective in the nominative case appears at the very beginning of the sentence is 0.0731339900. Those combinations that were not observed in the corpus were assigned a very small probability of 0.000000001; for instance an adjective in any other case than is not instrumental after an adjective in the instrumental case at the beginning of the sentence. The transition probabilities were stored in a JSON-format file.
The code written in python6 implements the Viterbi algorithm to find the most probable sequence of parse objects from the morphological analysis performed by pymorphy2. It takes as input the JSON file with the transition probabilities and a list that contains all the possible parse objects that pymorphy2 returns for each word. The output is a new list with only one parse object per word. The code and the file with the transition probabilities are available for download from a GitHub repository [9]. 4 Symbol for “start of sentence”. 5 Symbol for “end of sentence”. 6 System requirements: python3, pymorphy2 version 0.8.
Code implementation example from pymorphy2 import MorphAnalyzer #1 from hmmtrigram import MostProbableTagSequence #2 morph = MorphAnalyzer() #3 token_list = ['Мама', 'мыла', 'Раму', '.'] #4 pymorphy2_parsed = [morph.parse(token) for token in token_list] #5 mpts = MostProbableTagSequence('transition_probabilities.json') #6 mpts.get_sequence(pymorphy2_parsed) #7 #1: imports the class for morphological analysis from the package “pymorphy2” #2: imports the class for calculating the most probable tag sequence from “hmmtrigram” #3: instantiates the object “morph” from the class “MorphAnalyzer” #4: any list of tokens #5: the “parse” method of the “morph” object parses each token in the list and stores the parsing results in the new list “pymorphy2_parsed”
#6: instantiates the object “mpts” from the class “MostProbableTagSequence” with the name of the json file that contains the transition probabilities as argument
#7: the “get_sequence” method of the “mpts” object takes the list with the parse objects stored in “pymorphy2_parsed” and returns a new list with the most probable parsing for the sequence of tokens.
[ Parse(word='мама', tag=OpencorporaTag('NOUN,anim,femn sing,nomn'), normal_form='мама', score=1.0, methods_stack=((<DictionaryAnalyzer>, 'мама', 1907, 0),)), Parse(word='мыла', tag=OpencorporaTag('VERB,impf,tran femn,sing,past,indc'), normal_form='мыть', score=0.333333, methods_stack=((<DictionaryAnalyzer>, 'мыла', 1813, 8),)), Parse(word='раму', tag=OpencorporaTag('NOUN,inan,femn sing,accs'), normal_form='рама', score=0.5, methods_stack=((<DictionaryAnalyzer>, 'раму', 55, 3),)), Parse(word='.', tag=OpencorporaTag('PNCT'), normal_form='.', score=1.0, methods_stack=((<PunctuationAnalyzer>, '.'),)) ] (11) (12)
Table 5 shows the averaged precision and recall for the 20 part-of-speech tags (the base POS tags without case). This outlines the performance that can be expected of getting the right part-of-speech (POS) tag for a given word form. When grammatical case is also taken into account (augmented tags), these values drop significantly (compare with the baseline in table 6).
The testing results support the benefit of applying the trigram model to disambiguate the analysis preformed by pymorphy2. However, if pymorphy2 is being used only to get the part-of-speech or the dictionary form of the words within a bag-of-words model, the suggested extension will produce little improvement over the baseline. For that purpose, selecting the first parse object in the list will be enough. The implementation of the trigram model to choose the most probable sequence of parse objects is useful for those tasks that require valid POS tag sequences or greater precision in determining the grammatical case of a word. In Russian, by disambiguating the grammatical case of a word form, most of the time its gender and number are also obtained correctly. The effects of broadening the combination of part of speech + case with respect to number and / or gender remain to be explored.