We present an on-going experiment aimed at improving the results of Russian PoS tagging by means of increasing the size of morphological lexicon that is used for training the respective tagger(s). The frequency list of out-of-vocabulary (OOV) word forms along with the tags and lemmas assigned by the guesser is manually checked, corrected and classified by students in the framework of assignments, so that valid lexical items candidates for inclusion into the morphological lexicon could be identified. We expect to improve the lexicon coverage by the most frequent proper names and foreign words, as well as to create an auxiliary lexicon containing the most frequent typos.
Identification the language of the downloaded texts, removing the “incorrect” documents Segmenting the text into paragraphs and sentences Removing duplicate contents (identical or partially identical text segments) Tokenization—segmenting the text into words Linguistic (morphological, and possibly also syntactic) annotation—lemmatization and PoS Uploading the resulting corpus into the corpus manager (i.e., generating the respective index structures) that will make the corpus accessible for the users.
Our paper is devoted to the morphological tagging of input texts and, narrower, to the processing of out-of-vocabulary (OOV) tokens.
From the very beginning of the Aranea Project, only tools with an open-source or free license have
been used for all processing. As there are many languages to be processed, for morphosyntactic
annotation tools with many language models were preferred. This was especially the case of
TreeTagger, that was easy to integrate into our processing pipeline [
Despite being a rather old tool, TreeTagger is still being used by many projects. Its main advantage, from our perspective, is the processing speed than can be even by the order of magnitude faster than other tools for the same language. There are, however, some disadvantages as well. The quality of language models provided by its author varies from one language to another, depending on the training data and morphological lexicon available. Perhaps the greatest deficiency of TreeTagger is absence of any procedure that would guess lemmas for out-of-vocabulary (OOV) lexical items – those are simply tagged as “unknown”, leaving decision of further processing to the user.
Several Russian language models for TreeTaggers are available, for our work we rely on that provided by Serge Sharoff. Though its coverage, compared to other languages, is fairly high, the morphological lexicon based on Zaliznyak dictionary and manually disambiguated subcorpus of the Russian National Corpus naturally cannot cover all lexical items appearing in a “fresh” web-crawled corpus.
Russian belongs to languages with several taggers available, so an idea of looking for an alternative
is quite straightforward. None of them, however, can be simply declared as “better” – each of them has
some drawbacks as well. In our experiments we were using UDPipe [
The third tool included in our work was CSTlemma [
In an attempt to improve the results of annotation, we are planning to apply steps as follows. 1 Use the “ensemble tagging” approach, i.e., annotate the corpus by several different tools. 2 Aggregate the results by means of manually written rules. 3 Manually disambiguate the annotations for most frequent OOV lexical items 4 Use the disambiguated list to amend the morphological lexicon for next step of annotation.
The current paper describes the very first phase of or experiment targeted on correcting the lemma and PoS tags by means of crowdsourcing.
“Crowdsourcing" is a relatively recent concept that encompasses many practices. This diversity
leads to the blurring of the limits of crowdsourcing that may be identified virtually with any type of
Internet-based collaborative activity, such as co-creation or user innovation [
From this perspective, language data annotation performed by students in the framework of the endof-term assignments can well be considered “crowdsourcing”, even if only some of the above characteristics apply. It is also worth noting that, according to our experience, students appreciate the feeling that their work may be useful not only as a tool for classification.
The OOV lexical items observed in our corpora are of different nature. Besides the “true neologisms”, i.e., words qualifying for inclusion even into the traditional dictionary, proper nouns (such as personal and geographical names) and their derivates, we can find also items traditionally not considered as “words” – various abbreviations, acronyms and symbols, URLs or e-mail addresses, parts of foreign language quotations and – above all – all sorts of “typos” and “errors”. Inflected word forms apply to almost all previously mentioned categories, which makes the whole picture even more complex.
In the following text we present an experiment aimed at amending the morphological lexicon used for training the language model(s) by a manually validated list of most frequent OOV items derived from an annotated web corpus. The annotation is to be performed by graduate students of the Mathematical Linguistics Departmernt of the Saint-Petersburg University in the framework of end-ofterm assignment for the “Corpora in NLP” subject.
Having only limited “human power” (14 students in total) at hand, we decided to follow the threefold setup (i.e., each item to be annotated by three independent annotators) and make the task as simple as possible. This is why the annotators were not expected to check all the morphological categories provided by the respective tags, and they were asked to decide only on two parameters - lemma and word class (part of speech).
In the first step, we used data from and the Aranea TreeTagger pipelines, and subsequently merged into a single vertical file. Then, we converted the original MTE morphological tags to “PoS- only” tags and produced a frequency list of all lexical items indicated as OOV by both taggers. After deleting the unused parameters, the resulting lists contained the frequency, word form, lemma assigned by the CSTLemma and UDPipe taggers and PoS information derived from the tag assigned by TreeTagger (aTag, using the AUT notation). This decision has been motivated by an observation that TreeTagger is typically more successful in assigning morphological categories for unknown words than others.
As we naturally could expect to be able to process only the rather small part of the list, after some experimenting with various thresholds, we decided to pass into annotation only the most frequent items. This meant that each annotator would process approximately 1000 items.
The example of the source data (in alphabetical order, after applying the frequency cut-off) is shown in Table 1.
Word Росстата Ростех Ростехнадзора Ростова-наДону Ростове-наДону Ростове-наДону Ростов-наДону ротовую роуминг роуминга роутер РПЛ РСА РСО-Алания рубцов руд руду рулонных румяной РУСАДА РусГидро Руссель ручках ручном
aTag (TreeTagger)
Nn Nn Nn Nn Nn Nn Nn Aj Nn Nn Nn Zz Nn Nn Nn Nn Nn Aj Aj Nn Nn Nn Nn Aj uLemma (UDPipe) Росстат Ростех Ростехнадзор Ростова-наДон Ростове-наДон ростов-надону Ростов-наДон ротовый роуминг роуминг роутер РПЛ РСА РСО-Алания рубец руда руда рулонный румяный РУСАД РусГидро Руссель ручка ручной
uAtag (UDPipe)
Nn Dt Nn Nn Nn Nn Nn Aj Nn Nn Nn Nn Nn Nn Nn Nn Nn Aj Aj Nn Nn Nn Nn Aj
CLemma (CSTLemma) Росстат Ростеха Ростехнадзор ростова-надон ростове-надон ростове-надон ростов-на-дон ротовый роуминг роуминг роутер рплый РС РСО-Алания рубец руд руда рулонный румяный РУСАДА русгидро Руссель ручка ручный
We can observe several phenomena here. While most PoS categories are classified correctly, abbreviation are mostly tagged as “nouns”, but also as “determiners”, or even “punctuation”, and lemma form as well as its capitalization is sometimes guessed correctly, while sometimes not. The result of simple aggregation of the same data can be seen in Table 2.
The overall task for the annotators was to produce correct data for all lines in the table. To minimize the number of necessary keystrokes and to keep track of the changes, the data have been further modified to contain two newly added columns – Lemmb used as a template for correcting the value for Lemma (it is expected that most modifications will occur at the end of the respective string only) and bTag (to be filled only in case of wrong PoS assignment).
As has been already mentioned, each item (line of the table) has to be annotated by three independent annotators. We decided, however, not to split the data in a straightforward way, but to assign each alphabetical segment of the data to three annotators using a rule as follows: each group of four lines will be split into four tuples containing three lines with one missing line form the original group. Moreover, the whole lot of data has been split to three parts, so that each annotator could get three different sections of the alphabet in his or her data.
By applying this fairly “sophisticated” assignment scheme, we expected to improve the overall uniformity and quality of the output, as well as to prevent “collaboration” among students, as no two assigned lots were identical.
An excerpt of the data from Table 3 assigned to a single annotator is shown in Table 3. bTag aTag Nn Nn Aj Nn
Nn Zz|Nn
Nn Nn Nn Aj Aj Nn Nn Nn Aj
Note that the “missing” every third Id results from the assignment scheme.
The split data has been uploaded as excel spreadsheets to a shared Google disk and assigned randomly to the respective annotators. The task has been assigned in the middle of the semester, after the students already got acquainted with the basic concepts of corpus morphosyntactic annotation and acquired the elementary querying skills. The instructions for annotating the data as they are presented in Table 3 were as follows.
A Only Lemmb and bTag columns may be modified.
B If both Lemma and aTag values are correct, nothing has to be done.
C If aTag value is wrong, the correct value should be inserted in bTag.
D If Lemma value is wrong, it should be corrected in Lemmb.
E If the word form is obvious typo (missing or superfluous letter, exchanged letters), or the word does not contain the necessary diacritics, the correct lemma marked by an asterisk should entered in Lemmb.
F If the correct word form cannot be reconstructed by simple editing operations, i.e., cannot be recognized (e.g., part of the word as a result of hyphenation), the value of bTag will be “Er” (error).
G If the word form is obvious foreign word, the value of bTag will be “Yx”.
H It is not necessary to evaluate whether the word form is “literary” - words of “lower” registers (such as slang) also have “correct” lemmas.
The annotators were also instructed to check all “non-obvious” items by querying the corpus and analyzing the respective contexts. The initial training was performed during one teaching lesson in a computer lab, so that possibly all frequent problems could be explained.
Obviously, recommendations for reannotation of OOV word forms should be not so much technical as linguistic. They should analyze not only typical obvious cases, but also problems. In this case, we should proceed from the following: - Russian grammar rules - considering tokenization rules when annotating a corpus - contexts of using this token in the corpus - frequency data on certain uses.
The development and description of such an instruction is a matter of the future and a topic for a separate article, here we give some problems that cause difficulties during annotation.
1. Foreign words should be processed depending on the context: if they appear as part of a foreign language expression (quotation), then the correct PoS tag is "Yx". However, if they are part of a Russian-language phrase, for example, in the meaning of a noun (usually these are proper nouns), then it is reasonable to mark them with the tag "Nn" (or other relevant part of speech). 2. Many languages and taggers have problems with abbreviations. We can say that they are difficult for grammar as such. Usually, abbreviations include words written in capital letters, however, there are many other options. For example. The abbreviation ВУЗ (high school) is widely spelled in small letters (вуз), which has actually become a noun. There are a large number of standardized and nonstandardized abbreviations such as д-р, (doctor), изд-во (Publishing House) etc. Abbreviations like МХАТ, НИИМаш are often inflected as nouns and in fact, they are, without losing the spelling in large letters. Some abbreviature have several standard lemmas (spellings), eg, КзоТ, КЗОТ. (Labor Code, Labor Code).
3. It is desirable to include proper names in the morphological dictionary. The question arises, all or quite often used? So, if the dictionary contains the masculine name Давид , should the feminine name Давида be included with a different declension paradigm? Almost any adjective or common noun can occur as a surname. Should they be presented in the morphological dictionary as separate lexemes? Those common nouns in indirect cases, apparently, need to match two lemmas, for example, the word form Котов (surname) will receive a tag of a noun and two lemmas Котов|кот, or Юнг lemmas Юнг|юнга, the word form in genetivs case Серебряной (surname) gets lemmas Серебряная|серебряный and tags Nn | Aj.
4. The participles in MTE are carried out to the verb lemma (выделенный — выделить), but there are many cases when, along with the verbal lemma, an adjective lemma must also be indicated: добавленная стоимость - добавить|добавленный Vb|Aj.
The source data consisted of 5,040 producing 15,120 lines to annotate by 15 students. I.e., each of them had to process 1,008 lines. As only 14 files have been returned, the missing file has been reassigned to a student from a different group.
The resulting data has not been processed completely yet, but the first analysis looks promising – see table 4 and 5.
Source lines Triple Agreement both on Lemma and PoS Double Agreement both on Lemma and PoS Triple Agreement on Lemma Double Agreement on Lemma
There were several goals to be achieved by the annotation. Firstly, we would like to produce a validated list of most frequent neologisms to be included in the morphological lexicon; in this stage, we even do not expect to generate full paradigms for those lexical items. Secondly, we wanted to get the list of the most frequent typos and other types of errors that could also be used as a supplement to that lexicon, but also as source data for a future system for data normalization. And lastly, we also wanted to obtain a list of most frequent foreign lexical items appearing in Russian corpus data.
Although the detailed analysis of the annotated data is yet to be performed, some conclusions can be seen already. They can be summarized as follows:
(1) The Annotation Guidelines must be as precise as possible, showing not only the typical problems and their solutions, but also the seemingly “easy” cases. One-page instruction, as it was in our case, is definitely not sufficient.
(2) The most common errors were associated with the treatment of proper nouns. An automatic procedure based on frequencies of lower/uppercased word forms would most likely perform better.
(3) The other common issue was the proper form of lemma for adjectives (it should be masculine and nominative singular). As the morphology of the Russian adjectives is fairly regular, a procedure to fix it automatically would be feasible.
(4) One of the fairy frequent PoS ambiguity in our data was the “Nn”/“Yx” (noun/foreign) case. The manually annotated data, however, show that the real number of “foreign” is rather low, yet it introduces a lot of noise into the annotation process. It would therefore be reasonable to substitute all tags for “foreign” with that of “nouns” in the future annotation.
In the near future, besides the new round of a similar annotation effort with an improved setup, we would like to combine its results with those obtained in the framework of the ensemble tagging experiment. 10.Acknowledgment
Authors wish to express their sincere gratitude to the 1st year Master program students of the SPbU Mathematical Linguistics Department for their valuable help in annotating the data.