Part-of-speech (PoS) tagging constitutes a common task in Natural Language Processing (NLP) given its widespread applicability. However, with the advance of new information technologies and language variation, the contents and methods for PoS-tagging have changed. The majority of Italian existing data for this task originate from standard texts, where language use is far from multifaceted informal real-life situations. Automatic PoS-tagging models trained with such data do not perform reliably on non-standard language, like social media content or language learners' texts. Our aim is to provide additional training and evaluation data from language learners tagged in Universal Dependencies (UD), as well as testing current automatic PoStagging systems and evaluating their performance on such data. We use Italian texts from a multilingual corpus of young language learners, LEONIDE, to create a tagged gold standard for evaluating UD PoS-tagging performance on nonstandard language. With the 3.7 version of Stanza, a Python NLP package, we apply available automatic PoS-taggers, namely ISDT, ParTUT, POSTWITA, TWITTIRÒ and VIT, trained with diversified data, on our dataset. Our results show that the above taggers, trained on non-standard data or multilingual treebanks, can
1
Part-of-Speech (PoS) tagging relates to the
assignment of tags or labels to the words,
punctuation marks and symbols of a text. It
constitutes a basic task in NLP, with applications
ranging from machine translation to speech
recognition and beyond. PoS-tags usually
correspond to the morphosyntactic word classes
of a given language, i.e. nouns, verbs,
conjunctions, etc. Since each language contains
specific linguistic characteristics that distinguish
itself from others, tagsets are usually language
dependent. The first automatic tool for the
assignment of PoS-tags in the Italian language
was the TreeTagger built at the University of
Stuttgart
Originally, automatic tagging methods were
mainly employed with standard texts, such as
essays, literature, and newspaper articles
Learner corpora exhibit a number of
characteristics that differentiate them from the
rest. In particular, numerous code-switching and
code-mixing phenomena are common among
them, as well as the presence of orthographical,
syntactic and/or grammatical errors
The rest of the paper is organized as follows. Section 2 provides relevant details concerning the Universal Dependencies (UD), as well as available Italian treebanks and taggers1. A brief overview about the differences in tagging standard and non-standard texts is presented in Section 3. Section 4 describes the methods and metrics commonly used for the evaluation of automatic taggers. We outline the tools and methodologies used for our experiments in Section 5 and the gold standard in Section 6. Next, in Section 7, we report the obtained results and in the subsequent section, namely 8, we discuss our findings, consider possible future works and draw our final conclusions. 1 In this paper we use this term to refer to the Stanza models trained with the different available Italian Treebanks.
Over the years, alongside the different taggers and
treebanks of each language, a new
languageindependent framework in PoS annotation has
emerged, the Universal Dependencies (UD). UD
is a cross-linguistic project with the aim of
building common annotation frameworks for
several world languages. Underlying the
Universal Dependencies annotation scheme are
universal Stanford dependencies
Among the various available treebanks and
taggers for Italian, most have been created using
exclusively standard data, such as newspaper
articles, non-fictional texts, talks and Wikipedia
pages for training the models (as in the case of
VIT, ISDT, ParTUT and PUD). However,
recently more attention has been placed on the
creation of linguistic resources for nonstandard
language, as the quantity and dissemination of this
type of content increases exponentially, so does
the need for suitable tools for its analysis and
exploitation. In this respect, PoSTwita
When it comes to evaluating the performance of a
PoS-tagger, generally an annotated gold standard
reference corpus is used. The latter requires a
distribution of the particular linguistic phenomena
that is representative of the PoS-tagger’s target
application. Additionally, since a PoS-tagger
combines several functions, like tokenization,
word/sentence segmentation, and PoS-tag
disambiguation, one of these parts must be firstly
chosen as the test object. After selecting the aspect
under analysis, it is necessary to choose which
metrics to use to compare the results. The metrics
commonly adopted for the evaluation of the tags
assigned to a linguistic corpus are accuracy,
precision, recall, F1-scores and Cohen’s K
Although various available UD taggers for
Italian exist, little is known about how these
perform on non-standard data. Some evaluations
have been done on user-generated texts in social
media
In this study, we evaluate automatic PoS-tagging
on the LEONIDE corpus
Given the inaccessibility of an evaluation sample for UD PoS-tagging on Italian learner language, we built our own pre-tokenized gold standard sample (see Section 6). Once we had our gold standard, we created a processing pipeline to test available tagging models for Italian on our data. For this, we used Stanza, a Python natural language analysis package designed using the UD formalism, as it offered easy access to a number of pre-trained models for PoS-tagging UD in Italian. The following models have been used in our evaluation: ISDT, ParTUT, POSTWITA, TWITTIRÒ and VIT. In order to evaluate only the PoS-tag disambiguation step of the PoS-taggers, regardless from other steps such as tokenization, we tagged the pre-tokenized texts using Stanza but deactivated the tokenizer (tokenize_pretokenized=True) and selected a different model as parameter each time. With the results obtained from each model, we resorted to sklearn.metrics.classification_report and sklearn.metrics.cohen_kappa_score to evaluate the total number of tags assigned to the more than 7,000 gold standard tokens according to accuracy and Cohen's K. In this way, the use of the exact same tokens and comparison metrics would have allowed an equal and meaningful comparison.
We closely focused on the accuracy and
Cohen’s K values
As the available models had been trained on different data, both in quantity and type compared to each other but also compared to our corpus, it was particularly interesting to consider how they would deal with the young language learner data at hand, but also which type of errors they would make. We thus investigate common misclassifications for taggers and human annotators, discussing possible improvements and considerations to bear in mind when using these automatic PoS-tagging systems. For the latter, we used confusion matrices, so that we could check the types of errors made, and which were the most correctly assigned tags out of the total. 6
For the creation of our gold standard, we used a
subset of the Longitudinal lEarner cOrpus iN
Italian, Deutsch, English (LEONIDE)
Despite the generally good agreement between the annotators, some difficulties emerged. These mainly concerned cases of German codeswitching, particles, clitic pronouns and auxiliary verbs (see Discussion), and occasionally orthographical or overgeneralization errors (ex. Da grande facherò [X/VERB] il calciatore). For the gold standard these issues were unanimously resolved in accordance with the Italian UD guidelines4. 7
Table 1 displays the obtained results in terms of tagging models’ accuracy and Cohen’s K, this time comparing the gold standard and the taggers’ assigned tags, along with the accuracy scores reported in Stanza for the CoNLL 2018 Shared Task5 on UD v2.5 Treebanks evaluation.
Tagger Combined TWITTIRÒ ParTUT PoSTWITA ISDT VIT
Training
data (in tokens) Pre-trained 28,387 (ironic tweets) Multilingu al parallel treebank 119,238 (Tweets) 278,429 (articles, newspapers , legal texts, Wikipedia) 272,000 (news, bureaucrac y, finance, science, literature texts)
Accuracy (Stanza)
The highest accuracy on our gold standard for learner data has been achieved by the combination of models chosen by Stanza per default. We 5https://universaldependencies.org/conll18/evaluation. html would have expected better results from ISDT, considering the high accuracy values on the standard data used to train it, and PoSTWITA for non-standard texts. However, regardless of this, in respect to our gold standard, the best models for accuracy value and Cohen's K are TWITTIRÒ and ParTUT. These latter performed well despite the fact that their tagsets did not contain all the tags used in our gold standard. In fact, both TWITTIRÒ and ParTUT, as well as PoSTWITA, did not include the PART tag (contrary to the other treebanks such as VIT and ISDT) , and thus did not assign it to particles. However, our human annotators referred to this tag to mark the common use of pronominal, reflexive and adverbial particles, such as 'mi' and ‘si’ in the corpus (ex. Più lingue ci sono; Si deve studiare molto). Furthermore, the parTUT treebank also lacked the tag for interjections, INTJ, as opposed to other treebanks that did make use of this category. Nevertheless, the training data for TWITTIRÒ was the treebank provided in Stanza that was closest to our data in type. It was created using data from social networks, therefore far from the scientific, nonfictional, or journalistic canon. On the other hand, ParTUT had been designed using standard texts but in Italian, English, and French in parallel. 8
The results show that the performance of the models was significantly influenced by the particular type of data in our gold standard corpus, which presented incorrect orthographical or morphological tokens, but also contained numerous foreign words and abnormally disposed parts-of-speech within the sentence.
In fact, when inspecting the tags incorrectly assigned by the different models with confusion matrices (see Figure 1, 2 and 3 below), we noticed that: ●
The foreign or misspelled words, which according to the UD rules had to be assigned the X tag, proved to be those with the highest number of errors. In fact, they were often confused with proper nouns, PROPNs, especially in the case of codeswitching with the German language, where nouns are spelled with initial capital letters (ex. Dopo la scuola media voglio fare la Hotelfachschule [PROPN-X]). This ● ● was particularly evident with the ParTUT model that did not assign the X tag at all (see Figure 3); The second most incorrectly tagged words were particles, PART, which are not included in the tagsets of all models although they could have been assigned to pronominal, reflexive and adverbial particles (see section 7). Instead, these words were usually assigned the PRON tag for pronouns (ex. Si [PRON-PART] deve parlare questa lingua); The third most inaccurate group of tagged words was that of interjections, INTJ, which were also not included in all treebanks. These were often confused with particles or foreign words, PART or X (ex. Ehm [X-INTJ] ciao! fece Alessandra) as it is visible from Figure 2 in the case of the TWITTIRÒ model.
On the other hand, regarding discrepancies between the tags assigned by the human annotators, we found that: ● ●
The groups on which there was most disagreement between the two annotators concerned particles, PART, and auxiliary verbs, AUX6 (ex. Le strategie che funzionano peggio sono [AUX-VERB] studiare con il computer). Concerning the first, the models did not always include the PART tag in their employed tagsets. Auxiliary verbs, additionally, were also at times abnormally positioned within the sentence and were often automatically annotated incorrectly.
Foreign words were often not annotated according to the X tag, probably because the annotators also had knowledge of the German language and therefore tended to assign the corresponding tag in the other language (ex. Faccio la Landesberufschule [NOUN-X]).
We can therefore argue that there were errors
common to both automatic models and
annotators, although the reasons for the errors
were evidently different.
6 This might be due to the fact that annotators could be
influenced by the presence next to each token of a tag
automatically assigned by Stanza, that had performed
the tokenization of the texts.
Although all taggers managed to execute the task
of automatically PoS-tagging pre-tokenized
Italian non-standard language with an accuracy of
at least 75% (with the combined model offered by
Stanza showing the best performance with 95%
accuracy and 0.94 Cohen’s K), there were
differences in the performance shown by the
individual models. The best performing two
individual models were TWITTIRÒ (86%) and
ParTUT (84%), while ISDT and PoSTWITA, that
performed better in other evaluation tasks
Improvements could be made in the future regarding the adaptation of the models to the particular type of data used here. They could be, indeed, re-trained again in case a complete Treebank with Italian non-standard data becomes available. In addition, further attempts could be made to adapt or add the missing tags to the tagsets of all models so as not to have results biased by the lack of matching tags. Finally, as far as the annotators are concerned, they could be provided with the automatically pre-tokenized texts from the models, but in order to avoid preassigned tags influencing their annotation process, it would be preferable to omit these. Thus, human annotators would only get the taggers’ tokenized text versions, so that the same tokens will be available for everyone, while the assignment of PoS would be completely up to them. Artstein, Ron and Poesio, Massimo (2008). InterCoder Agreement for Computational Linguistics.
Association for Computational Linguistics.
Attardi, Giuseppe and Simi, Maria (2009). Overview of the EVALITA 2009 Part-of-Speech tagging task. Poster and Workshop proceedings of the 11th Conference of the Italian Association for Artificial Intelligence, 12 December 2009, Reggio Emilia.