In this paper we describe the preliminary work on a novel treebank which includes texts written by learners of Italian drawn from the VALICO corpus. Data processing mostly involved the application of Universal Dependencies formalism and error annotation. First, we parsed the texts on UDPipe trained on the existent Italian UD treebanks, then we manually corrected them. The particular focus of this paper is on a one-hundred-sentence sample of the collection, used as a case study to define an annotation scheme for identifying the linguistic phenomena characterizing learners' interlanguage.
The increasing interest in Learner Corpora
(henceforth LC) is twofold motivated. On the one hand,
LC are an especially valuable source of
knowledge for interlanguage varieties. They allow
indepth comparisons of non-native varieties,
helping to elucidate the properties of the
interlanguage developed by learners with different mother
tongues and learning levels. For this reason, LC
are important resources enabling data-driven
studies exploited within several research areas, such
as Second Language Acquisition, Foreign
Language Teaching, Contrastive Interlanguage
Analysis, Computer-aided Error Analysis,
ComputerAssisted Language Learning and L2
Lexicography (e.g.
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
and Paquot, 2015; Malmasi, 2016),
GrammaticalError Detection and Correction
In this paper we describe the development of a
novel learner Italian treebank, i.e. VALICO-UD,
in which Universal Dependencies (UD)
formalism is tied to error annotation. The considerations
of the annotation process, carried out on a set of
one hundred sentences selected from a subcorpus
of VALICO1 (see Table 1)
This paper is organized as follows: in Section 2 we provide an overview of LC, focusing on Italian resources in particular; in Section 3 we present the data and the error annotation of VALICO-UD; in Section 4 we offer some examples of how we applied literal annotation to the learner sentences (LS) and, finally, in Section 6 we present conclusion and future work. 2
LC, also called interlanguage or L2 corpora, are
collections of data produced by foreign or
second language learners
LC can be enriched with Part of Speech (PoS)
tagging, syntactic, semantic, discourse structure
and error-tagging (with explicit or implicit target
hypotheses2) annotation
2A reconstructed LS on which error identification is based
Among the 14 learner Italian corpora registered
in the Learner Corpora around the World list3,
the majority are in the form of plain texts, or they
only annotate PoS (COLI, LOCCLI and CAIL24,
and VALICO), while only MERLIN
Although MERLIN contains 816 texts written
in non-native Italian
The scholars involved in the annotation of the
ESL and CFL treebanks decided to follow a
wellestablished line of work, for which learner
language analysis is centered upon morpho-syntactic
surface evidence. This is motivated by various
studies, e.g.
The data of VALICO-UD are drawn from the
VALICO corpus
4COLI, LOCCLI and CAIL2 are developed at Universita` per Stranieri di Perugia and coordinated by Stefania Spina. collection of non-native Italian texts elicited by comic strips proposed to the learners. It consists of a selection of narrative and descriptive texts providing a large variety of structures beyond simple presentative/existential constructions.
The portion of VALICO that we selected for the treebank is made up of 237 texts (2,261 LS) organized in four sections as shown in Table 1.
L1
French (FR) German (DE)
Spanish (ES) EN+FR+DE+ES # Texts
# LS Tokens
Although the unpredictability and variation of
a learner product, in terms of vocabulary,
morphology and syntax, makes parsing a LC an
especially challenging task
Our annotation scheme for learner Italian uses
the inventory of the Italian UD PoS tags and
dependency relations
First, we trained UDPipe
For each sentence in VALICO-UD we provide two distinct versions both annotated in UD and tied to an error encoding system (see Section 3.1): one version for the LS and the other for its TH. The latter will differ from the former only when some errors occur. As a trial for this scheme, we selected one hundred sentences (i.e. sample set) containing each at least one error to be annotated.
In writing the TH we decided to adhere as much as
possible to the LS and to focus on linguistic
correctness (e.g. grammaticality) rather than
linguistic appropriateness (e.g. register)
Once the target hypotheses are written, we
applied to them a coding system based on Nicholls
(2003), which was used also in the NUCLE
6Although “rubadore” is reported and marked as obsolete in the Italian Dictionary Olivetti, “rubatore” is the variant reported in De Mauro (2016), our reference dictionary. 7http://www.valico.org/vinca.html sents the general type of error (e.g. wrong form, omission), while the second letter identifies the word class of the required word”.
To provide a finer-grained description of errors,
we used a large variety of letters in the first and
second position (e.g. I: inflection, X: auxiliary)
and a third letter which encodes information about
some grammatical features (e.g. T: tense, M:
mood, G: gender)
Figure 1 shows an annotation example of a LS along with its corresponding TH in the typical CoNLL-U format and with the resource-specific fields used to encode the error information. The sent id field contains the identification code of the sentence: in the example, NameSurname001 (anonymized here) indicates the unique identifier of the text and refers to the transcribers name and surname; the following two-digit number, 35 in the example, indicates the position of the sentence in the text; finally, LS or TH indicates learner sentence and target hypothesis, respectively. The text field contains the uncoded sentence (which can be the learner sentence or the target hypothesis). The err field contains the error annotation based on the coding scheme introduced above. The foreign field includes the index and the PoS of the words which are considered errors due to language transfer. The context field contains the index and the PoS of the words which need replacement due to wrong context-bound lexical choices8. Finally, in line with the ESL, we used the segment field when a sentence was wrongly divided and the typo field to indicate PoS distributional-morphological discrepancies.
In the error-annotated sentence (the “err” field
mentioned above), we report the wrong form(s)
inside the hii h/ii tag and the corrected form(s)
inside the hci h/ci tag. Figure 3 shows three
examples of nested tag and two examples of cascade
errors (i.e. an error which is due to the correction
of another token)
German word adapted to Italian and meaning luggages); thus, we have a cascade hIDG#i h/IDG#i tag which embeds a hFNLi h/FNLi tag (Form Noun Language transfer). The next three tags, hMARi h/MARi, hSARi h/SARi and hSVi h/SVi, indicate Missing pronoun (A) Relative (“che”, that), Spelling pronoun Relative (“ce” instead of “che”) and Spelling Verb errors (“qurda” instead of “guarda”, look), respectively. There is, finally, another example of nested tag involving an Inflection Determiner Gender and an Unneccessary preposiTion errors; this has been used to indicate the multiple-step shift from the LS “sulle” (on the Fem Pl) to its TH counterpart “i” (the Masc Pl): the shift involved a change in the gender of the article (from feminine to masculine) and the drop of the preposition “su” (on), mistakenly used in the LS.
In order to ensure consistency across different annotators, the error annotation guidelines provide a hierarchical order to be applied when dealing with nested tags. We organized the errors in a pyramid with at the bottom mechanical errors (i.e. tokenization, capitalization, spelling and punctuation) and, proceeding towards the apex, morphological (derivation and inflection), lexical (form and replace), and syntactic (missing, unnecessary and word order) errors. For example, following this hierarchical order, mechanical errors should be corrected before a syntactic error. However, cascade errors make an exception and change the correction order, as we seen in Figure 3 in which we have a cascade capitalization error (SVS#) caused by a missing pronoun error (MAX)
Form Inflection Spelling Word segmentation Word order Missing word Unnecessary word Replace word
S T W M U R – and a cascade inflection error (IDG#) due to a lexical error (FNL).
In the LS sample set, containing 1,860 tokens, we marked 496 errors (which represent 26,66% of the LS sample set tokens) distributed as shown in Table 2.
Tag # occ In this Section we describe how we applied literal annotation to the (morpho-)syntactic structure of the LS in particular, relying on the Universal Dependencies scheme.
We annotated UD PoS and relations sticking as
much as possible to the literal reading of the learner sentence, thereby creating a treebank in line with the two existing learner treebanks in the UD framework (ESL and CFL).
Argument Structure: When some extraneous or unnecessary prepositions occur, we annotate the dependencies accordingly. Figure 2 shows a LS in which the verb “guardare”, look, is used as an intransitive verb, thus we annotate its direct object as an oblique9.
Missing or Unnecessary Words: We annotate literally when there are missing or unnecessary words. In the example in Figure 2 the clitic pronoun “ci” is missing , thus we treated “sono” as a copular verb. There are other cases in which the clitic pronoun “ci” is mistakenly combined with the verb to be forming an existential clause, and consequently causing a distributional mismatch (e.g. LS: “[...] non ci era pericoloso o violento”, TH: “[...] non era pericoloso o violento”10). In these cases we mark in the “typo” field the morphological PoS and in the PoS column the distributional PoS, cf. Figure 1.
Extraneous Word Forms: When the learner misuses existent word forms, we annotate them literally. In Figure 4, the learner used a gerund, “leggendo” (reading), instead of the infinitive “ a 9In all the examples SE stands for spelling error, REFL for reflexive pronoun, PP for past participle, GE for gerund and Impf for imperfect tense.
10LS: “[...] not there it-be Impf dargerous or violent”, TH: “[...] not it-be Impf dangerous or violent”. leggere” (to read). We then labeled it as an adverbial clause in the LS (Figure 4) and as an open clausal complement in the TH (Figure 5).
Spelling: Some examples of spelling errors are presented in Figure 2. We lemmatize and PoStag them referring to their correct versions, similarly to Andorno and Rastelli (2009, p. 58). Thus, “ce” was treated as “che”, which,11, and “qurda” as “guarda” look.
Word Formation: We do not treat literally valid words that are contextually implausible. We consider them differently depending on the PoS of the intended word: if the intended word has the same PoS we signal it in the “context” field (e.g. LS: “[...] salvando una ragazza indefessa”, TH: “[...] salvando una ragazza indifesa”12), if it is different in the “typo” field (cf. Figure 1).
Nonexistent Words: In cases in which the learner wrote a word which does not exist in Italian and it is arguably a foreign word, we signal it in the “foreign” field13. In the example in Figure 1 the word “cara” (i.e. an adjective translatable into beloved) is arguably a transfer from the Spanish noun meaning face. In this case we lemmatize it with the correct lemma of “cara”. In addition, in the “typo” field we mark the occurring mismatch between distributional and morphological PoS. Word Tokenization: If one word is mistakenly segmented into two, we use the “goeswith” relation, as germane to UD annotation guidelines14. If two words are mistakenly segmented into one, we use X as PoS and decide the relation on a caseby-case basis. For example in LS: “[...] butta tutto perterra”, TH: “[...] butta tutto per terra”15 we assigned to “perterra” PoS ‘X’ and dependency relation ‘obl’.
As stated above, the complete manual revision of the treebank is still in progress; however, with the aim of assessing the annotation quality of this preliminary sample set, as well as the quality of the annotation guidelines (especially the ones con11When “ce” is used instead of “c’e`”, there is, we treat it as a single token and mark it as root, in line with what we would have done if it were “c’e`”.
12LS: “[...] saving a untiring girl”, TH: “[...] saving a vulnerable girl”.
13The lemma will be its Italian (quasi-)equivalent.
14https://universaldependencies.org/u/overview/typos.html
15[...] he-throw everything on the ground.
cerning the LS section) both LS and TH
sections were annotated by two independent
annotators. The inter-annotator agreement was then
computed, considering two measures in
particular: UAS (Unlabeled Attachment Score) and
LAS (Labeled Attachment Score) for the
assignment of both parent node and dependency relation,
and the Cohen’s kappa coefficient
set LS TH
UAS In this paper we introduced VALICO-UD and proposed an annotation scheme suitable for texts of learner Italian encompassing both UD and error annotation. Our scheme follows the principle of “literal annotation” and takes PoS and dependency morphological-distributional mismatches into account. Our error tag set seems adequate to bookmark errors, providing also a fine-grained description of some of them.
There are a number of possible applications for
the monolingual parallel treebank proposed in this
paper. In the near future, we plan to apply the tree
edit distance to LS and TH to measure linguistic
competence. Recently, the tree edit distance has
been applied to various tasks
16http://universaldependencies.org/conll18/evaluation.html