=Paper=
{{Paper
|id=Vol-2481/paper28
|storemode=property
|title=Towards an Italian Learner Treebank in Universal Dependencies
|pdfUrl=https://ceur-ws.org/Vol-2481/paper28.pdf
|volume=Vol-2481
|authors=Elisa Di Nuovo,Cristina Bosco,Alessandro Mazzei,Manuela Sanguinetti
|dblpUrl=https://dblp.org/rec/conf/clic-it/NuovoBMS19
}}
==Towards an Italian Learner Treebank in Universal Dependencies==
https://ceur-ws.org/Vol-2481/paper28.pdf
Towards an Italian Learner Treebank in Universal Dependencies
Elisa Di Nuovo Cristina Bosco
Dipartimento di Lingue e Letterature Alessandro Mazzei
Straniere e Culture Moderne Manuela Sanguinetti
University of Turin Dipartimento di Informatica
elisa.dinuovo@unito.it University of Turin
{bosco,mazzei,msanguin}@di.unito.it
Abstract and Paquot, 2015; Malmasi, 2016), Grammatical-
Error Detection and Correction (Leacock et al.,
In this paper we describe the preliminary 2015; Ng et al., 2014), and Automated Essay Scor-
work on a novel treebank which includes ing (Higgins et al., 2015).
texts written by learners of Italian drawn In this paper we describe the development of a
from the VALICO corpus. Data pro- novel learner Italian treebank, i.e. VALICO-UD,
cessing mostly involved the application of in which Universal Dependencies (UD) formal-
Universal Dependencies formalism and er- ism is tied to error annotation. The considerations
ror annotation. First, we parsed the texts of the annotation process, carried out on a set of
on UDPipe trained on the existent Ital- one hundred sentences selected from a subcorpus
ian UD treebanks, then we manually cor- of VALICO1 (see Table 1) (Corino and Marello,
rected them. The particular focus of this 2017), allowed us to test a pilot scheme which pin-
paper is on a one-hundred-sentence sam- points some of the features of L2 Italian.
ple of the collection, used as a case study This paper is organized as follows: in Section 2
to define an annotation scheme for identi- we provide an overview of LC, focusing on Ital-
fying the linguistic phenomena character- ian resources in particular; in Section 3 we present
izing learners’ interlanguage. the data and the error annotation of VALICO-UD;
in Section 4 we offer some examples of how we
1 Introduction applied literal annotation to the learner sentences
(LS) and, finally, in Section 6 we present conclu-
The increasing interest in Learner Corpora (hence- sion and future work.
forth LC) is twofold motivated. On the one hand,
LC are an especially valuable source of knowl- 2 Related work
edge for interlanguage varieties. They allow in-
depth comparisons of non-native varieties, help- LC, also called interlanguage or L2 corpora, are
ing to elucidate the properties of the interlan- collections of data produced by foreign or sec-
guage developed by learners with different mother ond language learners (Granger, 2008). Most LC
tongues and learning levels. For this reason, LC projects were launched in the nineties and focused
are important resources enabling data-driven stud- mainly on learner English (Tono, 2003), but re-
ies exploited within several research areas, such cently we have witnessed an increasing interest
as Second Language Acquisition, Foreign Lan- in LC for other target languages. This has con-
guage Teaching, Contrastive Interlanguage Anal- tributed to the establishment of learner corpus re-
ysis, Computer-aided Error Analysis, Computer- search (Tono, 2003).
Assisted Language Learning and L2 Lexicogra- LC can be enriched with Part of Speech (PoS)
phy (e.g. (Pravec, 2002; Granger, 2008; McEnery tagging, syntactic, semantic, discourse structure
and Xiao, 2011)). On the other hand, LC have and error-tagging (with explicit or implicit target
raised considerable computational interest, which hypotheses2 ) annotation (Garside et al., 1997). To
is closely related to their usefulness in tasks provide linguistic annotation, NLP tools are of-
such as Native Language Identification (Jarvis ten used (Huang et al., 2018) and combined with
1
Copyright c 2019 for this paper by its authors. Use http://www.valico.org/
2
permitted under Creative Commons License Attribution 4.0 A reconstructed LS on which error identification is based
International (CC BY 4.0). (Reznicek et al., 2013).
human post-editing in order to overcome issues collection of non-native Italian texts elicited by
arising from the failures of the automatic analy- comic strips proposed to the learners. It consists of
sis (Geertzen et al., 2013; Granger et al., 2009; a selection of narrative and descriptive texts pro-
Dahlmeier et al., 2013). viding a large variety of structures beyond simple
Among the 14 learner Italian corpora registered presentative/existential constructions.
in the Learner Corpora around the World list3 , The portion of VALICO that we selected for the
the majority are in the form of plain texts, or they treebank is made up of 237 texts (2,261 LS) orga-
only annotate PoS (COLI, LOCCLI and CAIL24 , nized in four sections as shown in Table 1.
and VALICO), while only MERLIN (Boyd et al.,
2014) annotates syntax and errors (with explicit L1 # Texts # LS Tokens
target hypotheses). English (EN) 60 8,285
Although MERLIN contains 816 texts written French (FR) 59 7,301
in non-native Italian (Boyd et al., 2014), they are German (DE) 58 7,417
not balanced for learners’ mother tongue and are Spanish (ES) 60 7,365
not annotated using a standard annotation for syn-
EN+FR+DE+ES 237 30,368
tax, which would allow comparisons with other re-
sources. To fill this gap, we decided to develop Table 1: VALICO-UD in figures – LS section.
VALICO-UD, a L1-balanced resource developed
within the UD formalism, thus providing a greater
potential for contrastive analysis. Indeed, a UD- Although the unpredictability and variation of
annotated LC can be compared with other LC a learner product, in terms of vocabulary, mor-
(therefore different interlanguages) or also with phology and syntax, makes parsing a LC an espe-
native corpora of the L1 involved. For all these cially challenging task (Corino and Russo, 2016;
reasons, we decided to develop this new learner Dı́az-Negrillo et al., 2010), it is highly recom-
Italian treebank within the UD formalism. Refer- mendable for smoothly retrieving interlanguage
ences were the English and Chinese experiences, features. Due to this peculiarity of interlanguage,
respectively the English Second Language (ESL) keeping separated the LS from its specifically built
(Berzak et al., 2016) and the Chinese Foreign Lan- target hypothesis (TH) is highly recommended
guage (CFL) (Lee et al., 2017) treebanks. (Lüdeling et al., 2005).
The scholars involved in the annotation of the Our annotation scheme for learner Italian uses
ESL and CFL treebanks decided to follow a well- the inventory of the Italian UD PoS tags and de-
established line of work, for which learner lan- pendency relations (Bosco et al., 2013; Bosco et
guage analysis is centered upon morpho-syntactic al., 2014) and the related guidelines. In addition,
surface evidence. This is motivated by various we tried to follow as much as possible the ESL
studies, e.g. (Dı́az-Negrillo et al., 2010; Ragheb treebank to have comparable resources.
and Dickinson, 2012), in which the difference First, we trained UDPipe (Straka et al., 2016)
between morphological and distributional PoS is on the Italian UD corpora, which include stan-
stressed. We decided to follow this line of research dard texts, ISDT (Bosco et al., 2014), and Twitter
annotating discrepancies between morphological posts, POSTWITA-UD (Sanguinetti et al., 2018).
and distributional PoS, as described in the next Second, we automatically parsed VALICO-UD.
sections. However, in lieu of carrying out manual Third, we manually corrected the treebank. This
annotation from scratch, such as in the ESL, we step is currently ongoing and we envision the tree-
combined automatic annotation and manual post- bank to be released in the UD repository in a few
editing (as shown in the next section). months.
For each sentence in VALICO-UD we provide
3 Data and annotation two distinct versions both annotated in UD and
The data of VALICO-UD are drawn from the tied to an error encoding system (see Section 3.1):
VALICO corpus (Corino and Marello, 2017), a one version for the LS and the other for its TH.
The latter will differ from the former only when
3
https://uclouvain.be/en/research- some errors occur. As a trial for this scheme, we
institutes/ilc/cecl/learner-corpora-around-the-world.html.
4
COLI, LOCCLI and CAIL2 are developed at Università selected one hundred sentences (i.e. sample set)
per Stranieri di Perugia and coordinated by Stefania Spina. containing each at least one error to be annotated.
# sent id = NameSurname00135LS # sent id = NameSurname00135TH
# text = Può essere un rubadore perche ha la cara chiusa e minacciata. # text = Può essere un rubatore perché ha la faccia chiusa e minacciosa.
# err = Può essere un hRNihiirubadoreh/iihcirubatoreh/cih/RNi # err = Può essere un hRNihiirubadoreh/iihcirubatoreh/cih/RNi
hMIihiipercheh/iihciperchéh/cih/MIi ha la hFNLihiicarah/ii hMIihiipercheh/iihciperchéh/cih/MIi ha la hFNLihiicarah/ii
hcifacciah/cih/FNLi chiusa e hDJihiiminacciatah/ii hcifacciah/cih/FNLi chiusa e hDJihiiminacciatah/ii
hciminacciosah/cih/DJi. hciminacciosah/cih/DJi.
# segment = # segment =
# typo = 8 ADJ, 11 VERB # typo = 8 ADJ, 11 VERB
# foreign = 8 NOUN # foreign = 8 NOUN
# context = 4 NOUN # context = 4 NOUN
1 Può potere AUX VM 4 aux 1 Può potere AUX VM 4 aux
2 essere essere AUX V 4 cop 2 essere essere AUX V 4 cop
3 un uno DET RI 4 det 3 un uno DET RI 4 det
4 rubadore rubadore NOUN S 0 root 4 rubatore rubatore NOUN S 0 root
5 perche perché SCONJ CS 6 mark 5 perché perché SCONJ CS 6 mark
6 ha avere VERB V 4 advcl 6 ha avere VERB V 4 advcl
7 la il DET RD 8 det 7 la il DET RD 8 det
8 cara caro NOUN S 6 obj 8 faccia faccia NOUN S 6 obj
9 chiusa chiuso ADJ A 8 amod 9 chiusa chiuso ADJ A 8 amod
10 e e CCONJ CC 11 cc 10 e e CCONJ CC 11 cc
11 minacciata minacciato ADJ A 9 conj 11 minacciosa minaccioso ADJ A 9 conj
12 . . PUNCT FS 4 punct 12 . . PUNCT FS 4 punct
Figure 1: Example of two CoNLL-U trees of the LS (left) and TH (right) number #35: He-can to-be a
thief because he-has the face closed and threaten PP.
3.1 Error Annotation sents the general type of error (e.g. wrong form,
In writing the TH we decided to adhere as much as omission), while the second letter identifies the
possible to the LS and to focus on linguistic cor- word class of the required word”.
rectness (e.g. grammaticality) rather than linguis- To provide a finer-grained description of errors,
tic appropriateness (e.g. register) (Reznicek et al., we used a large variety of letters in the first and
2013)5 . For this reason, sometimes we sacrificed second position (e.g. I: inflection, X: auxiliary)
naturalness for the sake of adherence to the LS. and a third letter which encodes information about
This principle was applied also to lexical errors re- some grammatical features (e.g. T: tense, M:
quiring replacement. For instance, in Figure 1, the mood, G: gender) (Simone, 2008, pp. 303–346)
term “rubadore” in the LS was replaced with “ru- and other phenomena involved (e.g. capitaliza-
batore” and not with its more common synonym tion, language transfer and government). Finally,
“ladro”, thief.6 With this principle in mind, we de- Nicholls included a catch-all code (CE: complex
cided to correct words if they are not present nei- error) to cover complex, multiple errors. In our
ther in the VINCA corpus7 (the reference corpus sample set, we did not use it because we managed
specifically compiled for VALICO and containing to describe all errors encountered using nested
texts based on the same comic strips but written by XML tags. However, we do not exclude that, ap-
Italian native speakers) nor in our reference dictio- plying the error codes to the whole corpus, we
nary, Il Nuovo Vocabolario di Base della Lingua might find particularly complex errors which need
Italiana (De Mauro, 2016). In fact, the VINCA to be marked using this code.
corpus is quite small and the language used sounds Figure 1 shows an annotation example of a LS
quite unnatural though being produced by speak- along with its corresponding TH in the typical
ers whose mother tongue is namely Italian (see CoNLL-U format and with the resource-specific
Corino and Marello (2017, p. 12)). fields used to encode the error information. The
Once the target hypotheses are written, we ap- sent id field contains the identification code of
plied to them a coding system based on Nicholls the sentence: in the example, NameSurname001
(2003), which was used also in the NUCLE (anonymized here) indicates the unique identifier
(Dahlmeier et al., 2013) and FCE (Yannakoudakis of the text and refers to the transcribers name and
et al., 2011) corpora. Our system follows surname; the following two-digit number, 35 in
Nicholls’s same principle: “the first letter repre- the example, indicates the position of the sentence
5
In the future we plan to provide a second TH, focusing in the text; finally, LS or TH indicates learner sen-
on linguistic appropriateness. tence and target hypothesis, respectively. The text
6
Although “rubadore” is reported and marked as obsolete field contains the uncoded sentence (which can be
in the Italian Dictionary Olivetti, “rubatore” is the variant re-
ported in De Mauro (2016), our reference dictionary. the learner sentence or the target hypothesis). The
7
http://www.valico.org/vinca.html err field contains the error annotation based on
Figure 2: LS #10.
Figure 3: Error-annotated sentence #10.
the coding scheme introduced above. The foreign German word adapted to Italian and meaning lug-
field includes the index and the PoS of the words gages); thus, we have a cascade hIDG#i h/IDG#i
which are considered errors due to language trans- tag which embeds a hFNLi h/FNLi tag (Form
fer. The context field contains the index and the Noun Language transfer). The next three
PoS of the words which need replacement due to tags, hMARi h/MARi, hSARi h/SARi and
wrong context-bound lexical choices8 . Finally, in hSVi h/SVi, indicate Missing pronoun (A) Rel-
line with the ESL, we used the segment field when ative (“che”, that), Spelling pronoun Relative
a sentence was wrongly divided and the typo field (“ce” instead of “che”) and Spelling Verb errors
to indicate PoS distributional-morphological dis- (“qurda” instead of “guarda”, look), respectively.
crepancies. There is, finally, another example of nested tag
In the error-annotated sentence (the “err” field involving an Inflection Determiner Gender and an
mentioned above), we report the wrong form(s) Unneccessary preposiTion errors; this has been
inside the hii h/ii tag and the corrected form(s) used to indicate the multiple-step shift from the
inside the hci h/ci tag. Figure 3 shows three ex- LS “sulle” (on the Fem Pl) to its TH counterpart
amples of nested tag and two examples of cascade “i” (the Masc Pl): the shift involved a change
errors (i.e. an error which is due to the correction in the gender of the article (from feminine to
of another token) (Andorno and Rastelli, 2009, masculine) and the drop of the preposition “su”
p. 52). The hMAXi h/MAXi tag at the beginning (on), mistakenly used in the LS.
of the sentence, for example, indicates a missing In order to ensure consistency across different
existential-construction pronoun, i.e. “Sono” annotators, the error annotation guidelines pro-
(are) instead of “Ci sono” (there are). After vide a hierarchical order to be applied when deal-
the insertion of the missing pronoun “Ci”, the ing with nested tags. We organized the errors
capital “S” in “Sono” needs to be changed into in a pyramid with at the bottom mechanical er-
a lowercase “s”: this is a case in which we have rors (i.e. tokenization, capitalization, spelling and
a cascade capitalization error and we mark it punctuation) and, proceeding towards the apex,
adding a hashtag after the normal error code, as morphological (derivation and inflection), lexical
in hSVS#i h/SVS#i. Another cascade error is (form and replace), and syntactic (missing, un-
found in the next nested tag: we have an Inflection necessary and word order) errors. For example,
Determiner Gender error which is caused by following this hierarchical order, mechanical er-
the correction of the expression “tanti cofferi”, rors should be corrected before a syntactic error.
involving a determiner and a noun (“cofferi” is a However, cascade errors make an exception and
8
change the correction order, as we seen in Figure
Only those choices in which there is no mismatch be-
tween distributional and morphological PoS are registered in 3 in which we have a cascade capitalization error
this field. (SVS#) caused by a missing pronoun error (MAX)
Figure 4: LS #88.
Figure 5: TH #88.
and a cascade inflection error (IDG#) due to a lex- much as possible to the literal reading of the
ical error (FNL). learner sentence, thereby creating a treebank in
In the LS sample set, containing 1,860 tokens, line with the two existing learner treebanks in the
we marked 496 errors (which represent 26,66% of UD framework (ESL and CFL).
the LS sample set tokens) distributed as shown in Argument Structure: When some extraneous or
Table 2. unnecessary prepositions occur, we annotate the
dependencies accordingly. Figure 2 shows a LS in
Error category Tag # occ % tot
which the verb “guardare”, look, is used as an in-
Derivation D 24 4.84% transitive verb, thus we annotate its direct object
Form F 71 14.31% as an oblique9 .
Inflection I 72 14.51% Missing or Unnecessary Words: We annotate
Spelling S 92 18.55% literally when there are missing or unnecessary
Word segmentation T 16 3.22% words. In the example in Figure 2 the clitic pro-
Word order W 15 3.02% noun “ci” is missing , thus we treated “sono” as a
Missing word M 76 15.32% copular verb. There are other cases in which the
Unnecessary word U 55 11.09% clitic pronoun “ci” is mistakenly combined with
Replace word R 75 15.12% the verb to be forming an existential clause, and
consequently causing a distributional mismatch
Total – 496 –
(e.g. LS: “[...] non ci era pericoloso o violento”,
Table 2: Error categories as encoded in the first TH: “[...] non era pericoloso o violento”10 ). In
letter (general error type) and their distribution in these cases we mark in the “typo” field the mor-
the sample set. phological PoS and in the PoS column the distri-
butional PoS, cf. Figure 1.
Extraneous Word Forms: When the learner mis-
4 From VALICO to VALICO-UD uses existent word forms, we annotate them lit-
erally. In Figure 4, the learner used a gerund,
In this Section we describe how we applied literal “leggendo” (reading), instead of the infinitive “ a
annotation to the (morpho-)syntactic structure of
9
the LS in particular, relying on the Universal De- In all the examples SE stands for spelling error, REFL
pendencies scheme. for reflexive pronoun, PP for past participle, GE for gerund
and Impf for imperfect tense.
Literal Annotation 10
LS: “[...] not there it-be Impf dargerous or violent”, TH:
We annotated UD PoS and relations sticking as “[...] not it-be Impf dangerous or violent”.
leggere” (to read). We then labeled it as an ad- cerning the LS section) both LS and TH sec-
verbial clause in the LS (Figure 4) and as an open tions were annotated by two independent anno-
clausal complement in the TH (Figure 5). tators. The inter-annotator agreement was then
Exceptions to Literal Annotation computed, considering two measures in partic-
Spelling: Some examples of spelling errors are ular: UAS (Unlabeled Attachment Score) and
presented in Figure 2. We lemmatize and PoS- LAS (Labeled Attachment Score) for the assign-
tag them referring to their correct versions, sim- ment of both parent node and dependency relation,
ilarly to Andorno and Rastelli (2009, p. 58). Thus, and the Cohen’s kappa coefficient (Cohen, 1960)
“ce” was treated as “che”, which,11 , and “qurda” for dependency relations only (similarly to Lynn
as “guarda” look. (2016)). UAS and LAS were computed with the
Word Formation: We do not treat literally valid script provided in the second CoNLL shared task
words that are contextually implausible. We con- on multilingual parsing (Zeman et al., 2018)16 .
sider them differently depending on the PoS of the The results are reported in Table 3, and though
intended word: if the intended word has the same showing slightly higher results for the TH set,
PoS we signal it in the “context” field (e.g. LS: overall they are very close across the sets. Espe-
“[...] salvando una ragazza indefessa”, TH: “[...] cially as regards the LS section, this is evidence of
salvando una ragazza indifesa”12 ), if it is different the guidelines clarity and of the annotators’ con-
in the “typo” field (cf. Figure 1). sistency, even when dealing with non-canonical
Nonexistent Words: In cases in which the learner syntactic structures.
wrote a word which does not exist in Italian and
it is arguably a foreign word, we signal it in the set UAS LAS kappa
“foreign” field13 . In the example in Figure 1 the LS 92.11% 88.63% 0.8988
word “cara” (i.e. an adjective translatable into TH 92.47% 88.88% 0.9068
beloved) is arguably a transfer from the Spanish
noun meaning face. In this case we lemmatize it Table 3: Agreement results on the sample set of
with the correct lemma of “cara”. In addition, in both LS and TH.
the “typo” field we mark the occurring mismatch
between distributional and morphological PoS.
Word Tokenization: If one word is mistakenly 6 Conclusion and future work
segmented into two, we use the “goeswith” rela- In this paper we introduced VALICO-UD and pro-
tion, as germane to UD annotation guidelines14 . If posed an annotation scheme suitable for texts of
two words are mistakenly segmented into one, we learner Italian encompassing both UD and error
use X as PoS and decide the relation on a case- annotation. Our scheme follows the principle of
by-case basis. For example in LS: “[...] butta tutto “literal annotation” and takes PoS and dependency
perterra”, TH: “[...] butta tutto per terra”15 we as- morphological-distributional mismatches into ac-
signed to “perterra” PoS ‘X’ and dependency rela- count. Our error tag set seems adequate to book-
tion ‘obl’. mark errors, providing also a fine-grained descrip-
tion of some of them.
5 Inter-Annotator Agreement There are a number of possible applications for
As stated above, the complete manual revision of the monolingual parallel treebank proposed in this
the treebank is still in progress; however, with paper. In the near future, we plan to apply the tree
the aim of assessing the annotation quality of this edit distance to LS and TH to measure linguistic
preliminary sample set, as well as the quality of competence. Recently, the tree edit distance has
the annotation guidelines (especially the ones con- been applied to various tasks (Emms, 2008; Tsar-
faty et al., 2011; Plank et al., 2015), and a study
11
When “ce” is used instead of “c’è”, there is, we treat it as has formalized the notion of syntactic anisomor-
a single token and mark it as root, in line with what we would phism (Ponti et al., 2018). We aim to explore a cor-
have done if it were “c’è”.
12
LS: “[...] saving a untiring girl”, TH: “[...] saving a vul- relation between these notions and the linguistic
nerable girl”. competence to describe the achievements of for-
13
The lemma will be its Italian (quasi-)equivalent. eign language learners.
14
https://universaldependencies.org/u/overview/typos.html
15 16
[...] he-throw everything on the ground. http://universaldependencies.org/conll18/evaluation.html
References Martin Emms. 2008. Tree Distance and Some Other
Variants of Evalb. In Proceedings of the Sixth In-
Cecilia Maria Andorno and Stefano Rastelli. 2009. ternational Conference on Language Resources and
Un’annotazione orientata alla ricerca acquisizionale. Evaluation, pages 1373–1379.
In Cecilia Maria Andorno and Stefano Rastelli, ed-
itors, Corpora di italiano L2: tecnologie, metodi, Roger Garside, Geoffrey N. Leech, and Tony McEnery.
spunti teorici, pages 49–70. Guerra. 1997. Corpus Annotation: Linguistic Information
from Computer Text Corpora. Taylor & Francis.
Yevgeni Berzak, Jessica Kenney, Carolyn Spadine,
Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Jeroen Geertzen, Theodora Alexopoulou, and Anna
Sebastian Garza, and Boris Katz. 2016. Universal Korhonen. 2013. Automatic Linguistic Annotation
Dependencies for Learner English. In Proceedings of Large Scale L2 Databases: The EF-Cambridge
of the 54th Annual Meeting of the Association for Open Language Database (EFCAMDAT). In Pro-
Computational Linguistics, pages 737–746. ceedings of the 31st Second Language Research Fo-
rum, pages 240–254.
Cristina Bosco, Montemagni Simonetta, and Simi
Maria. 2013. Converting Italian Treebanks: To- Sylviane Granger, Estelle Dagneaux, Fanny Meunier,
wards an Italian Stanford Dependency Treebank. In and Magali Paquot. 2009. International Corpus of
Proceedings of the 7th Linguistic Annotation Work- Learner English. Louvain University Press.
shop and Interoperability with Discourse, pages 61–
69. Sylviane Granger. 2008. Learner Corpora. In Anke
Lüdeling and Merja Kytö, editors, Corpus Linguis-
Cristina Bosco, Felice Dell’Orletta, Simonetta Monte-
tics, volume 1, pages 259–275. Walter de Gruyter.
magni, Manuela Sanguinetti, and Maria Simi. 2014.
The EVALITA 2014 Dependency Parsing Task. In
Derrick Higgins, Chaitanya Ramineni, and Klaus
Proceedings of EVALITA 2014 Evaluation of NLP
Zechner. 2015. Learner Corpora and Automated
and Speech Tools for Italian, pages 1–8.
Scoring. In Sylviane Granger, Gaëtanelle Gilquin,
Adriane Boyd, Jirka Hana, Lionel Nicolas, Detmar and Fanny Meunier, editors, The Cambridge Hand-
Meurers, Katrin Wisniewski, Andrea Abel, Karin book of Learner Corpus Research, pages 587–604.
Schöne, Barbora Stindlová, and Chiara Vettori. Cambridge University Press.
2014. The MERLIN corpus: Learner Language and
the CEFR. In Conference on Language Resources Yan Huang, Akira Murakami, Theodora Alexopoulou,
and Evaluation, pages 1281–1288. and Anna Korhonen. 2018. Dependency Parsing of
Learner English. International Journal of Corpus
Jacob Cohen. 1960. A Coefficient of Agreement for Linguistics, 23(1):28–54.
Nominal Scales. Educational and Psychological
Measurement, 20(1):37–46. Scott Jarvis and Magali Paquot. 2015. Learner Cor-
pora and Native Language Identification. In Syl-
Elisa Corino and Carla Marello. 2017. Italiano di viane Granger, Gaëtanelle Gilquin, and Fanny Me-
stranieri. I corpora VALICO e VINCA. Guerra. unier, editors, The Cambridge Handbook of Learner
Corpus Research, pages 605–628. Cambridge Uni-
Elisa Corino and Claudio Russo. 2016. Parsing di Cor- versity Press.
pora di Apprendenti di Italiano: un Primo Studio su
VALICO. In Proceedings of the 3rd Italian Confer- Claudia Leacock, Martin Chodorow, and Joel Tetrault.
ence on Computational Linguistics, CLiC-it 2016, 2015. Automatic Grammar- and Spell-Checking
pages 105–110. for Language Learners. In Sylviane Granger,
Gaëtanelle Gilquin, and Fanny Meunier, editors, The
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. Cambridge Handbook of Learner Corpus Research,
2013. Building a large annotated corpus of learner pages 567–586. Cambridge University Press.
English: The NUS corpus of learner English. In
Proceedings of the Eighth Workshop on Innovative John Lee, Herman Leung, and Keying Li. 2017. To-
Use of NLP for Building Educational Applications, wards Universal Dependencies for Learner Chinese.
pages 22–31. In Proceedings of the NoDaLiDa 2017 Workshop on
Universal Dependencies (UDW 2017), pages 67–71.
Tullio De Mauro. 2016. Il Nuovo Vocabolario
di Base della Lingua Italiana. Internazionale, Anke Lüdeling, Maik Walter, Emil Kroymann, and Pe-
http://www.internazionale.it/opinione/tullio-de- ter Adolphs. 2005. Multi-level error annotation in
mauro/2016/12/23/il-nuovo-vocabolario-di-base- learner corpora. In Proceedings of Corpus Linguis-
della-lingua-italiana. tics 2005, volume 1, pages 14–17.
Ana Dı́az-Negrillo, Detmar Meurers, Salvador Valera, Teresa Lynn. 2016. Irish Dependency Treebank-
and Holger Wunsch. 2010. Towards Interlanguage ing and Parsing. Ph.D. thesis, Dublin City Uni-
POS Annotation for Effective Learner Corpora in versity, Ireland and Macquarie University, Sydney,
SLA and FLT. Language Forum, 36(1-2):139–154. Australia.
Shervin Malmasi. 2016. Native Language Identifica- Yukio Tono. 2003. Learner corpora: design, develop-
tion: explorations and applications. Ph.D. thesis, ment and applications. In Proceedings of the Corpus
Macquarie University, Sydney, Australia. Linguistics 2003 conference, pages 800–809.
Tony McEnery and Richard Xiao. 2011. What cor- Reut Tsarfaty, Joakim Nivre, and Evelina Andersson.
pora can offer in language teaching and learning. In 2011. Evaluating dependency parsing: Robust and
Eli Hinkel, editor, Handbook of Research in Second heuristics-free cross-annotation evaluation. In Pro-
Language Teaching and Learning, volume 2, pages ceedings of the 2011 Conference on Empirical Meth-
364–380. Routledge. ods in Natural Language Processing, pages 385–
396.
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian
Hadiwinoto, Raymond Hendy Susanto, and Christo- Helen Yannakoudakis, Ted Briscoe, and Ben Medlock.
pher Bryant. 2014. The CoNLL-2014 shared task 2011. A new dataset and method for automatically
on grammatical error correction. In Proceedings of grading esol texts. In Proceedings of the 49th An-
the Eighteenth Conference on Computational Natu- nual Meeting of the Association for Computational
ral Language Learning: Shared Task, pages 1–14. Linguistics: Human Language Technologies, pages
180–189.
Diane Nicholls. 2003. The Cambridge Learner Cor-
pus: Error coding and analysis for lexicography and Daniel Zeman, Jan Hajič, Martin Popel, Martin Pot-
ELT. In Proceedings of the Corpus Linguistics 2003 thast, Milan Straka, Filip Ginter, Joakim Nivre, and
Conference, volume 16, pages 572–581. Slav Petrov. 2018. CoNLL 2018 shared task: Mul-
tilingual parsing from raw text to universal depen-
Barbara Plank, Héctor Martı́nez Alonso, Željko Agić,
dencies. In Proceedings of the CoNLL 2018 Shared
Danijela Merkler, and Anders Søgaard. 2015. Do
Task: Multilingual Parsing from Raw Text to Univer-
dependency parsing metrics correlate with human
sal Dependencies, pages 1–19.
judgments? In Proceedings of the Nineteenth Con-
ference on Computational Natural Language Learn-
ing, pages 315–320.
Edoardo Maria Ponti, Roi Reichart, Anna Korhonen,
and Ivan Vulić. 2018. Isomorphic transfer of syn-
tactic structures in cross-lingual NLP. In Proceed-
ings of the 56th Annual Meeting of the Association
for Computational Linguistics, pages 1531–1542.
Norma A. Pravec. 2002. Survey of Learner Corpora.
ICAME journal, 26(1):8–14.
Marwa Ragheb and Markus Dickinson. 2012. Defin-
ing syntax for learner language annotation. In Pro-
ceedings of COLING 2012: Posters, pages 965–974.
Marc Reznicek, Anke Lüdeling, and Hagen
Hirschmann. 2013. Competing target hypotheses
in the falko corpus. In Ana Dı́az-Negrillo, Nicolas
Ballier, and Paul Thompson, editors, Automatic
treatment and analysis of learner corpus data, vol-
ume 59, pages 101–123. John Benjamins Publishing
Company.
Manuela Sanguinetti, Cristina Bosco, Alberto Lavelli,
Alessandro Mazzei, Oronzo Antonelli, and Fabio
Tamburini. 2018. PoSTWITA-UD: an Italian
Twitter Treebank in Universal Dependencies. In
Proceedings of the Eleventh International Confer-
ence on Language Resources and Evaluation, pages
1768–1775.
Raffaele Simone. 2008. Fondamenti di linguistica.
Laterza.
Milan Straka, Jan Hajic, and Jana Straková. 2016.
Udpipe: Trainable pipeline for processing conll-u
files performing tokenization, morphological anal-
ysis, pos tagging and parsing. In Proceedings of
the Tenth International Conference on Language Re-
sources and Evaluation, pages 88–99.