English. In this paper we present MarkIT, a treebank of marked constructions in Italian, containing around 800 sentences with dependency annotation. We detail the process to extract the sentences and manually correct them. The resource covers seven types of marked constructions plus some ambiguous sentences, whose syntax can be wrongly classified as marked. We also present a preliminary evaluation of parsing performance, comparing a model trained on existing Italian treebanks with the model obtained by adding MarkIT to the training set.
Italiano. In questo lavoro presentiamo MarkIT, un treebank di costruzioni marcate in italiano che contiene circa 800 frasi annotate con strutture a dipendenze. Abbiamo descritto nel dettaglio il processo seguito per estrarre le frasi e correggerne manualmente la struttura sintassi. La risorsa comprende sette tipologie di costruzioni marcate oltre ad alcune costruzioni ambigue che potrebbero essere classificate erroneamente come marcate. Presentiamo inoltre una valutazione preliminare delle performance del parser in cui confrontiamo un modello allenato sui treebank esistenti dell’italiano con il modello ottenuto aggiungendo anche MarkIT.
In recent years, the goal to develop robust frameworks for consistent annotation of syntactic dependencies across different human languages has
Copyright © 2021 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
led to the creation of Universal Dependencies
(UD), an initiative covering nearly 200 treebanks
in more than 100 languages. Since UD treebanks
are then used to train syntactic parsers, it is
important that they account for as many phenomena
as possible that can be found in a language, and
not only for canonical expressions typically
written in news. The purpose to encompass the
variety of use in the Italian language has been
pursued by including different genres in the VIT
treebank
Our goal is to build a treebank of marked
constructions that reflects actual usage of Italian, in
particular of the neo-standard variant
The final corpus contains more than 2,500 essays and almost 1.5 million tokens. We extract around 800 sentences labeled with a marked structure and annotate them at syntactic level. Although the essays cannot be released because of copyright issues, the sentences in isolation, with no additional information related to the authors or the textual context, can be freely distributed.
The essays were written in a time span of 15 years by different authors and dealing with a number of different topics, which guarantees a high variability of the sentence content and structure. On the other hand, since they were part of a formal students’ examination, they tend to be free from jargon, grammatical errors and abbreviations that may derive from sentences extracted from social media and that may represent an additional challenge for parsers. 4
With marked sentences we refer to those
constructions which present a non-canonical order of
constituents. In Italian, the canonical order of the
syntactic structure is S V+fin V-fin OX , where S is
subject, V+fin is a finite verb or an auxiliary verb,
Vifn is a non-finite verb, O is the direct object and X
other complements
Left dislocated sentences entail the displacement
or anteposition of a specific syntagm to the left
of the sentence. The dislocated element connects
with the rest of the sentence thanks to an
introductory preposition (1) or a pronominal reprise (2), for
which a resumptive clitic pronoun pleonastically
co-refers to the displaced nominal element (the
topic). The clitic reprise is compulsory whether
the displaced element was the direct object, as
long as it is in the positive form
(1) A questo evento (ci) partecipano soltanto artisti gia` noti To this event (clitic) participate only artists already known (2) Molto meno successo Eminem lo ha avuto quest’anno Much less success Eminem it has had this year 4.2
In hanging topic sentences, similarly to left
dislocation, the dislocated element is moved to the left,
at the beginning of the sentence. However, in this
case, the displaced element is isolated at the
beginning of the sentence, and it is not syntactically
linked to the verb
(3) [...] ma il cervello, senza di esso non siamo niente
But the brain, without it we are nothing 4.3
Right dislocated sentences operate a topicalization of the comment and, differently from left dislocated structures, the pronominal reprise is not compulsory when the dislocated element is the direct object. Nevertheless, since the non-marked position of the right dislocated elements is still in postverbal position (apart from the subject), it makes the presence of the anticipatory clitic (4) or of the comma (5) compulsory.
(4) Sono sicuro che molti di voi non lo avevano mai fatto, questo ragionamento I am sure that many of you do not it have never done, this reasoning (5) Interessante, in questo senso, la riflessione di Paul
Interesting, in this sense, Paul’s thought 4.4
Cleft sentences are typically composed of a main
clause without a subject introduced by the verb ‘to
be’ in different forms, followed by the cleft
constituent and by a subordinate clause introduced by
“che” (that), whose function can be of relative
pronoun (6) or relative conjunction. Sometimes, the
subordinate clause can be introduced by “a” (to)
+ a verb in the infinite form (7), if the subject is
the element to put into focus
Marked structures, such as the ones described
above, are very difficult to parse, since they belong
to non-standard Italian constructions. In order to
annotate them syntactically, we therefore need to
follow a semi-automatic approach, by analysing
them first with a dependency parser and then
manually correcting them. The selected marked
constructions from the IPRASE corpus were
processed with the TINT parsing module
Concerning dislocated sentences, the main issue with TINT is that it assigns to the pronoun the role of direct object and treats the dislocated element as the subject, as in the example shown in Figure 6. The sentence was manually corrected by marking the dislocated element with the dislocated relation and the pronoun of reprise with the core argument relation which it represents (obj or subj), as we can see in Fig. 1 and Fig. 2.
As previously mentioned, hanging topics differ from left dislocated sentences because the element to the left is not syntactically linked to the verb and there is no clitic reprise of the lexical element. Since there is a sort of isolation of the topicalized element, we choose to use the parataxis relation to link it to the head of the sentence, since parataxis is defined as a relation between a word and other elements, without any explicit coordination, subordination, or argument relation with the head word, which is usually the verb. An example is reported in Figure 3.
As we have seen above, the “che” (that) before a subordinate clause can be a relative pronoun introducing a relative clause or a relative conjunction, followed by a structure whose nature is controversial. A relative clause is an instance of clausal modifier acl, which takes the specific name of acl:relcl, where the noun can be omitted or substituted by a relative pronoun, relative conjunction, or an adverb. As regards cleft sentences, we choose to use the same relation in two different ways, in order to distinguish between the case in which the cleft sentence comprehends a relative clause or an unspecified subordinate clause. When the dislocated element is the subject or the direct object (substituted by a relative pronoun) we use the acl:relcl relation, selecting the role of “che” (see Fig. 4). Instead, if there is no dislocation of the subject or the object, we use the acl relation but we do not select the function of “che”. Indeed, “che” is treated as a mere introducer for the subordinate clause with the mark relation (Fig. 5).
Table 1 shows the eight types of constructions in MarkIT. Beside the four types of marked structures described above, we found several other structures coming from the erroneous identification of cleft sentences and right dislocated sentences in the original IPRASE corpus. “Presentative there” (i.e. “there” + verb to be + nominal element + that)1 and “pseudo-cleft” structures (“what” clause + verb to be) 2 were wrongly identified as cleft sentences, while the structures with a postverbal subject were originally labeled as right dislocated. Furthermore, we include in the “Other” category the structures which resulted challenging to tag for the annotators and which are not “presentative there” nor pseudo-cleft constructions. “Other” structures are namely those which usually present an explicit subject in the main clause and are erroneously identified as cleft, for example La capacita` di concepire un insieme di diritti e` una facolta` che distingue l’uomo dagli altri esseri viventi (EN: The ability to conceive a set of rights is a faculty that distinguishes humans from other living beings). “Other” structures include also passive clauses, which were originally tagged as right dislocated because of the postverbal position of the subject. Sentences in this last category are particularly challenging both for parsers and for human annotators, since they were wrongly classified even by IPRASE experts (i.e. high-school teachers) and have been assigned the correct label only after our revision. 6
As already mentioned in the Introduction, the lack
of marked structures in treebanks used to train
syntactic parsers may affect the system robustness,
since structures which are not represented in the
training data tend to be poorly analysed. In order
to measure the impact of our novel treebank on
the dependency analysis of marked structures, we
compare the performance of the parser included in
TINT, part of Stanford CoreNLP
2e.g. Cio` che voglio davvero e` che tu te ne vada (EN: What I really want is that you go) we first split our novel treebank into training, dev, and test, respectively 80%, 10%, and 10%, proportionally with respect to the categories listed in Table 1. When the number of examples is tiny, we include a minimum of two examples for each class in each split, therefore test and dev set contain two examples of hanging topic each, leaving the three sentences for the training set.
We then compare two models: the original neural transition-based parser model used by TINT, which is trained using ISDT, VIT, and ParTUT (see Section 2), and the model obtained by adding to the above training data also the training set of MarkIT. We choose not to include the other Italian datasets available from Universal Dependencies (such as the ones derived from Twitter) because of their particularly informal language, which is very different from MarkIT sentences taken from students’ essays. In both cases, we use the concatenation of the development sets of the four datasets as development set during the training phase. Following the standard evaluation used in dependency parsing, we compute unlabeled attachment score (UAS) and labeled attachment score (LAS) in the two tests.
ISDT+VIT+ParTut ISDT+VIT+ParTut+MarkIT UAS 82.53 82.74
LAS 76.62 77.41
Results in Table 2 show that on the one hand adding MarkIT to the training set improves the classification of marked structures, but on the other hand performance gain is limited. This may be due to the fact that, compared to the other treebanks (more than 23k sentences in total), the number of training instances coming from MarkIT is small (around 650 sentences). More generally, the presence of both marked and not marked sentences (the “Other” category) in the test set represents a challenge for parsers, since very similar constructions are labeled differently, see for example the presence of comma to mark right dislocated elements. Indeed, if the first model is tested only on sentences taken from ISDT+VIT, it achieves 84.47 UAS and 80.69 LAS. 7
MarkIT is released under CC BY 4.0 license,3 and can be downloaded from Github.4 The annotation of the treebank will be soon completed with all marked sentences in the essays dataset (see Section 8) and proposed for publication on the Universal Dependencies website.5 Since the treebank is still being extended with new sentences, it may be that the content of the last version available online exceeds the size of the resource described in this paper. 8
In this work we present MarkIT, a novel treebank composed of 800 sentences with syntactic annotation of marked structures. The resource covers seven types of marked sentences, plus around 200 sentences whose structure is not marked but that may be misleading both for parsers and for human annotators. The treebank is made available to the community and is meant to make dependency parsers more robust to the different syntactic structures present in Italian, in particular in the neo-standard variant. The work is still in progress, since we plan to add to the resource other sentences from the IPRASE corpus. Our goal is to include all marked sentences present in the essays, so to analyse also the distribution of the different sentence structures in this type of texts. 3https://bit.ly/cc-by-40-intl 4https://github.com/dhfbk/markit 5https://universaldependencies.org/