English. This paper1 presents the early stages of the development of a new treebank containing all of Dante Alighieri's Latin works. In particular, it describes the conversion of the original TEI-XML files to CoNLL-U, the creation of a gold standard, the process of training four annotators and the evaluation of the syntactic annotation in terms of inter-annotator agreement and LA, UAS and LAS. The aim is to release a new resource, in view of the celebrations for the 700th anniversary of Dante's death, which can support the development of the Vocabolario Dantesco.
The research field of treebanking (i. e. the
building of corpora enhanced with syntactic metadata)
has evolved substantially since the time when
the first large-scale syntactically annotated
corpus, the Penn Treebank for English, was
published between the late Eighties and the early
Nineties
The set of textual genres covered by currently
available treebanks is quite diverse. While the first
corpora were built mostly collecting texts from
news, the last decade has seen a substantial growth
of treebanks of different genres, including literary
texts, mostly written in ancient or historical
languages.2 The first available treebanks for ancient
languages were those for Ancient Greek and/or
Latin, namely the Index Thomisticus Treebank
(ITTB)
Among the many Latin texts that still lack syntactic annotation are those by Dante Alighieri (1265-1321). Given the importance of Dante in the history of Italian literature (and beyond) and in the light of the celebrations for the upcoming 700th anniversary of his death, we have started a project (called UDante) aimed at performing a UD-compliant syntactic annotation of all his Latin texts. The syntactic annotation of Dante’s opera omnia in Latin fits into the larger project of the Vocabolario Dantesco, which aspires to provide a detailed description of the entire (both Vulgar and 1Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2Examples among the UD treebanks are the Kyoto
treebank of Classical Chinese
The choice of using the UD formalism in the UDante project is motivated by a number of benefits implied by the inclusion of a new set of annotated texts into such a large collection of treebanks sharing the same annotation style, among others the use of the several tools developed by the UD community with the goal of querying, editing, visualizing and (automatically) processing the (meta)data of the treebanks.4 Particularly, a remarkable added value is the possibility to run common queries on the almost 100 different languages provided with at least one treebank in the current version of UD (v2:6, released on May 15th, 2020). Furthermore, adopting a well known and widely used data format (CoNLL-U) and part-of-speech tagset (UPOS) fosters the dissemination and use of a treebank of Dante’s Latin works in the community of computational linguistics, laying the foundation for a closer collaboration with that of Italian phylology and, more generally, with scholars in the Humanities, leading to a mutual benefit.
This paper presents the process behind the development of the manually annotated UD treebank containing the full collection of Latin works of Dante Alighieri. More specifically, we describe the conversion of the original TEI-XML files into the CoNLL-U format, we give details on the creation of a gold standard5 and we report on the training of four annotators with no previous knowledge of the UD formalism, providing an evaluation of their annotation work. 2
The texts of the Latin works by Dante Alighieri
(De monarchia, De vulgari eloquentia, Eclogues,
Epistulae and Quaestio de aqua et terra) are made
available by the DanteSearch corpus
5https://github.com/CIRCSE/UDante.
6https://dantesearch.dantenetwork.it
7The DanteSearch corpus also contains the Vulgar works
by Dante Alighieri that were used to develop a part-of-speech
converted into the CoNLL-U format8 and then
revised and syntactically annotated using
ConlluEditor
We implement an own developed script to
automatically convert the TEI-XML files of the
DanteSearch corpus into the CoNLL-U format. First
of all, the script analyses the XML tag structure
to identify the internal organisation of the text
(i. e. the division of the work in books, chapters
etc.): this information is stored in the MISC field
so as to facilitate the recoverability of the original
structure of the text starting from the CoNLL-U
file. Then, sentences are split and the tag <LM>,
which for each token contains morphological
information, is parsed in order to extract lemma, part
of speech and morphological traits, and to convert
the codes used in DanteSearch into UPOS tags9
(originally inspired by
8https://universaldependencies.org/ format.html
9https://universaldependencies.org/u/ pos/index.html 1 ! VerbClass=LatA10 ci ! Aspect=Imp|Mood=Sub|Tense=Past |VerbForm=Fin s ! Number=Sing 3 ! Person=3 Ad hoc rules are added to cover specific cases. For example, in DanteSearch the lemma prius ‘before’ is marked only with the grammatical category r: a rule converts r into the UPOS tag ADV and adds the morphological feature Degree=Cmp (comparative degree).
Annotators, in addition to annotating syntax from scratch, have to check the correctness of the automatic conversion and to manually modify or add items not covered by it. For example, annotators have to: (i) modify the grammatical category of population names (such as Veronenses ‘inhabitants of Verona’), which are marked as proper names in DanteSearch, contrary to UD recommendations, for which they should be considered as adjectives;11 (ii) check the ambiguous case of some pronouns in the neutral gender which in DanteSearch have mistakenly been marked as nominative instead of accusative (e. g. quod ‘that’), or viceversa; (iii) disambiguate the PronType feature in the case it has more than one value: this happens because the types of pronouns in DanteSearch cannot always be matched to only one PronType value (e. g. quis ‘who/any’, interrogative or indefinite). 2.2
An important part of the UDante project consists in training a group of annotators on the formalism of UD with the goal of providing them with adequate competences to pursue the complete syntactic annotation of Dante’s works. To this aim, for each of the two parts of our training (Section 2.3) a small number of sentences is singled out from all across Dante’s Latin texts to be used as a common (first part) or individual (second part) benchmark for the assessment of the annotators’ progress.
The first part of the training makes use of 33 sentences out of the total 1 662 (corresponding to 10We add VerbClass as a language-specific feature to encode traditional verb conjugations. The value LatA indicates the first conjugation, which has thematic vowel ‘a’.
11From https://universaldependencies. org/u/pos/ADJ.html: “ADJ is also used for ‘proper adjectives’ such as European (‘proper’ as in proper nouns, i. e., words that are derived from names but are adjectives rather than nouns).” 950 tokens out of 55 666). These sentences are not chosen to be consecutive, nor do they follow a particular order, but they are allocated into three different groups of increasing complexity, corresponding to the three distinct phases of this part of the training. The distribution is of 15 sentences in the first, introductory group, 5 in the second, intermediate group and 10 in the third, more challenging group. The first two groups are rather homogeneous and mostly draw from the De vulgari eloquentia , while in the third one each work is represented by 2 sentences, and the Eclogues are featured only here.
The differences in complexity can be understood in terms of number of nodes, depth, and breadth of the resulting syntactic trees. While a sentence of the first group has a median number of 11 nodes, a median depth of 4 layers and most nodes (not counting the root) tend to be at depth ca. 3, for the second group the same figures are respectively 42, 7 and ca. 4; for the third group they are 46:5, 7:5 and ca. 4:5. The difference is especially marked between the first group and the other two. Besides such quantitative factors, other more qualitative ones, like difficult syntactic structures, contribute to the overall complexity.
As for the second part of our training, which consists of only one phase, textual cohesion substitutes increasing complexity as the main selection criterion: as such, each annotator is assigned, from the work they will respectively take care of, the first 10 sentences which have not been previously annotated. The complexity of the single sentences is thus more variable in this phase, but still well represents the whole corpus. In particular, we use sentences 1-4 and 7-12 from book I of the De monarchia, sentences 4-5 and 7-14 from book I of the De vulgari eloquentia, sentences 1-10 from the first of Dante’s Eclogues, and 1-10 from Epistle VII of the Epistulae. The Quaestio de aqua et terra, of uncertain authorship, is not assigned to any annotator at the moment.
All the selected sentences are priorly syntactically annotated by hand by a UD expert applying language-specific features and subrelations developed for Latin, while lemmas, parts of speech and morphological traits are corrected or enhanced where needed with respect to the CoNLL-U conversion (see Section 2.1). This way, on the one hand a tripartite, scaled gold standard is created for common evaluation, while on the other hand each annotator will be tested on an individual gold standard in the last phase (Section 2.4). 2.3
The training of the annotators (all with no background in treebanking, but provided with a solid knowledge of Dante’s works and academic background in Latin and Italian philology) is split into two main parts: three “training proper” phases (phase 1 to 3), and one further “control” phase (phase 4).
The first part is meant to lay out a common training ground where the annotators can learn the specifics of the UD annotational scheme, and their progress is overseen and periodically reviewed to prompt improvements. In the first phase, the basics of the UD formalism are presented, and the annotators are required to manually annotate a first group of sentences as a way to evaluate their understanding of the UD principles.12 In the second and third phases, various aspects of the performed annotation get to be discussed and more complex syntactic structures are introduced, each time assigning new, more challenging sentences for an overall evaluation of the annotator’s performance and their inter-annotator agreement (see Section 2.2). At every step, the focus is primarily on the syntactic level, since most aspects regarding lemmatisation, parts of speech and morphology are already mostly dealt with during the conversion phase (Section 2.1).
In contrast to the first three phases, the last, control phase is carried out individually for each annotator on separate sets of sentences (see Section 2.2), as a prelude to their actual annotation work.
DEPRELs
Phase 1 80% 84%
Phase 2 83% 92%
Phase 3 79% 91% 12https://universaldependencies.org/ guidelines.html
13Our script for calcuating IAA on CoNLL-U files is
available at https://github.com/johnnymoretti/
CoNLL-U_Fleiss_Kappa.
ture of syntactic trees (EDGES) and the choice
of dependency relations (DEPRELs), whereas in
Table 2 the correctness of the annotator’s
analyses are compared for all phases to the gold
standard according to label accuracy (LA), unlabelled
attachment score (UAS) and labelled attachment
score (LAS)
For what concerns the IAA, the scores are rather good (always >75%) and, together with the equally positive scores in Table 2, show that the basic principles of UD have been uniformly acquired by all the annotators during the first part of the training, especially the UD scheme of dependency relations. In general, going from phase 1 to phase 3, we notice that all scores are quite stable, and we only observe a slight decrease of the EDGES score in phase 3 which mirrors the noteworthy complexity of the corresponding test sentences (see Section 2.2); the same general decrease shown in Table 2. However, this is more than compensated by generally markedly improved scores for all annotators in phase 4: taking into account the greater variability in sentence complexity, these data show that all annotators have reached a good degree of confidence both with UD’s syntactic formalism and with the specific annotation guidelines developed for Latin, which have been constantly updated during this project.
In particular, if we consider only the labelling
of single nodes (LA), we register a decided mean
improvement in the last phase (89% vs. 79:75%),
showing that the annotators have factually
improved their assessment of syntactic dependency
relations. A similar trend for IAA in the first three
phases and quite close scores in Table 3 point to
the fact that those cases where annotators disagree
are also those for which they have greater
uncertainties at the syntactic level; this leads us to
con14Metrics calculated with MaltEval
The decrease of UAS and LAS in the third phase, when compared to the good results of the second phase, has to be expected, as the sentences of phase 3 are chosen to be particularly challenging and in some cases present open problems of syntactic annotation.15 Despite this, the differences between phase 1 and phase 3 still show a rather stable quality of the annotation from this angle. Then again, the last control phase registers much 15See for example the open issue on how to deal with singular subjects and plural copula at https: //github.com/UniversalDependencies/docs/ issues/714. improved performances also for UAS and LAS, displaying the good level of assurance reached by the annotators at all levels of annotation. 3
In this paper, we describe the preliminary steps towards the creation of a UD-compliant treebank of the Latin works by Dante Alighieri. To this end, we create a gold standard and we train and evaluate the work of a team of four annotators by means of a tripartite common set of sentences of increasing complexity annotated by a UD expert, complemented by specific gold standards for each annotator in a final control phase before the actual annotation work takes place.
Besides supporting the objectives of the
Vocabolario Dantesco project, the development of a
treebank based on Dante’s Latin works also serves
a wider scope, i. e. the inclusion of these latters
into the LiLa Knowledge Base, which makes
distributed linguistic resources for Latin
interoperable through the Linked Data paradigm
The complete annotation of Dante’s Latin
works will provide the community with a new,
manually annotated dataset of higher quality than
any automatic system. Table 4 reports LAS scores
computed on the sentences of our gold standard
and processed with UDPipe using the UD v2:5
models for Latin
LAS
IT-TB 40.83%
24.93%
PROIEL 29.98%
The addition of Dante’s Latin works into the thriving and expanding UD project and the newly acquired possibility to interact with a large number of other Latin texts of different genres and time periods makes us hope for a breakthrough of the world of treebanking into the wider community of the Humanities, which today can benefit from accessing a huge set of connected textual (meta)data like never before.
This work is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme via the “LiLa: Linking Latin” project - Grant Agreement No. 769994. The authors also want to thank Prof. Mirko Tavoni (University of Pisa) and the annotators Martina De Laurentis, Federica Favero, Giulia Pedonese and Elena Vagnoni.