Join Together? Combining Data to Parse Italian Texts

Abstract

In this paper, we create and evaluate non-combined and combined models using Old and Contemporary Italian data to determine whether increasing the size of the training data with a combined model could improve parsing accuracy to facilitate manual annotation. We find that, despite the increased size of the training data, in-domain parsing performs better. Additionally, we discover that models trained on Old Italian data perform better on Contemporary Italian data than the reverse. We attempt to explain this result in terms of syntactic complexity, finding that Old Italian text exhibits higher sentence length and non-projectivity rate.

Keywords

• eol>Parsing
• Universal Dependencies
• Combined Model
• Old Italian
• Contemporary Italian
• Non-Projectivity

1. Introduction

and Old Italian data. The objective is to determine whether a combined model with an expanded training High-quality textual data (semi-)manually enhanced dataset performs better compared to non-combined modwith diferent layers of metalinguistic annotation are els (see [ 2 ] for Spanish language and [ 3 ] for Stanza comextremely valuable resources for conducting linguistic bined models). analysis. As for the syntactic layer of annotation, the de The paper is organised as follows: Section 2 provides a facto standard for dependency-based annotation is Uni- brief description of the Italian language, the syntactic reversal Dependencies (UD),1 an initiative that provides sources and the Italian data available; Section 3 details the machine-readable annotations for a wide variety of lan- data used for the experiments, presents the performances guages, including historical languages [ 1 ]. At the current of non-combined and combined models, and evaluates state of art,2 Contemporary Italian is well-represented their performances; Section 4 analyzes the syntactic comin UD, whereas Old Italian is only represented by one plexity of each test set (Old and Contemporary Italian) annotated text (a portion of the Divine Comedy of Dante to address accuracy diferences; and finally, Section 5 Alighieri). The creation of additional Old Italian anno- provides the conclusion. tated data is therefore advisable.

Since a fully manual annotation process is timeconsuming and requires significant efort, we aim to expe- 2. Talking about Italian dite it by using a parser that pre-parses the data, leaving the human annotator with only a manual revision task.

To address this, given the scarcity of Old Italian data, we create a combined parser using both Contemporary

Italian is a Romance language derived from Latin, and its

development is closely connected with the political, cultural and economic system of Italy during the Late Middle Ages [ 4, 5, 6, 7 ]. Even though the evolution and history of the Italian language "can be properly understood only within the wider context of the evolution of the Italian dialects" [5, p. 3], the dialect spoken in Florence (Tuscany) in the thirteenth century, known as Florentine, played a pivotal role in establishing the foundation of the Italian language. The pre-eminence of Florentine over other Italian dialects was established due to the importance and prestige of Florentine literature. Its widespread success contributed to the codification of Florentine as the lingua volgare in the sixteenth century, distinguishing it as the spoken Italian language in contrast to Latin, which was still used for written cultural discourse [8].

Even though Florentine (and, more generally, Tuscan dialects) is considered conservative in its linguistic evolution [5, p. 5], it is now widely recognized by most scholars Concerning Old Italian, the only treebank present in as distinct from Contemporary Italian [9, p. 8]. Among UD is Italian-Old [18], encompassing the Divine Comedy, the diferences between Contemporary Italian and Flo- a poetic text written by Dante Alighieri (1 265-1 321). Currentine (henceforth referred to as Old Italian),3 several rently, Italian-Old contains the first two Cantiche of the syntactic distinctions have been noted [ 10, 11 ]. These poem, namely Inferno and Purgatorio, amounting 80 694 include, among others, the position and order of clitics, tokens, 82 644 syntactic words5 and 2 402 sentences.6 the use of the marker sì ’that’ as a thematic marker, and The divergence in annotated data available for Condiferences in the use of compound tenses [ 11, p. 425-444]. temporary Italian (around 875K syntactic words) versus Old Italian (82K syntactic words) is considerable. 2.1. Syntactic resources Considering that i) treebanks are essential for expanding the sample of comparable data and that ii) the manual High-quality (semi-)manually annotated treebanks, i. e. annotation of data is an extremely time-consuming efcorpora with annotations on various linguistic levels,4 fort, the development of automatic parsers is crucial to are indispensable tools for in-depth analysis of the syn- expedite and assist the annotation process. tax (and morphology) of languages. Treebanks not only The shortage of gold-annotated data for Old Italian, facilitate faster, easier, and more precise querying of syn- compared to the large amount of data available for Contactic structures, but also aid in tracking the evolution of temporary Italian, led us to recognize the potential of syntactic patterns in languages through time [13]. testing combined models, i.e., models with a training set

Among the dependency treebanks, UD is a pivotal composed of both Old and Contemporary Italian data. initiative displaying cross-linguistically consistent treebanks for many languages [14]. As of the current version 2.14, UD includes 283 treebanks and 161 languages, en- 3. Combining Old Italian with compassing historical languages such as Latin (e.g. Index Contemporary Italian data Thomisticus Treebank, ITTB [15]), Old French (PROFITEROLE [16]) and Ancient Greek (e.g. PROIEL [17]), among others.

In Subsection 2.2, we describe UD treebanks of Italian language.

2.2. Italian data

Regarding Italian, UD includes 9 Contemporary Italian treebanks, spanning various genres, as reported in Table 1.

3We adhere to the definition of Salvi and Renzi [ 9], who use the term

Old Italian to refer to the language spoken in Florence during the 13th and 14th centuries. 4Treebanks usually provide information on sentence tokenization, word lemmatization, and both morphological and syntactic details. Syntactic analysis is mandatory in a treebank, and can be encoded in either dependency syntax or constituency syntax [12].

Considering the aforementioned divergence in data, we

create and evaluate the performance of a combined Contemporary-Old Italian model to understand whether joining datasets from diferent periods could improve parsing accuracy.

We train models using Stanza [19], a neural pipeline for natural language processing, with diferent training sets. Specifically, we train models based on Contemporary Italian data (henceforth CI), Old Italian data (henceforth OI), and a combination of Contemporary and Old Italian data (henceforth Combi).

In Subsection 3.1 we detail the selection and partitioning of the data. Subsection 3.2 outlines the creation of models and presents the resulting scores. Finally, Subsection 3.3 discusses the combined Contemporary-Old Italian model.

3.1. Selection and partitions of data

To build the model based on OI data, we use the only Old

Italian treebank available, Italian-Old.

Among all the Contemporary Italian UD treebanks, we select two treebanks, ISDT (Italian Stanford Dependency Treebank) and VIT (Venice Italian Treebank). We select ISDT [20], as it is the Italian treebank with the highest

5We use the term "syntactic words" and "tokens" following the

UD definition (see https://universaldependencies.org/u/overview/ tokenization.html). 6The numbers refer to UD version 2.14, see https: //universaldependencies.org/treebanks/it_old/index.html.

UD star ranking. This ranking, designed by the UD orga- Table 3

nizers, quantifies various qualities of the corpora, such as Evalutation metrics with VIT1 and OI models (where "->" their usability and the variety of genres they encompass. stands for "on").

Moreover, since Italian-Old is based on the poetry genre, VIT1 -> VIT1 OI -> OI VIT1 -> OI to minimize a potential genre gap (the influence of genre LAS 71.60 75.86 42.83 on parsing has been addressed in [21]), we also select UAS 77.70 82.24 56.13 VIT [22], that includes, albeit with a limited number of words, literary texts.7 We point out that, up to now, no CI treebanks contain poetry (see Table 1). Table 4

To avoid the CI data overwhelming the OI data due Evalutation metrics with ISDT and OI models. to their size disparity, we partition the CI data. The VIT ISDT -> ISDT OI -> OI ISDT -> OI treebank, consisting of 259.625 tokens, 280.153 syntactic LAS 88.55 75.86 51.62 words, and 10.087 sentences, allows us to partition the UAS 91.41 82.24 63.03 data into three parts, with each part closely matching the size of the Italian-Old dataset. Specifically, we divide the VIT dataset into three partitions of 34%, 33% and 33%, that pertain to the same textual domain as the test set respectively named VIT1, VIT2 and VIT3. Additionally, (VIT1 on VIT1, OI on OI, and ISDT on ISDT), yields higher we further divide each partition (VIT1, VIT2 and VIT3) performance than using out-of-domain data (ISDT on OI, into train, test, and dev sets with a split of 70%, 15%, and VIT1 on OI, and OI on VIT1 and ISDT). These results 15%, the same used in Italian-Old dataset. Unlike the align with literature on in-domain testing [25]. VIT treebank, the ISDT is not directly partitionable, as While analyzing the scores of out-of-domain parsing it counts 278 461 tokens, 298 375 syntactic words, and (ISDT on OI, VIT on OI, and OI on ISDT and VIT), we 14 167 sentences. Therefore, we shufled the data and notice that the model trained on OI data performs better extracted a total of 82 500 tokens (the same size of OI on CI data in both scenarios, whereas CI models yield data), which were then partitioned into train, dev, and lower scores when applied to OI text. The diferences in test sets with a ratio of 70%, 15%, and 15%, respectively. scores are approximately 20 points in favour of the OI

We report in Table 2 the partition of each datasets in model, specifically 25.7 (LAS) and 19.4 (UAS) compared train/dev/test. to VIT1, and 23.21 (LAS) and 17.9 (UAS) compared to ISDT.

We attempt to explain the outperformance of the OI

3.2. Creation of models and scores model in Section 4.

3.3. Joining model

With each partition (OI, VIT1, VIT2, VIT3 and ISDT), we

train 5 models using Stanza, with the training and dev sets, and we evaluate them on the respective test sets. Within the CI-VIT datasets, we retain only the model that performs best, namely VIT1.

We then use the model built on OI data to parse the CI test sets, and vice versa.

In Table 3 and Table 4, we report the scores of both Label Attachment Score (LAS) and Unlabel Attachment Score (UAS)8 of the OI model and the VIT1, and of the OI and the ISDT respectively.

For both VIT1 and ISDT scenarios, results show that using a model trained on in-domain data, namely data

To challenge the results obtained in 3.2, we build com

bined models with Stanza by merging OI data with CI data. Specifically, we create two models: CombiVIT, and CombiISDT. For each combined model, the test, dev, and train sets are created by merging the corresponding test, dev, and train sets of the VIT1 data and ISDT data with those of the OI data.

In Table 5, we report the UAS and LAS scores obtained.

We notice that in both scenarios the combined models perform better on CI data than on OI data, with the combined models outperforming by 13.74 (LAS) and 10.1 7The VIT treebank contains 10 000 words of literally genre [22, 23]. (UAS) for CI-VIT data and 12.58 (LAS) and 8.87 (UAS) for Refer also to the read.me to further details (see A). CI-ISDT data. 8Refer to [24] for an insight into the aforementioned metrics.

4. An insight to OI and CI data

To analyze the complexity of tree structures in each test set (CI-ISDT, CI-VIT, and OI), we calculate: • type-token ratio (TTR): the number of types divided by the number of tokens (excluding punctuation); • tree depth (Depth): the longest path from the root of an oriented a-cyclic graph (i.e, the syntactic tree) to a leaf; • lexical density (Lex. Den.): the number of content words, i.e. words that possess semantic content and contribute to the meaning of the sentence,9 divided by the total number of syntactic words (excluding punctuation marks); • sentence length (Length): the number of syntactic words (excluding punctuation marks) in each sentence.

9We select as content words all words belonging to the following

Universal parts of speech [26]: NOUN ’noun’, VERB ’verb’, ADJ ’ adjective’, ADV ’adverbs’, and PROPN ’proper noun’.

Among the measures described, the OI test does not

difer significantly from the CI values. The only measure in which the OI test difers from the CI tests is sentence length (Avg. Length): OI presents a higher average sentence length, surpassing the CI-ISDT average by 13 points and the CI-VIT average by 3.5.

Therefore, considering the parameters evaluated, only the sentence length could be considered to explain the possible overperformance of OI on CI data.

In Subsection 4.1, we evaluate another parameter that is related to the complexity of tree structure, namely non-projectivity (i.e., the number of structures where a head and its dependents form a discontinuous constituent). It has been demonstrated [27] that sentence length is interconnected with non-projectivity. Specifically, non-projective sentences exhibit greater sentence length compared to projective ones. By calculating non-projectivity, we aim to determine whether sentence length (which has been proven to be higher in OI test) and non-projectivity might indicate more complex structures in OI texts, thereby contributing to the overperformance of the OI model on CI data.

4.1. Non-projectivity

Non-projectivity arises when sentences exhibit non-local

dependencies. While constituency approaches may handle similar structures using empty categories and coindexation [28], dependency-based approaches result in discontinuous dependencies that lead to non-projectivity.

We illustrate an example of non-projectivity, showing the non-local dependency relation of the oblique (obl) dependency relation of the node fóri ’holes’, which is a dependent of the node piena ’full’. This relation causes non-projectivity with the node pietra ’rock’, which is dependent on the root (root) of the sentence vidi ’saw’ with an object (obj) dependency relation.

Inferno, xix, vv. 13–14: Io vidi per le coste (...) / piena la pietra livida di fóri ‘Along the sides (...), / I saw that livid rock was perforated’ obj

of the model, does not lead to better LAS and UAS acroot obl curacy scores. This confirms, in line with other studies advcl:pred [ 30, 31, 21, 32, 3 ], that having an in-domain training set obl:lmod is preferable.

case Additionally, we notice that the model trained on OI nsubj det det amod case data performs better on Contemporary Italian texts than the reverse (i.e. models trained on Contemporary data Io vidi per le coste piena la pietra livida di fóri on OI texts). To explain these results, we investigate The non-projectivity of syntactic dependency trees the syntactic complexity of each test set (OI, CI-ISDT, presents a challenging task for parsing in natural lan- and CI-VIT). Specifically we evaluate sentence length, guage processing [29], with non-projective structures tree depth, lexical density and the type-token ratio. We proving more dificult to parse. Concerning our task, we notice that the tests difer only in the sentence length. We investigate the number of non-projective structures in then proceed to calculate another parameter of syntactic each test set to determine whether the overperformance complexity, namely non-projectivity. of OI on CIdata may be associated with a higher preva- We discover that OI texts present a higher number lence of non-projective structures, thereby confirming of non-projective sentences. We hypothesize that the that having more non-projective structures in the train- high level of non-projectivity could be connected to the ing set is beneficial. genre of OI text, namely poetry. Thus far, the lack of

We calculate non-projectivity of the OI, CI-VIT, and UD treebanks for OI prose texts and for CI poetry texts CI-ISDT test sets. In Table 7 we report the total number have prevented us from investigating whether the high of edges, the number of non-projective edges, and the degree of non-projectivity observed in OI test (based on ratio of non-projectivity expressed in percentage of each the Italian-Old treebank) is characteristic of the poetry test set. genre or specific to OI. Such question will be left for further studies.

Table 7 Finally, we are currently working to increase the Non-projectivity of OI, CI-VIT, and CI-ISDT test sets. amount of manually annotated OI data, expanding both

OI CI-VIT CI-ISDT the range of authors and the genres of the texts considTotal edges 12 307 11 473 12 402 ered. This will allow us to evaluate the model’s perforNon-projective edges 176 24 7 mance both within and outside its domain (in terms of Non-projectivity ratio in % 1.43% 0.21% 0.06% authorship and text typology), as well as to assess its potential applicability to other OI texts.11

As shown in Table 7, OI shows a higher rate of nonprojectivity compared to CI texts. In particular, the nonprojectivity in OI is 7 times higher than in CI-VIT and 24 times higher than in CI-ISDT. The high rate of nonprojective structures in OI could be related to the genre of the text, i.e., poetry, which reflects a more creative use of language and frequently employs inversions.

5. Conclusion

In this paper, we create and evaluate non-combined and

combined models of Old Italian and Contemporary Italian data.10 In light of the scarcity of manually annotated Old Italian data compared to the richness of Contemporary Italian data, the aim of this work is to determine whether combining data to train a combined model could lead to better accuracy in parsing, thereby facilitating the process for human annotators.

We observe that combining Contemporary Italian and Old Italian data, even though it increases the data size 10Models are available for public use at https://github.com/CIRCSE/

Old_Italian_Model. [7] G. Rohlfs, Grammatica storica della lingua italiana EVALITA 2014: 9-11 December 2014, Pisa, Pisa Unie dei suoi dialetti, Torino : Einaudi, 1968. versity Press, 2014, pp. 1–8. [8] M. Vitale, La questione della lingua, Palermo : [21] F. Mambrini, M. C. Passarotti, Will a parser overtake

Palumbo, 1978. achilles? first experiments on parsing the ancient [9] G. Salvi, L. Renzi (Eds.), Grammatica dell’italiano greek dependency treebank, in: Proceedings of antico, il Mulino, Bologna, Italy, 2010. URL: https: the Eleventh International Workshop on Treebanks //www.mulino.it/isbn/9788815134585. and Linguistic Theories (TLT11). 30 November–1 [10] M. Dardano, Sintassi dell’italiano antico. La prosa December 2012, Lisbon, Portugal, Edições Colibri, del Duecento e del Trecento, volume 1, Carocci, 2012, pp. 133–144.

2012. [22] L. Alfieri, F. Tamburini, (almost) automatic conver[ 11 ] M. Dardano, G. Frenguelli, SintAnt. La sintassi sion of the venice italian treebank into the merged dell’italiano antico, Roma, Aracne, 2004. italian dependency treebank format., in: CEUR [12] A. Abeillé, Treebanks: Building and using parsed WORKSHOP PROCEEDINGS, volume 1749, Accorpora, volume 20, Springer Science & Business cademia University Press, 2016, pp. 19–23.

Media, 2003. [23] R. Delmonte, A. Bristot, S. Tonelli, Vit-venice italian [13] A. Taylor, Treebanks in historical syntax, Annual treebank: Syntactic and quantitative features., in:

Review of Linguistics 6 (2020) 195–212. Sixth International Workshop on Treebanks and [14] J. Nivre, M.-C. de Marnefe, F. Ginter, J. Hajič, C. D. Linguistic Theories, volume 1, Northern European Manning, S. Pyysalo, S. Schuster, F. Tyers, D. Ze- Association for Language Technol, 2007, pp. 43–54. man, Universal Dependencies v2: An evergrowing [24] S. Buchholz, E. Marsi, CoNLL-X Shared Task on multilingual treebank collection, in: N. Calzolari, Multilingual Dependency Parsing, in: L. Màrquez, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. De- D. Klein (Eds.), Proceedings of the Tenth Conferclerck, S. Goggi, H. Isahara, B. Maegaard, J. Mar- ence on Computational Natural Language Learning iani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (CoNLL-X), Association for Computational Linguis(Eds.), Proceedings of the Twelfth Language Re- tics (acl), New York City, nj, usa, 2006, pp. 149–164. sources and Evaluation Conference, European Lan- URL: https://aclanthology.org/W06-2920. guage Resources Association, Marseille, France, [25] M. Khan, M. Dickinson, S. Kübler, Towards domain 2020, pp. 4034–4043. URL: https://aclanthology.org/ adaptation for parsing web data, in: Proceedings 2020.lrec-1.497. of the International Conference Recent Advances [15] M. Passarotti, The project of the index thomisticus in Natural Language Processing RANLP 2013, 2013, treebank, Digital classical philology. Ancient Greek pp. 357–364. and Latin in the digital revolution 10 (2019) 299–320. [26] S. Petrov, D. Das, R. McDonald, A universal partURL: https://doi.org/10.1515/9783110599572-017. of-speech tagset, arXiv preprint arXiv:1104.2086 [16] S. Prévost, L. Grobol, M. Dehouck, A. Lavren- (2011).

tiev, S. Heiden, Profiterole: un corpus morpho- [27] J. Macutek, R. Cech, J. Milicka, Length of nonsyntaxique et syntaxique de français médiéval, Cor- projective sentences: A pilot study using a Czech pus (2023). UD treebank, in: X. Chen, R. Ferrer-i Cancho (Eds.), [17] D. T. Haug, M. Jøhndal, Creating a parallel treebank Proceedings of the First Workshop on Quantitative of the old indo-european bible translations, in: Pro- Syntax (Quasy, SyntaxFest 2019), Association for ceedings of the second workshop on language tech- Computational Linguistics, Paris, France, 2019, pp. nology for cultural heritage data (LaTeCH 2008), 110–117. URL: https://aclanthology.org/W19-7913. 2008, pp. 27–34. doi:10.18653/v1/W19-7913. [18] C. Corbetta, M. Passarotti, F. M. Cecchini, [28] J. Nivre, Constraints on non-projective dependency G. Moretti, Highway to Hell. Towards a Univer- parsing, in: D. McCarthy, S. Wintner (Eds.), 11th sal Dependencies Treebank for Dante Alighieri’s Conference of the European Chapter of the AssoComedy., in: CLiC-it, 2023. ciation for Computational Linguistics, Association [19] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Man- for Computational Linguistics, Trento, Italy, 2006, ning, Stanza: A python natural language processing pp. 73–80. URL: https://aclanthology.org/E06-1010. toolkit for many human languages, arXiv preprint [29] J. Nivre, J. Nilsson, Pseudo-projective dependency arXiv:2003.07082 (2020). parsing, in: K. Knight, H. T. Ng, K. Oflazer [20] C. Bosco, F. Dell’Orletta, S. Montemagni, M. San- (Eds.), Proceedings of the 43rd Annual Meeting guinetti, M. Simi, The evalita 2014 dependency of the Association for Computational Linguistics parsing task, in: Proceedings of the First Italian (ACL’05), Association for Computational LinguisConference on Computational Linguistics CLiC-it tics, Ann Arbor, Michigan, 2005, pp. 99–106. URL: 2014 & and of the Fourth International Workshop https://aclanthology.org/P05-1013. doi:10.3115/

A. Online Resources • Italian-Old, • Italian-ISDT, • Italian-VIT, • ITTB, • PROFITEROLE, • PROIEL, • Stanza, • Old-Italian-Model.

References

[1] M.-C. de Marnefe , C. D. Manning , J. Nivre , D. Zeman , Universal Dependencies, Computational Linguistics 47 ( 2021 ) 255 - 308 . URL: https://aclanthology.org/ 2021 .cl- 2 .11. doi: 10 .1162/coli_a_ 00402 .

[2] F. Sánchez-León , Combining diferent parsers and datasets for capitel ud parsing , in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ), 2020 .

[3] A. Zeldes , N. Schneider , Are ud treebanks getting more consistent? a report card for english ud , 2023 . URL: https://arxiv.org/abs/2302.00636. arXiv: 2302 . 00636 .

[4] B. Migliorini , Storia della lingua italiana , Bompiani , 2019 .

[5] M. Maiden , Linguistic History of Italian, A , Routledge, 2014 .

[6] A. Vàrvaro , La parola nel tempo . Lingua, società e storia, Bologna : Il Mulino, 1984 .

11For an overview of Old Italian resources , refer to [ 18 ]. 1219840 .1219853.

[30] M. Khan , M. Dickinson , S. Kuebler , Does size matter? text and grammar revision for parsing social media data , in: C. Danescu-Niculescu-Mizil , A. Farzindar , M. Gamon , D. Inkpen , M. Nagarajan (Eds.), Proceedings of the Workshop on Language Analysis in Social Media , Association for Computational Linguistics, Atlanta, Georgia, 2013 , pp. 1 - 10 . URL: https://aclanthology.org/W13-1101.

[31] M. Khan , M. Dickinson , S. Kübler , Towards domain adaptation for parsing web data , in: R. Mitkov, G. Angelova, K. Bontcheva (Eds.), Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013 , INCOMA Ltd . Shoumen, BULGARIA , Hissar, Bulgaria, 2013 , pp. 357 - 364 . URL: https://aclanthology.org/ R13-1046.

[32] C. Corbetta , M. Passarotti , G. Moretti, The Rise and Fall of Dependency Parsing in Dante Alighieri's Divine Comedy , in: R. Sprugnoli, M. Passarotti (Eds.), Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, ELRA and ICCL , Torino , Italia, 2024 , pp. 50 - 56 . URL: https://aclanthology.org/ 2024 .lt4hala- 1 .7.