This paper presents a straightforward application of Large Language Models (LLMs) for Dependency Parsing. The parsing process is approached as a sequence-to-sequence task, where a language model takes a sentence as input and generates a bracketed form, allowing for the deterministic derivation of the dependency graph. The experimental evaluation explores the feasibility of utilizing LLMs for this purpose, while also assessing the process's sustainability with modest parameter sizes (training on a single GPU with limited resources) and investigating the impact of incorporating multilingual data during training. The results demonstrate that an end-to-end dependency parsing process can indeed be formulated using a task-agnostic architecture.
data. It employs a Bi-LSTM architecture fed with endto-end, character-level, pre-trained, and contextualized Dependency parsing is a crucial component of natural embeddings. The model was trained on an extensive language processing that plays a significant role in cap- dataset of over a million sentences across diferent lanturing the syntactic intricacies within sentences [1]. The guages to capture cross-lingual relations efectively. The primary objective of dependency parsing is to establish system was later extended [8] as UDPipe+ by incorporatdependency relations among words. This allows humans ing multilingual BERT [9] in its token representations. to understand how words are connected and how they However, while these methods have achieved state-ofdepend on one another in the sentence’s structure [2]. the-art results in various languages, they are tailored Such understanding is instrumental in a wide range of specifically for the structure prediction problem based applications, including semantic interpretation, machine on ad hoc methods. translation, relation extraction, and various other linguis- More recently, models based on the Transformer artic tasks. chitecture [10] have gained popularity for their ability
One notable parsing technique is the shift-reduce to perform classification, regression, and rewriting tasks. method, as exemplified in [ 3, 4]. This parser processes These models operate on a sequence and they output ansentences from left to right, word by word while main- other sequence. For instance, the work in [11] introduced taining a bufer for words that are yet to be fully pro- an end-to-end seq2seq method for dependency parsing, cessed. Other approaches have been proposed, based on where the model directly predicts the relative position of machine learning techniques, such as the biafine neural the head for each word in the sentence. It also utilized a networks, as in [5]. These networks, based on Bi-LSTMs, beam search decoder with tree constraints and sub-root have proven efective in capturing complex dependencies decomposition to improve the results. Moreover, in [12] between words. Another intriguing parsing approach is the authors have experimented with a multi-task, multiUDPipe [6], focused on parsing with the Universal De- lingual version of BERT [9]. This model was pre-trained pendency Framework [7]. UDPipe stands out because it on 104 languages and could predict not only dependency performs dependency parsing and other essential tasks parsing trees but also lemmas, part-of-speech tags, and like tokenization, morphological analysis, part-of-speech more for each word in an input sentence. One notable tagging, and lemmatization for multiple languages. UD- Transformer-based architecture is the LLaMA [13] founPipe performs all these tasks without relying on external dational models. LLaMA is a large model with billions of parameters that generates output sequences in an autoCLiC-it 2023: 9th Italian Conference on Computational Linguistics, regressive manner based on the input and previously Nov 30 — Dec 02, 2023, Venice, Italy generated output tokens. It has been recently applied in $ hromei@ing.uniroma2.it (C. D. Hromei); croce@info.uniroma2.it [14] to a variety of linguistic tasks by instruction-tuning (D. Croce); basili@info.uniroma2.it (R. Basili) a monolithic architecture to solve them all. (D.0C0r0o9c-e0)0;0000-0802-0040-0520-2132(1C3.-0D8.2H8 r(oRm.Beia)s;i0li0)00-0001-9111-1950 In this work, we raise a crucial question about the CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ©CCo2Em02mU3oCRnospLWyicreiognhsrtekfAostrthrtihboiusptpioanpPe4rr.0obIynctietesrenaaduttiihononragsl.s(CUCs(eCBpYEer4mU.0i)tR.te-d WundSe.roCrregat)ive applicability of models like LLaMA for predicting treelike structures. Specifically, we seek to explore whether such models can be used to define an end-to-end parsing process without relying on architecture choices that are task-dependent. We envision a system that, given an input sentence, predicts an output sequence in a parenthetical form as in [15]. This output sequence allows for the reconstruction of the dependency tree of the original sentence. The experimental results on three Italian treebanks demonstrate that such an approach is not only feasible but also capable of achieving results comparable to the state-of-the-art while requiring minimal training resources, such as training on a single GPU with modest memory.
In the rest of the paper, Section 2 described the proposed approach, Section 3 presents and discusses the experimental evaluations, while Section 4 derives the conclusions. parameters for downstream tasks while avoiding additional inference latency. Additionally, [23] introduces Quantized-LoRA, an optimization that further reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pre-trained language model into LoRA.
One of the challenges in modeling tasks with LLMs (Large Language Models) is that these models take sequences as input and produce sequences as output. For instance, consider this (Italian) sentence2: “Tutti gli esseri umani hanno capacità non sfruttate, non utilizzate.”
(
Figure (
Model By converting the sentence into an arboreal structure,
The architecture of Transformers [10] has revolutionized a Dependency Tree (Figure (2)) can be obtained. This tree
Natural Language Processing (NLP), achieving increas- illustrates the hierarchical structure of the sentence, with
ingly higher results. In fact, the Architecture can be the main verb (hanno) serving as the ROOT and all other
divided into three major families: Encoder-only models words depending on it. Non-terminal nodes in the tree
like BERT [9], RoBERTa [16], and DeBERTa [17] that are represent the labels of the dependencies from the
Depenresponsible for encoding input sequences and generating dency graph in Figure (
However, training and fine-tuning such models require heavy computational resources, i.e. countless GPUs. Re- [sfruttate] [CONJ [PUNCT [,]] cently, a method for eficient training has been intro- [ADVMOD [non]] [utilizzate]]]] duced, called Low-Rank Adaptation (LoRA [22]). LoRA [PUNCT [.]]] involves freezing the weights of the pre-trained model Finally, it is worth noting that the process is reversible, and introducing trainable rank decomposition matrices meaning the DP tree can be constructed from the linguisinto each layer of the Transformer architecture. This tic representation and vice versa. This ability facilitates approach significantly limits the number of trainable
2In English: “All human beings have untapped, unused capacities.”.
DET:PREDET
DET
AMOD
NSUBJ OBJ
ADVMOD
CONJ
PUNCT
ADVMOD
Tutti
gli
esseri
umani
hanno
capacità
sfruttate
,
non
utilizzate
.
,
non
various computational tasks involving language model- in parenthetical form, showing the efectiveness of the
ing and analysis and, more importantly, allows the usage approach in parsing Italian sentences. The input/output
of such LLMs for predicting the Dependency Tree of pairs used for LLaMA consist of sentences (as in Eq. (
In fact, several studies, such as [21], have highlighted prompted with a simple instruction (“Parse this sentence.”) the impressive few-shot learning capabilities of Lan- to guide it in generating the desired output. guage Models. These models can generalize information from only a limited number of input examples provided through prompting, producing coherent and accurate 3. Experimental Evaluation output. In this paper, we explore the application of the LLaMA 7 and 13 foundational models to Italian sen- The training of the model utilized PyTorch and the Hugtences with the goal of extracting DP Trees in parenthet- gingface library, along with the Peft packages, to imical form. LLaMA is one of the Large Language Models plement the Q-LoRA technique. The LLaMA models that operates by taking a sequence of words as input underwent 3 epochs of training with a learning rate of and predicting the next word to generate text recursively. 3 · 10− 4 and a batch size of 32. To optimize the model’s The model is built on the popular Transformers archi- performance, a linear scheduler with warmup was utitecture [10], with several key diferences. Firstly, to en- lized, using a warmup ratio of 0.1. The training process hance training stability, the RMSNorm function [24] is employed Q-LoRA 4-bit to refine the transformer’s applied before each layer for normalization. Secondly, the and modules, as in [23]. The LoRA matrices had a SwiGLU activation function [25] is utilized. Lastly, Ro- matrix rank of 8 and a parameter of 16. The traintary Embeddings (RoPE) [26] replace absolute positional ing was performed on a single Tesla T4 GPU with 16GB embeddings. The combination of these modifications, of memory. This is particularly interesting as we have along with the vast size of parameters and training data implied the two smallest available models, i.e. with 7 and (trillions of tokens), makes LLaMA a highly promising 13-billion-parameters, to demonstrate that it can be used model for various natural language processing tasks. even on standard architectures. It doesn’t rule out the
It’s important to stress that LLaMA operates as possibility of evaluating larger models like LLaMA 65, a sequence-to-sequence model, following an auto- but currently, they require such computational power regressive approach, where text is fed as input, and text that would limit their applicability in real-world scenaris generated as output. This allows the model to capture ios, due to their extensive training duration and memory complex linguistic structures and dependencies in the requirements. input sentences and produce corresponding DP Trees We used the Universal Dependency Parsing dataset and, to align with [6], we utilized version 2.3 of the dataset and focused on the same subsets of examples in the Italian language, i.e., three Treebanks: IT-ISDT obtained by conversion from ISDT (Italian Stanford Dependency Treebank), IT-ParTUT a conversion of a multilingual parallel treebank and consisting of a variety of text genres, and IT-PoSTWITA a collection of Italian tweets. The neural architecture was trained on the union of these three datasets, comprising 20, 270 training examples, 1, 391 development examples, and 1, 309 test sentences. This initial set of experiments, referred to as , involved training and evaluating the neural architecture using examples from the same language. ing material. In this case, while the test data remained unchanged, being in Italian, the training dataset consisted of 101,284 examples. The development dataset was also kept in Italian for comparison purposes. To evaluate the performance of the LLaMA model, we have selected two key metrics: UAS (Unlabeled Attachment Score), and LAS (Labeled Attachment Score). UAS assesses the accuracy of the model’s dependency tree structure by verifying if the correct head and dependency arcs are generated.
On the other hand, LAS provides a more comprehensive evaluation by measuring the accuracy of the dependency labels assigned to each arc in the dependency tree.
Objectives. In this experimentation, our primary objec- ported in Table 1, and 2 for the UAS and LAS metrics, retive is to address three crucial experimental questions spectively. Here we compare our approach with UDPipe related to natural language processing using the LLaMA [6] and the subsequent extension UDPipe+ [8], as these model. First and foremost, we seek to determine if this are the state-of-the-art systems for the Italian Treebanks. process efectively works and if LLaMA is capable of Notice that our LLaMA-based models fail in 0.5-1% of achieving state-of-the-art performance. Secondly, we aim the times to correctly rewrite the whole sentence, i.e. to explore the potential advantages of employing larger they sometimes skip a word and do not produce any laarchitectures (from 7 to 13 parameters): traditional bel, diferently from UDPipe which covers 100% of the large models have been criticized for their considerable words in a sentence. Please note that, for the purpose of computational and environmental costs. By investigat- comparison, we have applied gold-standard tokenization ing the use of bigger architectures in the LLaMA model, in these initial experiments, as done in [8]. we strive to determine if advancements in performance Our models are divided into two categories: ita, which can be achieved without compromising sustainability. is trained exclusively on Italian data, and multi, trained Furthermore, we want to investigate the significance of using material from other languages. From Tables 1 and multilingual data in enhancing the LLaMA model’s per- 2, it is evident that the 7 models fall short of achieving formance. We draw inspiration from previous works such state-of-the-art performance; however, they only slightly as UDPipe [6], which have demonstrated the positive im- lag behind UDPipe. Advancing to the 13 models shows pact of multilingual training on various natural language a modest performance increase, but considering their processing tasks. As the LLaMA model supports multi- larger size, their practicality may be limited. Furtherple languages, we aim to analyze whether incorporating more, we observe a performance boost when incorporatmultilingual data leads to improved overall performance ing multilingual data during fine-tuning, as the LLaMA on the Dependency Parsing task. models support multiple languages. By enriching the
In a second set of experiments (referred to as ), Italian training set with data from other languages, we efwe trained the system by incorporating data from English, fectively leverage valuable relations and structures from French, and Spanish datasets. Specifically, we added diverse linguistic sources. training examples from the English-EWT, English-GUM, Moreover, since the dependency parsing process of a English-LinES, English-ParTUT, French-GSD, French- sentence is a global property of the entire sentence, we ParTUT, French-Sequoia, French-Spoken, French-Old, have also investigated more complex decoding processes, Spanish-AnCora, and Spanish-GSD datasets to the train- such as adopting deterministic beam search [27] during the decoding process. In a nutshell, beam search involves Table 4 exploring up to possible sequences during decoding LAS computed on the end-to-end process, where the tokenizauntil the completion of individual generations. This way, tion is performed by the model. we believe that the generated sequence is not penalized Model IT-ISDT IT-ParTUT IT-PoSTWITA by locally optimal choices for the decoding process but 7B_ita_b1 87.27% 87.77% 76.26% rather optimized at the sentence level. This enables us to 7B_ita_b4 88.51% 88.45% 77.13% enhance decoding strategies by adopting a larger beam 13B_ita_b1 89.81% 90.52% 77.90% search size (b43) instead of relying solely on greedy search 713BB_m_itual_tib_4b1 8990..9130%% 9911..3024%% 7788..0749%% (b1). As a result, the adoption of beam search systemati- 7B_multi_b4 90.57% 91.62% 78.78% cally improves performances for both the 7 and 13 parameter models. The results indicate that the model excels in generating the syntactic structure of sentences, 4. Conclusions with comparable performance to UDPipe. Remarkably, the neural model remains task-agnostic. Additionally, the In this paper, we investigate the application of recent study suggests that incorporating data from other lan- popular LLMs, specifically the LLaMA foundational modguages, if possible, is more beneficial than merely scaling els, to address the Dependency Parsing problem. Our up to larger architectures. exploration aimed to answer three key questions: Can
Notice that these evaluations used the Gold Standard we utilize LLaMA in a sequence-to-sequence
scetokenization of the sentences available from the Tree- nario to rewrite Dependency Parsing Trees from
banks, both during training and inference. For this rea- input sentences? The answer is afirmative. Although
son, we trained and evaluated the same models using LLaMA did not achieve a new state-of-the-art
perforthe “raw” sentences, without any tokenization and re- mance, our results demonstrate that the adopted model
quiring the model to produce the resulting DP tree. For and approach are competitive with UDPipe, the current
instance, the sentence from Eq (
Model 7B_ita_b1 7B_ita_b4 13B_ita_b1 13B_ita_b4 7B_multi_b1 7B_multi_b4
IT-ISDT 90.13% 91.27% 92.31% 92.57% 92.65% 93.22%
IT-ParTUT 91.19% 91.96% 93.80% 94.34% 94.12% 94.31%
IT-PoSTWITA 81.36% 82.18% 82.90% 83.59% 83.31% 83.99%
parameter ‘b’, but none of them yielded significant performance improvements except for when ‘b’ was set to 4.
We would like to thank the “Istituto di Analisi dei Sistemi ed Informatica - Antonio Ruberti” (IASI) for supporting the experimentations through access to dedicated computing resources. Claudiu Daniel Hromei is a Ph.D. student enrolled in the National Ph.D. in Artificial Intelligence, XXXVII cycle, course on Health and life sciences, organized by the Università Campus Bio-Medico di Roma. We acknowledge financial support from the PNRR MUR project PE0000013-FAIR.