The objective of this work is to assess the performance of Large Language Models (LLMs) on the task of Word Sense Disambiguation (WSD) for Latin.We evaluate state-of-the-art LLMs-including GPT-4o-mini and LLaMA variants-in both zero-shot and fine-tuned settings, using a dataset derived from the SemEval-2020 Latin Lexical Semantic Change task. Our study aims to determine whether instruction tuning and task-specific fine-tuning can significantly improve the models' ability to disambiguate Latin word senses. Results show that while LLMs demonstrate a non-trivial baseline ability in zero-shot settings, fine-tuning - particularly instruction-based - provides improvements in accuracy and F 1 scores. These findings highlight the potential of LLMs when applied to low-resourced historical languages.
CLiC-it 2025: Eleventh Italian Conference on Computational Linguistics, September 24 — 26, 2025, Cagliari, Italy * Corresponding author. † These authors contributed equally. $ e.ghizzota@phd.uniba.it (E. Ghizzota); pierpaolo.basile@uniba.it (P. Basile); lucia.siciliani@uniba.it (L. Siciliani); giovanni.semeraro@uniba.it (G. Semeraro)
0000-0002-0751-3891 (E. Ghizzota); 0000-0002-0545-1105 (P. Basile); 0000-0001-7116-9338 (L. Siciliani); 0000-0002-9421-8566 (G. Semeraro) © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License 1https://www.github.com/dbamman/latin-bert
Attribution 4.0 International (CC BY 4.0). tionised the landscape by providing vast corpora and ex- period to the 21st century. It achieves state-of-the-art tensive knowledge graphs extracted from online sources, performance in Latin part-of-speech tagging across all thereby amplifying the capabilities of both supervised Universal Dependency datasets. To capture the full range and knowledge-based methods. The introduction of of linguistic variation, the model was trained on multiple transformer-based architectures [22] marked a signifi- corpora, including the Corpus Thomisticum, the Internet cant turning point. These models use dense vector rep- Archive, the Latin Library, Patrologia Latina, Perseus, and resentations to capture semantic meaning in context, re- the Latin Wikipedia. Latin BERT uses Latin-specific sensulting in further advancements in disambiguation tech- tence and word tokenizers from the Classical Language niques. A significant development in this domain is the Toolkit, resulting in a vocabulary of 32,895 subword units. rise of Large Language Models (LLMs), which are built To assess Latin BERT performance in the WSD task, the upon the Transformer architecture and trained on exten- authors reformulated it into a binary classification task sive text corpora. LLMs exhibit proficiency in a myriad and created an ad hoc dataset of Latin sense examples of tasks in zero-shot or few-shot contexts, ruling out extracted from the Lewis and Short Latin Dictionary [23]. the necessity of task-specific training data. This implies In order to be selected, headwords must have at least an inherent capacity for semantic understanding within two distinct senses – typographically denoted by “I.” and these models. Nonetheless, LLMs can also be fine-tuned “II.” – supported by at least 10 sentences each, and longer on particular tasks by utilising tailored training data, en- than five words. For the task, only the two major senses hancing their performance in specific applications. of a headword were retained; the final dataset consists of
Considering these premises, the intent of this work is 8,354 examples for 201 dictionary headwords. For each to assess how state-of-the-art LLMs perform on under- headword, an instance of Latin BERT was fine-tuned on represented languages like Latin through the lens of a 80% of the examples. The number of training instances long-standing task in NLP like WSD. In particular, our in- per headword ranges from 16 (8 per sense) to 192 (96 per vestigation has two objectives. First, we want to test mod- sense); 59% of headwords have 24 or fewer training exels out-of-the-box ability to disambiguate Latin senses in amples. Latin BERT achieves 75.4% accuracy, compared a zero-shot setting. In this way, we aim to first establish to the 67.3% of a bidirectional LSTM with static word how well the models inherent multilingual knowledge embeddings. These results show that, even with few performs in accurate sense prediction. Next, we also per- training examples, Latin BERT was able to disambiguate form task-specific fine-tuning, which enables us to adapt senses. both standard and instruction versions of LLMs. The aim A few years later, [24] fine-tuned Latin BERT on a is to gauge the gain obtained with this additional training portion of sense representations in the Thesaurus Linstep. guae Latinae2 (TLL). The TLL is the first comprehensive
The paper is structured as follows: Section 2 provides dictionary of ancient Latin usage up to 600 AD, ofering an overview of works related to solving the WSD task a comprehensive, documented overview of every Latin with LLMs; Section 3 introduces the corpus of choice word’s history, including meanings and constructions, for this study, while 4 illustrates the methodology. Sec- etymology, inflexion peculiarities, spelling, and prosody, tion 5 describes the experimental setting and discusses as well as comments from ancient sources on the word itthe results and the limitations of the proposed strategy, self. The ongoing TLL project begun in 1894 and has been while Section 6 summarises the takeaway messages of regularly updated since; currently, it contains lemmata this paper and suggests some future works. from a to resurge¯sco, and it is estimated to contain approximately 56,000 entries. Inspired by the WSD dataset created by Bamman and Burns for Latin BERT, the au2. Related Work thors requested data for the same lemmata from TLL, obtaining 25,227 quotes for 40 lemmata. The new dataset 2.1. Latin Word Sense Disambiguation leads to a performance gain, with the Mean Macro F1 Currently, solving the WSD task for Latin using language increasing from .695 to .794. models remains an unexplored strategy, with very few Although both [17] and [24] achieved promising reworks investigating this line of research in recent years. sults, Latin is still an under-represented language for The idea of using WSD for measuring the ability of lan- which very few annotated resources are available, when guage models to deal with Latin is supported by the work compared to English. [25] proposes a language pivoting proposed by [17] in which Latin BERT is tested on the framework for Latin. Language pivoting, borrowed from sense disambiguation task. Machine Translation [26], consists of propagating anno
Latin BERT is a contextual language model tailored tations from high-resource languages to lower-resource for Latin, trained on a corpus of 642.7 million words ones. Starting from the 40 lemmata manually annotated drawn from diverse sources ranging from the Classical
for SemEval-2020 [13], the authors extract an aligned currence within a sentence, the LLM must generate the
Latin-English dataset in which these lemmata occur. To appropriate definition; and ( ii) given a word occurrence
thi
Over the years, LLMs have consistently demonstrated 3. Dataset
their ability to perform various tasks in a zero- or
fewshot setting with minimal or no specific training data,
suggesting an intrinsic capability of LLMs to grasp the 3.1. Resource
semantics behind language [28, 29]. The dataset of choice is the Latin annotated dataset for
[30] demonstrates that BERT-like models are capable of the Unsupervised Lexical Semantic Change Detection
efectively diferentiating between various word senses, (L
For each target word, its primary sense definitions annotation statistics are summarised in Table 2 below. were taken from the Latin portion of the Logeion On- We take full advantage of the annotations in the dataset line Dictionary9, which includes Lewis and Short’s Latin by creating a separate prompt for each of the judgments English Lexicon [23], Lewis’s Elementary Latin Dictio- assigned to each of the proposed senses for a single sennary [41], and Du Fresne Du Cagne’s Glossarium mediae tence. For example, if the annotator marked virtute as “4 et infimae latinitatis [42]. Depending on the cases, the - Identical” for the “manliness, courage, virtue, strength” sense inventory was simplified, or the definitions were and “1 - Unrelated” for the sense “virtue, personified as a shortened, while maintaining the principal distinction deity”, two separate prompts are created, each structured between senses. Finally, for each target word 60 passages as shown in Listings 1 and 2. sample sentences were extracted, 30 from 1 and 30 2 Listing 1: Prompt generated by each sense annotation for respectively, for a total of 2,398 passages. regression task.
The lack of native Latin speakers adds a further layer of complexity to the sense annotation process. 10 an- Instruction: Given the target word ‘‘virtute’’ notators with a high-level knowledge of Latin were re- ainsdentchleosseendtbeynctehien[TiAnRpGuEtTw]hteraeg,tahnedwtohred cruited, ranging from undergraduate students to senior following meaning ‘‘virtue, personified as researchers. Annotators – only one per target word – a deity’’, assign a score between 0 and scored the relatedness between a usage and a sense def- 4. The score meaning is the following: inition according to the Diachronic Usage Relatedness 0: Cannot decide (DUReL) framework [43], specially designed for lexical 1: Unrelated semantic change annotations. The DUReL framework 2: Distantly Related consists of a 4-point scale for quantifying the relatedness 3: Closely Related of a word usage and a sense, or score 0 if the annotator 4: Identical cannot decide: Answer just with the score.
Input: <left context> [TARGET] virtue [TARGET] <right context> • 0 - Cannot decide • 1 - Unrelated • 2 - Distantly related • 3 - Closely related • 4 - Identical
This process yields a total of 8,989 prompts for the regression task.
As for the binary classification task, the DuREL 1-to-4
Table 1 shows an example of the usage annotation scale was binary encoded as follows: for target word beatus. The senses presented to the an- • Pairs of sense and sentence scores equal to or notators were: (a) “blessed”, (b) “rich”, (c) “fortunate”, above 3 were labelled as yes; (d) “happy” and (e) “rewarded”. Let’s focus on the sense • Pairs of sense and sentence scores equal to or “blessed”, which only emerged later with the advent of below 2 were labelled as no. Christianity. Notice how it scores 1 for the first usage, The prompt is the following: dated 46 BC, while it scores 4 for the second usage, dated circa 1100 AD. Listing 2: Prompt generated by each sense annotation for
Target word virtus was chosen for calculating the inter- binary classification task. annotator agreement between four annotators: the average pairwise agreement computed as Spearman correlation coeficient was 0.69, comparable with interannotator agreement for modern languages, e.g., English 0.69, Swedish 0.57 and German 0.59 [43]. See [6] for the detailed process behind the creation and annotation of the dataset.
Instruction: Given the target word ‘‘virtute’’ and the sentence in input where the word is enclosed by the [TARGET] tag, and the following meaning ‘‘virtue, personified as a deity’’, assign a label "yes" or "no".
The label meaning is the following: "yes": The sense for the target word occurrence
is correct "no": The sense for the target word occurrence
is not correct Answer just with the label.
Pairs of sense and sentence were split in a stratified manner, based on the scores assigned to each sense. This 9https://logeion.uchicago.edu/ Input: <left context> [TARGET] virtue [TARGET] <right context>
Pairs of sense and sentence with score 0 were not considered in this experiment; thus, with respect to the scores distribution in Table 2, the training set for binary classification task consists of 6,255 instances instead of 6,299, and the testing set has 2,675 examples instead of 2,690, yielding a total of and 8,930 prompts. This binary encoded dataset comprises 956 instances of class yes and 1,719 no, resulting in a very imbalanced dataset in which class yes represents only 35.73% of the entire dataset.
The idea behind this work is to leverage this dataset for building a benchmark for the evaluation of LLMs in disambiguating Latin words as described in the following section.
As stated in the introduction, one of the aims of this paper is to assess whether fine-tuning on LLMs can improve their performance on a lower-represented language, compared to a zero-shot setting. To do so, we exploit the prompt dataset created from LatinISE, described in Section 3. Tables 3 and 4 introduce the LLMs of choice and summarise their characteristics. • LLaMA-3 instruction-tuned. We use publicly available checkpoints of Meta’s LLaMA 3.3-70B
During training, we use the following parameters: = 32, ℎ = 64, _ = 2 − 4 and ℎ_ = 32. We train all models for five epochs on the whole training dataset. The training was performed using a single GPU NVIDIA RTX A6000 with 48GB of memory.
and 3.1-8B variants with instruction tuning, ac- As mentioned in Section 1, our study has two objectives. cessed via the TogetherAI API10 and Unsloth First, we want to test the models ability to disambiguate API11, respectively; Latin senses in a zero-shot setting. In this way, we aim to • GPT-4o-mini. accessed via Microsoft Azure API, first establish how well the model inherent multilingual this model is used without any task-specific train- knowledge performs in accurate sense prediction. Next, ing. Prompting is designed to simulate realistic we perform task-specific fine-tuning, which enables us WSD instructions. to adapt both standard and instruction versions of LLMs.
The objective is to quantify the gain obtained through
For zero-shot WSD, we directly use the prompt test this additional training step. set, unseen during fine-tuning (see Section 4.2). After a It is worth noticing that the dataset of choice was preliminary prompt engineering step, we use the prompt initially devised for the Unsupervised LSCD task [13], in Listing 3, which is the same as the one used for fine- not for WSD; therefore, comparing the results of the tuning. shared task with the results of this work is not feasible. GPT-4o-mini and LLaMA-3.3-70B-instruct-turbo 4.2. Fine-tuning act as a zero-shot baseline for this experiment, to assess the capabilities of models not specially devised or fineUsing the training split of the dataset, we fine-tune the tuned for the Latin WSD task. open-weight LLaMA-3.1-8B model. Given the compu- It is crucial to note that the dataset is highly imbaltational constraints associated with full fine-tuning of anced, as many instances are annotated with 1, since each large models, we adopt a parameter-eficient fine-tuning word occurrence is generally assigned a single meaning; (PEFT) approach based on Low-Rank Adaptation (LoRA). consequently, all other meanings receive the lowest score.
LoRA [44] introduces trainable, low-rank matrices into Notice that all the metrics are computed with the dataset each transformer layer to adapt the model to a down- imbalance in mind. Balanced Accuracy12 is defined as stream task. Instead of updating all model parameters, the average recall obtained inch class. Weighted PreciLoRA freezes the pre-trained weights and injects a low- sion13, Recall14 and F115 calculate metrics for each larank decomposition into the linear projections of the bel, and find their average weighted by support. Finally, self-attention and/or feed-forward layers. This strategy Macro F1 and Micro F1 scores are variants of F1. The significantly reduces the number of trainable parameters former is the only metric that does not take into account and memory usage, allowing eficient fine-tuning even label imbalance, but computes metrics for each label and on consumer-grade GPUs. We use the implementation ifnds their unweighted mean; the latter calculates metprovided by the Unsloth library, which enables us to re- rics globally by counting the total true positives, false duce the required memory and accelerate the training negatives and false positives. Details about the DuREL process. During the training, we format the instruction annotation statistics are reported in Table 2. data using the prompt reported in Listing 3 by relying We release the following resources, available on on the chat template specific to the LLaMA models. GitHub16: i) the source code; ii) instruction fine-tuning Listing 3: Prompt used for the fine-tuning. and testing data; iii) links to the fine-tuned models on HuggingFace and the outputs of all evaluated models.
System: <Instruction> User: <Input> Assistant: <Output> 10https://www.together.ai/ 11https://unsloth.ai/
Table 5 illustrates the results of the WSD task. Mean This study explores the ability of Large Language Models Squared Error (MSE) and Root Mean Squared Error (LLMs) to address Word Sense Disambiguation (WSD) (RMSE) show that the fine-tuned model is better at pre- in Latin, a historically rich yet computationally lowdicting the annotation score. To give a complete overview resourced language. The first contribution of our work is of the results, we also provide classification metrics. Al- the release of a dataset for evaluating the WSD abilities though GPT-4o-mini shows a higher precision, LLaMA- of LLMs in Latin. This dataset is created by leveraging 3.1-8B-instruct-ft outperforms every other model. It an existing manually annotated dataset. Then, using the is interesting to note the high diference in performance new dataset and through both zero-shot and fine-tuned between LLaMA-3.3-70B-instruct-turbo and LLaMA- evaluations, we observed that while general-purpose 3.1-8B-instruct-ft. These results prove that the fine- LLMs exhibit a promising baseline ability to handle Latin tuning of a medium-sized LLM using a single GPU can WSD, significant improvements are achieved through overcome a model of the same family with about nine task-specific fine-tuning. The fine-tuned LLaMA-3.1-8Btimes the number of parameters. instruct model outperformed larger and more resource
To better understand the behaviour of each model, intensive models in accuracy and F1 scores, underscorwe report the confusion matrix of each model in B. The ing the impact of targeted instruction tuning, even on matrices of GPT-4o-mini (Figure 1) and LLaMA-3.3-70B medium-sized architectures. Nevertheless, challenges (Figure 2) show that the models often confuse the label 1 remain. The dataset’s inherent class imbalance, with a with other labels. It is interesting to note that GPT-4o- predominance of “unrelated” sense labels, likely influmini confuses the label 1 with the label 4 508 times. This enced the models’ predictions and underscores the need behaviour is more evident in LLaMA-3.1-8B-instruct for more balanced and semantically diverse training data. (Figure 3) where 913 instances labelled as 1 are confused Future work will focus on three main directions: i) Exwith label 3 and 579 with labels 4. panding the annotated dataset to include more lemmata
The fine-tuned model LLaMA-3.1-8B-instruct-ft and a broader variety of senses; ii) Evaluating model per(Figure 4) is the best at recognising label 1. This be- formance on additional semantic tasks, such as definition haviour is evident since the model tends to overfit on the generation and contextual paraphrasing in Latin; iii) Exmore frequent class. ploring multilingual and cross-lingual transfer learning strategies, leveraging annotations from related Romance 5.2. Binary Classification task languages to further boost Latin model capabilities. Results of the WSD task framed as a binary classification task are in Table 6, as well as the confusion matrix Acknowledgments of each model in Appendix B. Our proposed fine-tuned model LLaMA-3.1-8B-instruct-ft shows a strong per- We acknowledge the support of the PNRR project FAIR formance boost with respect to LLaMA-3.1-8B-instruct Future AI Research (PE00000013), Spoke 6 - Symbiotic AI and LLaMA-3.3-70B-instruct-turbo. On the other (CUP H97G22000210007) under the NRRP MUR program hand, GPT-4o-mini performance is in line with LLaMA- funded by the NextGenerationEU. 3.1-8B-instruct-ft, and even surpasses it in Precision and Accuracy. In general, our LLaMA-3.1-8B-instruct- References ft outperforms the baseline models. Figure 8 shows that LLaMA-3.1-8B-instruct-ft performs the best on class no, while GPT-4o-mini predicts class yes better.
Precision Recall Accuracy
Cicero’s Tuscolanae Disputationes la: [...] Dico enim constanter grauiter sapienter fortiter. Haec etiam in eculeum coiciuntur, quo uita non adspirat beata. - Quid igitur? solane beata uita, quaeso, relinquitur extra ostium limenque carceris, cum constantia grauitas fortitudo sapientia reliquaeque uirtutes rapiantur ad tortorem nullumque recusent nec supplicium nec dolorem? [...] en: For I say constantly, gravely, wisely, and strongly.
These things are also cast into the rack, to which life does not aspire for happiness. - What then? Is a blessed life alone, I pray you, left outside the door and threshold of the prison, when constancy, gravity fortitude, wisdom and the other virtues are snatched away to the torturer and refuse neither punishment nor pain? Robertus Grossetest’s De libero arbitrio la: [...] Ex quo fit, ut de nihilo creauerit omnia.” Eadem itaque ratione solus facit ominia, nulla adiutus natura. Horum autem obiectorum solutio haberi potest ut uidetur ex uerbis beati Bernardi sic dicentis: “Ipsa gratia Liberum arbitrium excitat, cum seminat cogitatum. Sanat, cum mutat afectum; roborat, ut perducat ad actum; seruat, ne sentiat defectum.” [...] en: From which it comes about that He created all things out of nothing.” Therefore, by the same reasoning, He alone creates all things, without any help from nature. But the solution to these objections can be found, as can be seen from the words of Blessed Bernard, who says thus: “Grace itself awakens Free will when it sows thought. It heals when it changes afection; it strengthens, so that it may lead to action; it preserves, so that it may not feel a deficiency.”
Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) and Grammarly in order to: Paraphrase and reword, Improve writing style, and Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.