English. In this paper we analyze the performance of the NLLB-200 models from Meta AI on a manually built parallel corpus of Ligurian (specifically, the Genoese variant), consisting in 283 sentences and their respective Italian translation. Our experiments highlight some issues with NLLB-200, especially regarding local knowledge, deriving from some choices done for the training process. Italiano. In quest'articolo analizziamo la performance del modello NLLB-200 di Meta AI su un corpus parallelo, costruito manualmente, di 283 frasi in genovese e la loro rispettiva traduzione in italiano. Mostriamo i punti deboli di NLLB-200, in particolare il trattamento dei toponimi ed altri termini in relazione con un contesto locale ligure, evidenziando alcuni problemi derivati dalle scelte fatte nel training di questo modello.
lem that may afect MT tools and methods. In particular,
the syntax of Ligurian has not been completely
standardNLLB (No Languages Left Behind) [
The content in Ligurian on the web is very scarce. The texts in Ligurian to train a neural model for the normalmain source is Wikipedia, which has only 11, 1721 ar- ization of Ligurian texts. The example in Figure 1 allows ticles in Ligurian, many of them being a bit more than to appreciate the high variability of Ligurian spelling. drafts. In comparison, the number of articles in Welsh, which has an estimated equivalent number of native speakers (500, 000) is twenty times as much as Ligurian.2 This diference is easily explained by the fact that Welsh is an oficial language, supported by the local government while Ligurian is mostly orally spoken.
The rarity of content in Ligurian is not the only prob
Given this premise, it is important to evaluate whether the NLLB-200 model is able to deal with these problems. For this reason, we conducted an evaluation by composing a test dataset in Genoese that was not used for NLLB training. The copyright-free subset of this
dataset is available at the following address: h t t p s : cross-entropy with respect to the word-level distribution //github.com/dbuscaldi/zeneize. of the teacher model.
In the following section we describe the NLLB-200 model and how the training has been carried out. In Section 3 we show the experiments carried out on our 3. Experiments and Results data set and discuss the results. Finally, in Section 4 we draw some conclusions of this analysis and propose some ideas to improve the model on the basis of this experimentation.
We collected a dataset of texts in Genoese from three
diferent sources. First of all, 95 lines from two of the
most famous songs of Fabrizio De André, “Crêuza de
mä" (small path to the sea) and “’A çimma" (the cima is a
typical Genoese dish) with their respective translation in
2. NLLB-200 Italian found on the oficial Fondazione De André page 5
(retrieved on 2022-08-13), and the popular song “Trilli
To be able to interpret the results we must first take a bet- trilli". Then, 188 sentences from “Zêna e contórni", a
ter look at NLLB-200: the dataset on which it was trained, translation in Genoese from Charles Dickens section on
and the characteristics of NLLB-200 and the models used Genoa, Italy, from his “Pictures of Italy" work. We used
in our experiments. the wikisource text6 and an Italian translation obtained
from the English one with Deepl7.
2.1. Training Data We applied the NLLB-200 models on this dataset,
obtaining the results in Tables 1 and 2. We calculated the
The training data for Ligurian were created by compos- results using SacreBLEU [
Model NLLB dist-600M NLLB dist-1.3B NLLB 1.3B NLLB 3.3B spBLEU dist-600M is never able to translate correctly “Genova" for for..."). The larger model (dist-1.3B) is not immune to into “Zena", it always translates it the same as in Italian this behaviour although it happens only 2 times instead (Genova). Finally, NLLB dist-1.3B correctly translates it of 9. The non-distilled models don’t present this problem. 6 times. The problem seems also to afect other proper The fact that the models fall into this kind of repetition nouns, such as Saint Peter (San Pietro in Italian) which could be due to the lack of suficient training data for is correctly translagted in Genoese as “San Pê". In fact, the word-based online distillation process. Therefore the is translated as “San Peixe", which is Portuguese for fish, probability distribution for the tokens is skewed towards in both distilled models. The 3.3B model translates it as some frequent words (“pe" - for, “ti" - you, “ben" - well). “San Pêo" and the 1.3B one translates it as “San Peçio". We observed that the minimum frequency in the NLLBBoth these translations make no sense in Genoese. seed dataset of words that are repeated is 39 (for the word
In the Genoese-Italian direction, only NLLB dist-1.3B “sciâ": probably as part of “in sciâ" - on top of). translates “Zêna" correctly in 3 out of 9 cases. Both 1.3B models translate in one case it as “Ginevra" (Geneva, in Switzerland). Other spelling errors show “Giena" by 4. Conclusions the dist-600M model and “Gênes" (in French instead of Italian) by the 3.3B model. Looking into the tokenizer From our preliminary analysis, carried out on a dataset we observed that “Genova" is not in the dictionary and specific to the Genoese culture, we can afirm that curis tokenized as Gen-ova, and “Zêna" is tokenized as Z- rently NLLB-200 is not good enough to deal with Genoese êna. On the other hand, “San Pê" is correctly translated texts or to translate text into Genoese. In particular, we by all models. Due to the output occurring sometimes found out that local toponyms are dificult to translate: in diferent languages than the target ones, we suspect how good is an MT tool that is not able to correctly transthat the previous errors may result from the LASER-3 late the name of the largest city where the language is embeddings which are language-family based. spoken or the name of the language itself? Given the
Both distilled models fall into repetitions. For instance, information provided regarding NLLB-200 models, we in the Dickens text, NLLB dist-600M translates “Ma, per can identify two main elements explaining this behaviour. il momento, gironzolo qui intorno, in tutti i buchi e gli The first one is the training data: they do not cover loangoli del quartiere, in un perpetuo stato di forzata sor- cal information, but general English Wikipedia articles, presa" (“But, as yet, I stroll about here, in all the holes and so they lack to provide the context in which Genoese corners of the neighbourhood, in a perpetual state of for- is usually spoken. The second one is the tokenization process and the LASER-3 embeddings: given the high lorn surprise") into “Ma, pe-o momento, o l’é in sciâ çitae, spelling variability of the Ligurian language, we suspect in tutti i buchi e in tutti i cantoni do quartiere, in un stato that the tokenization process may not be precise and that dneo wso,rhpereissâofnortçoâp poef tpheepceitpye, ipneaplletphee pheolpees..a."n(d“Bcourtn,eforsr it may map some tokens into a position in the embedding of the neighbourhood, in a state of forced surprise for for space that does not correspond to their actual “meaning", maybe also because of a sort of interference from other Western Romance languages that are very close to Ligurian.
However, NLLB-200 is a big step forward making
endangered languages such as Ligurian and its variants
available to everyone. From our point of view, we think
that NLLB-200 could be improved in various ways, for
instance fine-tuning the model on more “local" datasets;
and possibly including knowledge regarding
Out-OfVocabolary words that are often named entities, for
instance with the methods proposed by [