We propose MAGNET - MAchines GeNErating Translations, a CALAMITA Challenge which aims at testing the ability of large language models (LLMs) in the hot topic of automatic translation, focusing on Italian and English (in both directions) to overcome the marginality with which Italian is considered by the machine translation community. We propose a benchmark composed of two portions with diferent distribution policies (one free to use, the other not discloseable), allowing to handle data contamination issues. The publicly available section of the benchmark is distributed on Hugging Face, whereas in this report we describe the details of our challenge, including the prompt formats to be used. Additionally, we report the performance of five models, including a LLM and diferent sized translation models, in terms of four evaluation metrics, whose scores allow an overall evaluation of the quality of the automatically generated translations.
Machine Translation (MT) refers to the process, carried out
by a computer program, of translating text from one
language to another without human involvement. The idea of
using digital computers to translate natural languages dates
back to the 1940s, making MT one of the oldest fields of
artificial intelligence. Since then, the improvement in translation
quality has been constant and achieved through increasingly
efective approaches (rule-, example- and statistical-based);
however, the most significant advances have likely been
observed over the last few years, thanks to the introduction
of neural networks. Neural models specifically trained for
accomplishing the translation task, like DeepL Translator,1
reach outstanding quality, even if the so-called human
parity has not been achieved yet, especially in unrestricted
domains and for language pairs not involving English.
Recently, an alternative neural-based method is gathering a
lot of interest due to its undoubted potential; it consists in
prompting generative large language models (LLMs), like
GPT models [
The MAGNET challenge provides a framework for assessing the ability of LLMs in translating Italian text into English and vice-versa. It is organized following the blueprint of other long-standing MT shared tasks, such as those proposed in the WMT3 and IWSLT4 conferences, where Organizers prepare and distribute development and test sets, define the training conditions, possibly providing specific training data, establish the evaluation modalities, typically via automatic metrics and occasionally enriched by human evaluations, collect and evaluate participants’ submissions, and finally disclose the results.
The MAGNET challenge supplies a benchmark divided in two portions: one based on a publicly available MT benchmark and a private one (see Section 3). This allows participants not only to evaluate their models but possibly to also fine-tune them, by exploiting the open portion of the MAGNET benchmark for development purposes.
Multiple evaluation metrics are employed so as to have a comprehensive overview of the quality of the translations generated by a specific model. Indeed, shared tasks on automatic metrics are still being organized,5 as evidence of the fact that none of the metrics designed up to now by the scientific community has proven capable of covering every single aspect that defines a “good” translation by itself . 2https://www.linkedin.com/pulse/machine-translation-mt-market- size2024-suhoe/ 3https://www2.statmt.org/wmt24/translation-task.html 4https://iwslt.org/2024/#shared-tasks 5https://www2.statmt.org/wmt24/metrics-task.html
In addition, in order to allow for comparisons, scores measured on the translation generated by Llama3-8B-Instruct and a number of other models are made available (see Section 4).
We test LLMs’ ability to translate between Italian and English using a parallel corpus composed of two parts: an OPEN portion and a CLOSED one.
OPEN For the OPEN portion of the MAGNET benchmark
we propose FLORES+, the latest version of FLORES-2006 [
CLOSED The CLOSED subset is a MT test set developed by FBK by collecting texts of English and Italian news, and then commissioning their professional translation to a specialized company. This resource is private and not publicly accessible. See Section 3.3 for statistics on this portion of the MAGNET benchmark.
Both subsets allow for the evaluation of MT quality in
both translation directions, i.e. English→Italian and
Italian→English. The decision to split our benchmark in two
subsets is primarily motivated by their current distribution
policy, which is inherently linked to growing concerns about
data contamination [
The datasets are organized in a parallel text format, i.e. every entry is composed of a sentence in one language and the corresponding translation. The OPEN portion of the benchmark is publicly available on Hugging Face,7 whereas access 6https://github.com/openlanguagedata/flores 7https://huggingface.co/datasets/FBK-MT/ MAGNETbenchmark4CALAMITA24 to the CLOSED portion is only provided to the Organizers of the task.
In Table 2 detailed statistics are provided on the various sections of the benchmark in terms of number of segments (#seg), and of English (|en|) and Italian (|it|) words.
We evaluate LLMs’ performance in translation using a set
of four automatic metrics selected in light of the ongoing
challenges in MT evaluation, which still pose an open
problem. New metrics are indeed continually proposed, and
evaluation campaigns aimed at assessing these metrics are
organised periodically (for example, the annual WMT
Metrics Shared Task [
sacreBLEU [
All of them are quality metrics, that is the higher the score the better the translation. The overview of the scores from all these metrics allows for a robust assessment of the quality of individual models, and a fair comparison between diferent models as well.
We provide reference performance on our challenge of one of the most popular open LLMs, and four state-of-theart MT models: 8sacreBLEU signature: nrefs:1|case:mixed| |eff:no|tok:13a|smooth:exp|version:2.0.0 9sacreBLEU signature: nrefs:1|case:mixed| |eff:yes|nc:6|nw:0|space:no|version:2.0.0 prompt en-it it-en content Translate the following sentence into Italian: <On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.> <Nella giornata di lunedí, alcuni scienziati della Scuola di Medicina dell’Università di Stanford hanno annunciato l’invenzione di un nuovo strumento diagnostico capace di ordinare le cellule in base al tipo: un chip minuscolo che può essere stampato utilizzando stampanti a getto di inchiostro al costo di circa 1 centesimo di dollaro l’uno.> Translate the following sentence into English: <Nella giornata di lunedí, alcuni scienziati della Scuola di Medicina dell’Università di Stanford hanno annunciato l’invenzione di un nuovo strumento diagnostico capace di ordinare le cellule in base al tipo: un chip minuscolo che può essere stampato utilizzando stampanti a getto di inchiostro al costo di circa 1 centesimo di dollaro l’uno.> <On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.>
Llama-3-8B-Instruct:10 a LLM from the Llama 3 model
family [
HelsinkiMT:12 the Language Technology Research Group at the University of Helsinki made available under the CC-BY-4.0 license a set of neural MT models trained with MarianNMT13 on OPUS data,14 including English-Italian15 and Italian-English16 models.
mBART50:17 a multilingual neural translation model
that covers any pair from a set of 50 languages, English and
Italian included [
NLLB:19 No Language Left Behind (NLLB) is also a multilingual neural translation model that covers any pair from more than 200 languages, including the two we are interested in. The code was developed by Meta/Facebook as a branch of fairseq and is released under the MIT license. Five 10https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 11https://huggingface.co/docs/transformers/main/en/chat_templating 12https://github.com/Helsinki-NLP/Opus-MT 13https://marian-nmt.github.io/ 14https://opus.nlpl.eu/ 15https://huggingface.co/Helsinki-NLP/opus-mt-en-it 16https://huggingface.co/Helsinki-NLP/opus-mt-it-en 17https://huggingface.co/facebook/mbart-large-50 18https://github.com/facebookresearch/fairseq 19https://github.com/facebookresearch/fairseq/tree/nllb diferent NLLB models are available under the CC-BY-NC 4.0 license, which mainly difer in size, ranging from the smallest with 600M parameters to the largest with 54.5B parameters. On the basis of their manageability and oficial performance claimed by the authors, we decided to include two NLLB models in this investigation, the distilled variant with 1.3B parameters (NLLB_1.3B) and the one with 3.3B parameters (NLLB_3.3B).
Table 3 provides the scores measured for each model on all evaluation sets of the benchmark, except for the OPEN dev set, since we reserved that subset as the source of the exemplars used for few-shot prompting with Llama-3-8BInstruct. First of all, we note that the performance of the three multilingual translation models mBART50, NLLB_1.3B and NLLB_3.3B are strictly in increasing order according to their number of parameters, with respect to all metrics (with only one microscopic exception). In general, Llama-38B-Instruct performs better than mBART50 and worse than NLLB_1.3B.
The behavior of HelsinkiMT is more dificult to frame: there are cases in which it is definitely the best performing model (CLOSED-IT, it→en) or at least competitive with NLLB_3.3B (CLOSED-UK, en→it; CLOSED-IT, en→it); others in which it is only slightly better than mBART50 (OPEN devtst, it→en; CLOSED-US, it→en). This can probably be explained by the fact that HelsinkiMT is not a single model, rather a collection of models specifically trained for covering the translation between specific languages. That is, HelsinkiMT en→it and it→en models were trained independently, on diferent training data. Therefore, it is possible that their performance when compared to that of other models may not be consistent across the various sections of our benchmark.
In summary, we can state that Llama-3-8B-Instruct, a general purpose, generative model only conditioned towards performing translation by four task exemplars, compares well to translation models; likely, fine-tuning Llama-3-8BInstruct on the translation task could allow it to achieve even better performance. However, it should be considered that this version of Llama-3-8B-Instruct – which is also the smallest of that model family – has 8B parameters, more than twice the parameters of NLLB_3.3B and an order of magnitude more than mBART50. system
mBART50
mBART50
mBART50
mBART50
0.8656 27.53 0.8494 23.88 0.8774 29.31 0.8805 29.95 0.8795 26.36
0.8949 57.35 0.8776 47.46 0.8954 55.12 0.8968 56.00 0.8985 39.29
Nowadays, LLMs are trained on huge amounts of data mostly crawled from the web. Therefore, as already pointed out in Section 3, it is hard to be sure that there is no data contamination, that is no overlap between training and evaluation data. Data contamination makes the evaluation of LLMs unreliable since their performance may be inflated.
Concerning our specific case, the risk that OPEN/FLORES+ data are contaminated is not negligible; however the results shown in Table 3, which are good but realistic, do not seem to indicate any contamination.
In theory, the contamination risk of the CLOSED section is lower than for the CLOSED one, since the translations of the original texts have never been released. On the other hand, original texts are available on the web (although only for private use), therefore it cannot be ruled out that the models “know” them, in some way. For example, the exceptionally high results of HelsinkiMT on the CLOSED-IT set seem to be an anomaly, likely due to data contamination.
Our proposal does not focus on ethically charged topics. While the data we propose for the evaluation of automatic translation may mention sensitive topics or be aflicted by ethical issues such as social biases (e.g., gender bias), here we focus solely on MT quality evaluation and leave the investigation of ethical aspects to other resources and analyses.
The OPEN section of our benchmark is part of the FLORES+ dataset which is licensed under the Creative Commons Attribution Share Alike 4.0 International,20 which requires derivatives to be distributed under the same or a similar, compatible license. We opted for the same license.
There is no license associated with the CLOSED part of our benchmark as it is not distributed and can only be used by CALAMITA Organizers for evaluation purposes.
The work presented in this paper is funded by the European Union’s Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI Mediator for Virtual MEETtings BetWEEN People) and the PNRR project FAIR - Future AI Research (PE00000013), under the NRRP MUR program funded by the NextGenerationEU. 20https://github.com/openlanguagedata/flores/blob/main/LICENSE