Large Language Models (LLMs) have demonstrated remarkable performance in machine translation (MT), specifically concerning high-resource European languages. However, their extensive computational requirements raise sustainability concerns. This paper investigates the potential of smaller, fine-tuned language models as a more sustainable alternative for MT tasks. We conduct a comparative analysis of model performance in terms of translation quality and CO2eq emissions, and examine the key errors associated with using smaller models. Furthermore, we propose a novel metric that balances translation quality against environmental impact, aiming to inform more sustainable model selection in MT research and practice.
larger models. This setup allows us to assess the
realworld viability of small models for machine translation
MT has been a core topic in natural language processing when fine-tuned for specific language pairs and domains.
(NLP) for several decades, evolving from rule-based sys- We conduct a comprehensive analysis of model
perfortems to statistical methods, and more recently to neural mance, in terms of translation quality and CO2eq
emismachine translation (NMT) and transformer-based mod- sions, validating our results with a human evaluation
els. The emergence of LLMs has significantly advanced of the key errors associated with each model. Finally,
the state-of-the-art in MT, demonstrating remarkable we introduce a metric called Carbon-Adjusted Quality
performance on various NLP tasks [
Their ability to generate fluent, context-aware trans- lection, that quantifies the trade-of between translation
lations in diferent domains has positioned LLMs at the quality and sustainability.
forefront of MT research [
However, this performance comes at a significant envi- 2.1. LLMs and Translation
ronmental cost. Training and deploying LLMs consumes LLMs have achieved state-of-the-art results in MT, by
enormous computational resources, leading to consider- leveraging extensive pretraining on multilingual corpora,
able carbon emissions and infrastructure demands [
This paper investigates whether smaller language mod- lel corpora, LLMs are pretrained on massive web-scale
els can serve as eficient and environmentally sustainable monolingual and multilingual datasets. This enables
valid alternatives to LLMs in MT. Specifically, we will them to generate high-quality translations even in
doifne-tune the Gemma-3-4B[
LLMs increasingly favored in professional translation Moreover, recent studies show that even highly comsettings. plex capabilities like multi-step reasoning, previously
The adoption of LLMs, however, requires substantial thought to emerge only in models over 100B parameters,
computational resources and infrastructure, which may can be acquired by SLMs through targeted fine-tuning
not be feasible for all organizations or languages. Beyond and distillation. Distilling chain-of-thought reasoning
these practical limitations, the widespread adoption of abilities, for instance, from GPT-3.5 into FlanT5 variants
LLMs also raises significant concerns about their envi- (250M to 3B) resulted in significant performance
improveronmental sustainability. ments on math reasoning tasks without the need for full
retraining of the model’s weights [
In addition to training, the inference phase of LLMs bined with modern fine-tuning strategies and lightweight
also significantly contributes to their overall carbon foot- architectures, could ofer a pragmatic and sustainable
print, particularly in large-scale deployments. While the path forward for machine translation and other NLP
apenergy cost of a single inference is lower than that of plications.
training, the cumulative emissions can become
substantial depending on usage patterns. For example, serving
a single ChatGPT prompt may emit over 4g of CO2eq, 3. Fine-tuning a SLM
more than 20 times the emissions of a typical web search
[
The same study emphasizes that total environmental impact depends on a combination of factors: model size, batch size, and hardware type. The latter reflects the impact of producing high-performance GPUs, which involves substantial embodied carbon emissions. Although these emissions occur at production time, they contribute to the model’s overall environmental cost throughout its operational lifetime.
tainable alternatives to larger, more resource-intensive
models in machine translation, we compare two
stateof-the-art models: GPT-4o-mini [
We fine-tune Gemma-3-4B on a carefully curated
subset of the OpenSubtitles corpus, obtained from the Opus
Corpus [
and importance of SLMs as eficient alternatives to LLMs in constrained environments [12, 13]. SLMs, typically ranging from hundreds of millions to a few billion parameters, are substantially more resource-eficient and accessible, especially when tailored to specific tasks.
SLMs benefit from architectural simplifications, such
as compact tokenizers and reduced model width and
depth, which are optimized to preserve key
capabilities while minimizing parameter overhead [14]. Small
models, like Gemma [
of the OpenSubtitles corpus, made available through the Opus Corpus repository. While OpenSubtitles is a rich resource for dialogue-based translation data, it also contains a considerable amount of noise due to its automatic extraction and alignment process. Therefore, careful curation was necessary to ensure the quality and relevance of the dataset.
We began by removing duplicate entries and any
empty lines. Following this, we applied the langdetect
[
Finally, we applied COMET-QE [21], a quality estima- of the electricity grid based on geographic location.
tion model, to score the remaining sentence pairs. Using The fine-tuning session consumed approximately 0.65
these scores, we selected the top 100,000 highest-quality kWh, resulting in an estimated 162 g CO2eq under an
translations for use in our fine-tuning experiments. The average EU grid intensity of 250 gCO2/kWh.
strategy of mining large datasets and selecting top-k
sentence pairs based on quality metrics for fine-tuning helps 3.3. Gemma-3 Evaluation
to further filter out noisy segments and ensures that the
limited available data contribute maximally to model
training [
segments from the same corpus, ensuring no overlap with the training data. Table 2 reports the evaluation of EN–IT translation performance for Gemma-3-4B before and after LoRA fine-tuning, using BLEU [ 27], chrF [29], and COMET [30] as quality metrics. Our fine-tuned Gemma-3-4B model, with only 0.42% of additional trainable parameters, shows a notable improvement over the base version, achieving a +4 point gain in BLEU, a modest increase in chrF, and a +1 point gain in COMET. These results place our model on par with GPT-4o in COMET and above GPT-4o-mini in all three metrics.
In addition to performance, we also measure the
environmental impact of inference using the CodeCarbon
library. The estimated carbon emissions per inference for
the fine-tuned model are approximately 0.028g CO 2eq,
twice that of the base model, but significantly lower than
GPT-4o models, each exceeding 0.42g per inference as
estimated in a relevant study [
Our evaluation demonstrates that fine-tuning Gemma3-4B with LoRA leads to competitive performance gains with low additional environmental cost.
The Gemma-3-4B model was fine-tuned for three epochs
using Low-Rank Adaptation (LoRA) [
Our fine-tuning pipeline was implemented using the inference using the CodeCarbon framework. Importantly,
Hugging Face Transformers library [28], leveraging its we emphasize in our approach that a sustainable model
integration with the PEFT library. For the LoRA config- choice should not be based on its parameter size alone,
uration, we set the rank (r) to 16 and the scaling factor but actual carbon emissions.
(alpha) to 16, with a dropout rate of 0.05 to improve As shown in Table 3, we highlight that the
relationgeneralization. The training was carried out on a single ship between model size and emissions is non-linear.
NVIDIA A100 GPU using mixed-precision (fp16) compu- For instance, Qwen-3B [
Similarly, the assumption that larger models necessarily produces more carbon emissions does not always hold. 5. Error Analysis This is the case for models developed with a Mixture-ofExperts (MoE) architectures. In these models, only a sub- To complement the quantitative results and better set of the total parameters is activated during inference. understand the practical implications of the qualityAs a result, MoE models like Mixtral, although large in sustainability trade-of, we conduct a manual error analaggregate size, can have lower or comparable emissions ysis on the translations generated by four representative to smaller, densely activated models. This decoupling models: our fine-tuned version of Gemma-3-4B, and the of parameter size and runtime eficiency highlights the baseline instruction-tuned Gemma-3-27B, Llama-3.2-3B need for measuring more empirical results, such as CO2eq and Llama-3.3-70B. emissions.
Therefore, we introduce a Carbon-Adjusted Quality Score (CAQS) metric as a measure of model costefectiveness, and we calculate it on each corpus translation generated by the models evaluated in our study. Our CAQS score penalizes each gram of carbon emissions exponentially, while ensuring that low-quality models are not rewarded more than high-quality ones, regardless of their eficiency. We define the CAQS metric as follows.
larger and environmentally demanding model, Llama-3.3- In this study, we investigated the potential of SLMs as 70B. In terms of weighted scores, both models show simi- sustainable alternatives to LLMs, for MT tasks focusing lar results, with very few major errors and a comparable on the EN-IT language pair. Our results demonstrate number of minor ones. The smallest Llama checkpoint that parameter-eficient fine-tuning of SLMs can achieve presents a very high number of both major and minor competitive translation quality while dramatically reducerrors, when compared to the Gemma-3-4B model. The ing environmental impact. The fine-tuned Gemma-3-4B ifndings may suggest that Llama-3’s architecture is sub- model achieved performance comparable to GPT-4o and optimal for translation tasks across model sizes, given outperformed GPT-4o-mini across all metrics, while conthat Gemma-3-4B matches the performance of its largest suming approximately 15 times less energy per inference. checkpoint. However, the results should be interpreted We complement these results with a MQM human evalwith caution, as our evaluation was limited to a small uation across a set of representative models, confirming test set and a single language pair. that Gemma-3-4B performed comparably to the much
In terms of error category distribution increasing pa- larger Llama-3.3-70B, producing only minor fluency and rameter size leads to an overall performance improve- spelling errors. ment, as seen in Table 5. This trend is particularly evident We also highlighted that the relationship between within the Gemma models, where the jump from 4B to model size and carbon emissions is non-linear and highly 27B parameters results in a significant drop in errors dependent on architectural choices, emphasizing the across all categories. In contrast, Llama-3.2 models ex- need for accurate measurements of carbon emissions. hibit a less linear improvement, suggesting diminishing Given the non-linear relation between model size returns from scaling model size. This observation, how- and environmental impact, we introduced the CAQS, a ever, is limited by the fact that only the smallest Gemma novel metric specifically designed to facilitate sustainable model was LoRA-adapted, while the LLaMA models were model selection by integrating translation quality and evaluated in their original form. A more rigorous com- carbon emissions. CAQS includes a sensitivity paramparison, involving both original and adapted versions eter that allows users to adjust how strongly quality is across model sizes, is left for future work. penalized by the model’s carbon footprint. According to
When comparing Gemma-3-4B and Llama-3.3-70B, we this metric, Gemma-3-4B and Magistral-Small emerged ifnd that most of the errors in the Gemma model are con- as the most eficient models in our study, ofering optimal centrated in surface-level issues, especially in spelling trade-ofs between sustainability and translation quality. diacritics. These errors, however, do not compromise
programme in Artificial Intelligence, partnered by University of Pisa and University of Naples “L’Orientale”, through a doctoral grant (ID 39-411-24-DOT23A27WJ6603) established by Ex DM 318, of type 4.1, co-financed by the National Recovery and Resilience Plan. In light of practical constraints related to time and resources, the main limitations of our study lie in the relatively small sample of segments and the domain-specific nature of the OpenSubtitles corpus, used for both training and inference. For this reason, we highlight that our evaluation results may not be reproducible in other domains.
As our evaluation focuses on a relatively high-resource language pair (EN-IT), our findings may not be applicable for distant or low-resource pairs. Finally, our carbon emission measurements are specific to the computational infrastructure used (NVIDIA A100 GPUs, EU electricity grid). Results may difer when deploying models on diferent hardware configurations, cloud providers, or geographical regions.
Declaration on Generative AI