This paper presents the development and comprehensive assessment of a Hindi-Punjabi machine translation system tailored specifically for the MTIL (Machine Translation for Indian Languages) track of FIRE 2023. Leveraging neural machine translation techniques, we developed a robust translation model to facilitate seamless communication between Hindi and Punjabi, two prominent Indian languages despite low resource availability. The methodology involved fine-tuning a pretrained NLLB-1.3B to adapt to the Hindi-Punjabi translation task. To evaluate the efficacy of the translation system, we conducted comprehensive experiments using standard evaluation metrics on FLORES, as well as, on our own testset. Our results demonstrate promising performance of Punjabi-Hindi language pair, showcasing highest score in terms of BLEU, chrF and TER metrics across all domain specific translations in the track. Similarly, our Hindi-Punjabi pair also scored the highest in all domains except Governance domain where our chrF and COMET scores marginally second highest though BLEU and TER were still the highest. The findings underscore the viability and potential of our developed machine translation system, contributing to the advancement of translation technology for Indian languages in diverse applications.
One of the key advancements in NMT was the development of the encoder-decoder architecture
(Sutskever et al., 2014) [
This paper outlines the building a high-performance translation model for the Hindi-Punjabi language pair which poses a significant challenge due to the scarcity of parallel training data. To address this limitation, we employed NLLB-200-3.3B, a state-of-the-art multilingual neural machine translation model designed to excel in low-resource settings. NLLB-200's ability to effectively utilize data from multiple languages, including Hindi and Punjabi, made it an ideal starting point for our translation model. By fine-tuning NLLB-200 on a carefully curated dataset of parallel Hindi-Punjabi sentences, we were able to achieve significant improvements in translation accuracy in compared to training a transformer model from scratch using the available corpus. The resulting translation model demonstrates the potential of NLLB-200 for low-resource machine translation tasks and its ability to bridge the communication gap between speakers of Hindi and Punjabi.
Our team participated in the FIRE-2023 MTIL challenge for both Punjabi to Hindi and Hindi to Punjabi language pairs. We employed NLLB-200, a cutting-edge machine translation model optimized for resource-constrained environments, as the foundation for our submissions. We further enhanced the model's performance by training it on general domain data, governance domain data, and healthcare domain data as described in next section. The effectiveness of our model was evaluated using chrF (character-level F-score) [8], the official metric for the task. Our model achieved the highest chrF scores Punjabi to Hindi in all domains. For Hindi to Punjabi language pairs, except for Governance domain, our model again achieved highest chrF score among all participating teams.
The dataset for this task consisted of a parallel corpus of Hindi-Punjabi sentence pairs collected from diverse domains, including General, Agriculture, Tourism, Education, Science & Technology, Governance, Health and News articles. Overall 140K parallel sentences were collected and curated for this task. Majority of the corpus, though human translated, but needed some vetting, cleaning and preprocessing before sending for training of the translation model. The dataset was split into training, development, and test sets to facilitate model training and evaluation. In addition, FLORES test set [9] is also used to evaluate the system which contains 1012 sentence pairs.
In order to build Hindi-Punjabi and Punjabi-Hindi translation models, we fine-tuned the NLLB-2001.3B pretrained model with our corpus. For evaluating both the models, we used two datasets i.e. our own (CDACN) test set containing 1000 sentences of mixed of domain and another publicly available FLORES test set. It was necessary to maintain the fairness and avoid biasness.
For our training purpose, we used SOTA Opennmt-Py toolkit which provides different configurations of building NMT models to play with. In this work, we built Transformer model from scratch and Fine-tuned model using the same toolkit. We used following methodology to train our models: 3.1.
Preprocessing plays a pivotal role in training Neural Machine Translation (NMT) models within the OpenNMT-py toolkit [10], serving as the foundational step in transforming raw textual data into a format suitable for effective model learning and translation. Primarily, the preprocessing workflow encompasses cleaning, tokenization, normalization, subword segmentation, and vocabulary construction. Tokenization, the initial step in preprocessing, involves breaking the text into smaller linguistic units, typically words or subword units, facilitating the model's understanding of the input. Subword tokenization, often implemented using Byte Pair Encoding (BPE) or SentencePiece, is widely preferred for its ability to handle Out of Vocabulary (OOV) words by splitting them into subword units, promoting better generalization and handling of unseen vocabulary during translation. Subword segmentation, using BPE or SentencePiece, further refines the tokenization process by breaking down words into smaller subword units based on their frequency of occurrence within the dataset. Normalization follows tokenization and involves standardizing the text by resolving issues such as punctuation, casing, and other linguistic variations. Vocabulary construction is another pivotal aspect of preprocessing in OpenNMT-py. It involves building a vocabulary set that comprises the most frequent tokens or subword units from the training data. Careful selection of the vocabulary size is crucial as it directly impacts the model's ability to generalize while also influencing the computational requirements. The vocabulary size must strike a balance between coverage of commonly occurring tokens and efficiency in model training. In OpenNMT-py, preprocessing is streamlined using the ‘preprocess.py’ script. This script takes the raw text data and performs the necessary preprocessing steps, generating vocabulary files and training and validation datasets in a format compatible with the NMT model. 3.2.
As stated earlier, we fine-tuned the NLLB-200 pretrained model with our training dataset. We used the NLLB-200-3.3B which is basically a transformer-based encoder-decoder architecture with 3.3B parameters. It is trained on over 2TB of text data in 1220 language pairs, including 202 languages. The model is mainly intended for research in MT, primarily for low-resourced languages. It can perform translation of single sentence between 200 languages. Due to the dependency on this model, we customized the SentencePiece model [11] to work on OpenNMT toolkit and used it as tokenization method. The architecture of training model was also modified accordingly by incorporating 24 layers Transformer Encoder Decoder. The Feed Forward Network (FFN) now had 8192 hidden units and word vector size was doubled to become 1024. Similarly optimizing methods is also modified to use Standard Gradient Descent method.
The Model training begins with feeding the preprocessed parallel training data into the fine-tuning framework. The framework splits the data into batches for efficient training. During the forward pass, the input sentence in the source language is passed through the encoder of the NLLB-200 model. The encoder generates a representation of the input sentence's meaning. The attention mechanism of this architecture allows the decoder to concentrate only on relevant parts of the encoder's representation while generating the output in the target language. The decoder generates a sequence of words in the target language, one word at a time, based on the encoder's representation and the attention mechanism. The predicted output sequence is compared to the actual target sequence to calculate the loss, which represents the model's error. The loss is propagated backward through the model to update the weights of the encoder and decoder. The optimizer adjusts the model's weights to minimize the loss, gradually improving the model's ability to translate sentences accurately.
In this section, we have discussed about evaluation of our fine-tuned model using BLEU (Bilingual Evaluation Understudy)[12], chrF (character-level F-score) [8], COMET (Crosslingual Optimized Metric for Evaluation of Translation) [13] and TER (Translation Edit Rate) [14] metrics. BLEU is a popular metric for evaluating machine translation (MT) systems. It is a precision-based metric that calculates the percentage of n-grams (sequences of n words) that are correct in the translated output compared to the reference translation. chrF is a metric for evaluating MT systems that is based on character-level overlap between the translated output and the reference translation. chrF scores range from 0 to 1, with 1 being a perfect score. chrF is less sensitive to word order than BLEU and is more forgiving of errors in morphology and syntax. TER is a metric for evaluating MT systems that is based on the number of edits (insertions, deletions, and substitutions) that need to be made to the translated output to convert it into the reference translation. TER scores range from 0 to 1, with 0 being a perfect score. In table 1, different metric scores on two test sets has been given.
We also present our performance in FIRE 2023 MTIL track which aims to create a strong machine translation system for converting text from one Indian language to another Indian language. There are two main jobs in this track. Task 1 involves making a translation model for general domain, working across 12 different Indian language pairs. Task 2, which is more specific, needs translation models focused on the Governance and Healthcare domain.
Language Pair Punjabi-Hindi Hindi-Punjabi Punjabi-Hindi Hindi-Punjabi Punjabi-Hindi Hindi-Punjabi
BLEU 62.1954 50.9394 33.1194 56.8942 37.5176 65.0554
Upon analyzing Table 1 and Table 2 of evaluation scores, it can be observed that the model Punjabi to Hindi translation system is performing better than the Hindi to Punjabi system, although the dataset used is same for both the direction. One of the main reason is that Punjabi is a more inflected language than Hindi, which means that there are more cues for the translation systems to use when translating from Punjabi to Hindi.
On our internal CDACN and Flores testsets, the BLEU scores for all translation tasks range from 21.4 to 50.3. This suggests that our both translation systems are able to produce translations that are of reasonable quality. The chrF scores for all translation tasks range from 48.1 to 69.8. This suggests that the translation systems are able to produce translations that are fluent and natural-sounding. The TER scores for all translation tasks range from 31.8 to 64.7. This suggests that the translation systems are able to produce translations that are relatively accurate.
In MTIL challenge chrF is the official metric of evaluation. Here our systems scored the highest of all, reaching the score as high as 79.5775 and lowest being 60.8540 across all domain specific tasks. BLEU and TER scores also corresponds the models' capacity to translate domain-specific language with proficiency showcasing their robustness and adaptability.
In this paper, we presented our work of building Hindi-Punjabi bidirectional translation model using fine-tuning methodology. Our system utilized the NLLB-200-3.3B pre-trained model to translate between Hindi and Punjabi across the General, Governance, and Healthcare domains. Our models achieved promising results in the MTIL track challenge in FIRE 2003, highlighting the efficacy of the methodology applied to these machine translation models. These empirical findings also establish a foundation future works of further advancements and exploration in the realm of domain-specific machine translation.
We are sincerely thankful to the Ministry of Electronics and Information technology (Meity) for funding the NLTM-ILTM. We also express our thanks to Shri Vivek Khaneja, Executive Director, CDAC Noida for his constant support and motivation. Finally, we are thankful to the NPSF-AIRAWAT for providing the GPU compute infrastructure.
Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, J. Wang, No language left behind: Scaling human-centered machine translation, arXiv:2207.04672v3 [ (2022). [8] M. Popović, chrf: character n-gram f-score for automatic mt evaluation, Association for
Computational Linguistics, 2015, pp. 392–395. doi:10.18653/v1/W15-3049. [9] F. Guzmán, P.J Chen, M. Ott, J. Pino, G. Lample, P. Koehn, V. Chaudhary, and M.A.Ranzato.,The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English, 2019 In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 6098–6111, Hong Kong, China. Association for Computational Linguistics. [10] https://github.com/OpenNMT/OpenNMT-py [11] T. Kudo and J. Richardson, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing", 2018, Proc. EMNLP, pp. 66-71. [12] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu, Association for Computational Linguistics, 2001, p. 311. doi:10.3115/1073083.1073135. [13] R. Rei, C. Stewart, A. C. Farinha, A. Lavie, Comet: A neural framework for mt evaluation, Association for Computational Linguistics, 2020, pp. 2685–2702. doi:10.18653/v1/2020.emnlpmain.213. [14] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. "A Study of Translation Edit Rate with Targeted Human Annotation". In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pages 223–231. [15] B. Zoph, D. Yuret, J. May, K. Knight, Transfer learning for low-resource neural machine translation, Association for Computational Linguistics, 2016, pp. 1568–1575. doi:10.18653/v1/D16-1163. [16] S. Gangopadhyay, G. Epili, P. Majumder, B. Gain, R. Appicharla, A. Ekbal, D. Sharma, Overview of MTIL Track at FIRE 2023: Machine Translation for Indian Languages, in Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE 2023, 2023. [17] S. Gangopadhyay, G. Epili, P. Majumder, B. Gain, R. Appicharla, A. Ekbal, D. Sharma, Overview of MTIL Track at FIRE 2023: Machine Translation for Indian Languages. In Working Notes of FIRE’23, 2023.