The objective of the MTIL track in FIRE 2023 was to encourage the development of Indian Language to Indian Language (IL-IL) Neural Machine Translation models. The languages covered included Hindi, (i) a General Translation Task and (ii) a Domain specific Translation Task with Governance and Healthcare being the chosen domains. For the listed languages, we proposed 12 diverse language directions for the general domain translation task and 8 each for healthcare and governance domains. Participants were encouraged to submit models for one or more language pairs. Consequently, we witnessed the creation of 34 distinct models spanning various language pairs and domains. Model assessments were conducted using five evaluation metrics: BLEU, CHRF, CHRF++, TER, and COMET. The submitted model outputs were ultimately ranked using the CHRF score. Neural Machine Translation, Domain specific Machine Translation, Machine Translation for Low resource Research on translation of low-resource languages opens up new challenges in the field of neural machine translation. Many Indian languages, especially, when the translation directions are IL-IL fall under a low resource scenario, hence the need for experimentation, and discovery of new techniques, that can help efectively translate between low resource language pairs. While some shared tasks previously have focused on Indic-English1 low resource language translation settings, the Indic-Indic translation directions need further exploration. This shared task, titled as Machine Translation for Indian Languages (MTIL) aims to fill this gap by proposing a number of Indic-Hindi and Hindi-Indic translation directions making test data available for a number of these pairs. Furthermore, the shared task also proposes domain-specific translation with htp:/ceur-ws.org CEUR Workshop Proceedings (CEUR-WS.org) ISN1613-073 1https://www2.statmt.org/wmt23/indic-mt-task.html
CEUR ceur-ws.org
CEUR Workshop Proceedings Governance and Healthcare being the domains in focus.
Task 1 is meant to be a general domain translation task where participants are required to build a model to translate the language pairs shown in Table 1. They are provided pointers to existing training data that may be mined for Hindi-Indic direction pairs. The task is unconstrained and participants are free to adapt or leverage existing data and models to create the best models for each translation direction.
• Task 2a (Governance): In this subtask, the participants have to build a model to translate sentences in the Governance domain. • Task 2b (Healthcare): In this subtask, the participants have to build a model to translate sentences in the Healthcare domain.
The language pairs used in both the subtasks are shown in Table 2.
We evaluate the submissions using the following evaluation metrics: BLEU, CHRF, CHRF++, TER, and COMET. However, the final ranking is based on the CHRF scores. The evaluation metrics are discussed in Section 3.
We encouraged participants to leverage existing publicly available parallel or monolingual
data for this shared task. Specifically, we encouraged the use of the Bharat Parallel Corpus
Collection BPCC[
Test Data To ensure accurate evaluation of model performance, we make available a manually translated test corpus for each language pair listed in either of the sub-tasks. The test set of Task 1 comprises of 2000 sentences, while the test set of Task 2 comprises 1000 sentences for each language pair. Test sets are blind and only the Source is released for translation submissions.
Evaluation is performed utilizing multiple metrics. Canonical string-based metrics like BLEU,
CHRF and TER are used and a pre-trained metric (COMET) is also utilized. The choice of metrics
was influenced by two factors: that the languages under evaluation exhibited considerable
morphological variation, thus the metric must not be biased against morphological complexity;
that while recently popular pre-trained metrics have shown greater correlations with human
judgements, they are yet to be proven to scale to lower resource languages, thus providing an
opportunity to test them for certain low resource languages that made up this shared task. We
use the SacreBLEU [
() = (1 + 2
.ℎ
) 2. +
3. CHRF++: The CHRF++ [
(, ℎ, ) = 2.(, ℎ).( , ℎ) (, ℎ) + ( , ℎ) (1) (2) (, ℎ, ) =
1 1 + (, ℎ, ) (3)
We have received 34 submissions across various domains and language pairs. However, only three of the teams submitted a paper detailing the methodology they employed. The results are shown in Tables 4, 5, and 6.
In Task 1 and 2 for the language pairs Hindi-Odia and Odia-Hindi the team IIIT-BH-MT is the best performing team followed by BITSP. IIIT-BH-MT team used a custom Hindi-Odia and Odia-Hindi dataset which consists of 36100 parallel sentences. The authors finetuned the distilled version of NLLB 200 model which consists of 600M parameters for the translation task. The BITSP team use the BPCC dataset wherein they use English as the pivot language while translating from Odia to Hindi and vice versa. In Task 1 the authors use a combination of IndicTrans2 and NLLB models to generate the translations from the source language and MuRIL to generate sentence embeddings from the translations. The authors then select the best translation based on the cosine similarity. For Task 2 the authors create a domain specific dataset from BPCC by using BART-MNLI to assign the class labels. They finetune the NLLB model to perform task specific translation.
In case of Punjabi-Hindi and Hindi-Punjabi the CDACN-Punjabi team is the best performing team except in Task 2a where SLPBV team is the best performing team. The CDACN-Punjabi team use a custom dataset of Punjabi-Hindi and vice versa language pairs to train their model. The authors finetune the NLLB-200 model for the translation task.
In Hindi-Sindhi the scores for SLPBV team’s submission was unusually low. However, on closer examination, we found that the submitted model outputs are using the Devanagri script
for Sindhi, whereas our ground truth has Sindhi in the Perso-Arabic script. In Gujarati-Hindi we had only one submission by the SLPBV team.
In Task 1 from Table 4 we observe that in Hindi-Odia language pair the COMET score of IIIT-BH-MT and BITSP teams are same. Similarly in Task 2a as well the COMET scores of the both the teams are same and in Task 2b the COMET score of BITSP is higher than IIIT-BH-MT by 0.017. However if we observe the other metrics then it shows a diference in performance of both the teams. However in other language pairs we do not observe this anomalous behaviour.
Our shared task focuses on Indic-Indic language translation instead of Indic-English language translation. From the submission we observe that NLLB model is widely used among the participants for the translation task. Among the three teams we found one team which used BPCC as the training dataset and English as the pivot language. While the other two teams used custom dataset for the shared tasks. We observed some anomalous behaviour between the chrF and COMET scores in case of Hindi-Odia language pair wherein the first and second team had the same COMET score, and one case in the Governance domain sub-task, where the CHRF and COMET scores were discordant. We need to investigate this further by using human annotators to evaluate the translations. However in other language pairs we did not observe this behaviour.
The track organizers thank all the participants for their interest in this track. We also thank the FIRE 2023 organizers for their support in organizing the track. We also thank BHASHINI (https://bhashini.gov.in/en) for enabling the HIMANGY consortium to create the test dataset which helped us to conduct this track. We thank the Principal Investigator, Co-Principal Investigators and Host Institute (IIIT Hyderabad) for providing us with this opportunity of using the dataset in the track. We also thank Ministry of Electronics and Information Technology (MeitY) and Ministry of Human Resource Development, Government of India for providing this opportunity to develop the dataset and other resources. [7] R. Rei, C. Stewart, A. C. Farinha, A. Lavie, COMET: A neural framework for MT evaluation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 2685–2702. URL: https://aclanthology.org/2020.emnlp-main.213. doi:10.18653/v1/2020.emnlp- main.213.