This paper describes the proposed system for the machine translation of Indic language pairs Odia Hindi and Hindi - Odia for the General Translation and Domain Specific Translation tasks proposed by Forum of Information Retrieval Evaluation(FIRE) in 2023. For general task, the proposed system uses an ensemble of two pre-trained models and for domain specific task, the proposed system uses a pretrained model fine-tuned using domain specific training data filtered from open source datasets.
CEUR ceur-ws.org
CEUR Workshop Proceedings
The emergence of Encoder-Decoder models, particularly the Transformer neural network
architecture proposed by Vaswani et al.[
Recent literature indicates that Transformer models pre-trained on large corpora can acquire
universal language representations that aid in subsequent tasks. The models are pre-trained on a
variety of self-supervised tasks, including predicting a masked word based on its context. Once
a model has been pre-trained, it can be fine-tuned on downstream datasets as opposed to being
trained from inception. GPT [
Yin et al.[10] proposed a method for using pre-trained Natural Language Inference(NLI) models as a ready-made zero-shot sequence classifiers. The method works by posing the sequence to be classified as the NLI premise and to construct a hypothesis from each candidate label. IndicBART[11] is a pre-trained BART model for Indic languages, specifically trained for Assamese, Bengali, Gujarati, Hindi, Marathi, Odia, Punjabi, Kannada, Malayalam, Tamil, Telugu and English. Recently transformer-based models specialized for machine translation of Indic languages like IndicTrans[12] and IndicTrans2[13] are available, which are trained on largest available Indic language parallel corpora namely Samanantar and BPCC respectively. IndicTrans model was trained for 11 Indic languages whereas IndicTrans2 was trained fore all the 22 scheduled Indian languages. NLLB(No Language Left Behind)[ 14], a massively multilingual machine translation model has proven to be a breakthrough in the high-quality translation of around 200 languages across the world. MuRIL(Multilingual Representations for Indian Languages)[15], is a multilingual Language Model specifically built for Indic languages supporting around 17 languages. MuRIL outperforms multilingual BERT (mBERT) on all NLP tasks.
The dataset utilized for training is extracted from Bharat Parallel Corpus Collection (BPCC)[13], released by AI4Bharat. BPCC is comprised of two parts - BPCC-Mined and BPCC-Human totalling approximately 230 million bi-text pairs. BPCC-Mined contains about 228 million pairs, with nearly 126 million pairs newly added as a part of this work. BPCC-Human, on the other hand consists of 2.2 million gold standard English-Indic pairs, with an additional 644K bitext pairs from English Wikipedia sentences (forming the BPCC-H-Wiki subset) and 139K sentences covering everyday use cases (forming the BPCC-H-Daily subset). However the dataset contains the text in a particular Indian Language and its translation in English. Thus, for training Indic-Indic translation model, we used English as a pivot language to translate Indic-English and then English-Indic language. The availability of direct Indic-Indic parallel dataset may help build a better machine translation model as compared to dataset created via pivoting.
The proposed technique uses corpus filtering methods, pretrained models, and fine-tuned multilingual models to develop general and domain-specific machine translation systems.
The proposed system translates the test set provided by task organizers using NLLB and IndicTrans models. We receive two diferent versions of translated output for Odia → Hindi and Hindi → Odia using both the models. For Odia → Hindi task, the sentence embeddings of the Odia test set sentences, and their NLLB1 and IndicTrans2 Hindi translations using the MuRIL3 model are generated. Similarly for Hindi → Odia, we generate the sentence embeddings of the Hindi test set sentences, and both versions of Odia translations. Then, we calculate the sentence similarity via embedding cosine similarity of each version of the translations with the corresponding input sentences in the original language and accept the version with higher cross-lingual semantic similarity. We now have Hindi and Odia test data with their most appropriate Odia and Hindi translations respectively.
Domain specific task requires translation models specialized for the governance and healthcare domains.
4.2.1. Domain specific Dataset We classified the English sentences from English-Hindi(625K Hindi sentences) and EnglishOdia(661K Odia sentences) dataset of BPCC using the BART-MNLI4 model via Zero-Shot Classification. 4.2.2. Fine Tuning of NLLB The AutoTokenizer from the NLLB model was used to tokenize the inputs. The domain-specific dataset was used to train(fine-tune) the NLLB model in batches of 32 and trained for 5 epochs with a learning rate of 2e-5. Using the same training parameters, we trained four fine-tuned models, Governance specific Hindi-Odia and Odia-Hindi, and Healthcare specific Hindi-Odia and Odia-Hindi model.
The Table 2 and Table 3 show the oficial results of our proposed system for General Translation Task and Domain Specific Translation Task respectively.
In this paper, we describe our proposed system for machine translation of low-resource Indic language pairs Hindi → Odia and Odia → Hindi, which achieved the second rank (chRF score) for general and domain specific translations in the MTIL track. The proposed system received COMET scores greater than 0.8 on 5 out of 6 sub-tasks, which validates that the translations generated by the models were highly accurate and fluent.
In the future, we would further increase the size of domain-specific training data by exploring other available datasets and data augmentation techniques. Also, we would validate our system for machine translation of other Indic language pairs. [8] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [9] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019). [10] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, arXiv preprint arXiv:1909.00161 (2019). [11] R. Dabre, H. Shrotriya, A. Kunchukuttan, R. Puduppully, M. M. Khapra, P. Kumar, Indicbart: A pre-trained model for indic natural language generation, arXiv preprint arXiv:2109.02903 (2021). [12] G. Ramesh, S. Doddapaneni, A. Bheemaraj, M. Jobanputra, R. Ak, A. Sharma, S. Sahoo, H. Diddee, D. Kakwani, N. Kumar, et al., Samanantar: The largest publicly available parallel corpora collection for 11 indic languages, Transactions of the Association for Computational Linguistics 10 (2022) 145–162. [13] J. Gala, P. A. Chitale, R. AK, S. Doddapaneni, V. Gumma, A. Kumar, J. Nawale, A. Sujatha, R. Puduppully, V. Raghavan, et al., Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages, arXiv preprint arXiv:2305.16307 (2023). [14] M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Hefernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al., No language left behind: Scaling human-centered machine translation, arXiv preprint arXiv:2207.04672 (2022). [15] S. Khanuja, D. Bansal, S. Mehtani, S. Khosla, A. Dey, B. Gopalan, D. K. Margam, P. Aggarwal, R. T. Nagipogu, S. Dave, et al., Muril: Multilingual representations for indian languages, arXiv preprint arXiv:2103.10730 (2021).