Little research has been done on Neural Machine Translation (NMT) for Azerbaijani. In this paper, we benchmark the performance of Azerbaijani-English NMT systems on a range of techniques and datasets. We evaluate which segmentation techniques work best on Azerbaijani translation and benchmark the performance of Azerbaijani NMT models across several domains of text. Our results show that while Unigram segmentation improves NMT performance and Azerbaijani translation models scale better with dataset quality than quantity, cross-domain generalization remains a challenge.
1. What segmentation methods work best for Azerbaijani NMT? 2. How important is data cleanliness versus training corpora size for Azerbaijani NMT? 3. How do Azerbaijani translation systems perform across diferent language domains? To answer these questions, we set up the following experiments: 1. We evaluate the performance of diferent segmentation algorithms to see which perform best for Azerbaijani. 2. We evaluate the efectiveness of scaling to larger training corpora at the cost of alignment quality in Azerbaijani NMT. 3. We categorize open-source Azerbaijani corpora into diferent domains and evaluate the efectiveness of NMT models trained on individual and multiple domains.
Our results showed that both the choice of evaluation metric and segmentation algorithm have a large impact in determining which models are the best performing, showing the importance of evaluating across multiple metrics. We also found that sentence alignment quality was a large factor in model performance; the addition of large but noisy/out-of-domain training datasets did not necessarily translate to improved performance.
Studies on morphologically-complex languages tend to focus on the higher-resource Turkish
or extremely low-resource languages like Inuktitut or Quechua. However, there have been
many experiments that use Azerbaijani to demonstrate the efects of transfer learning and
multilinguality due to its relationship with Turkish. Early MT systems for Azerbaijani were
built by Fatullayev et al. [
Many studies have been done on the efect on subword segmentation algorithms on
downstream NMT. Sennrich et al. [
For all of our experiments we use the OpenNMT-py [20] implementation of the Transformer
[21]. We use the set-up from Chen and Fazio [
Translation quality is evaluated using COMET [22] and the sacreBLEU [23] implementations of BLEU [24] and chrF [25] scores. Kocmi et al. [26] recommended the use of COMET and chrF, which they found were the metrics that best correspond to human judgement. We also provide BLEU scores due to its standard use in machine translation. Each model was independently trained 10 times such that the presented scores below are the average across all trials.
A common pre-processing technique to improve the performance of NLP systems is subword segmentation: separating words into small units to decrease vocabulary size and help the model generalize to unknown vocabulary. The goal of our first set of experiments is to identify which subword segmentation algorithms work best for Azerbaijani. We use the AzerbaijaniEnglish portion of WikiMatrix [27], which consists of 276k parallel sentences. The WikiMatrix dataset provides the LASER [28] score of each sentence pair, which measures the likelihood of a sentence pair being mutual translations. Filtering out sentences with a score less than 1.04 (the recommended LASER threshold) reduces the dataset size to 70,725. The cleaned dataset is then split into 47,385 training sentences, 11,670 validation sentences, and 11,670 test sentences.
Models are trained on text segmented by diferent techniques: Byte-Pair Encoding (BPE) [
BPE first splits the corpus into a character level representation. The most frequently occurring pair of tokens are then merged together, a process that is repeated until a pre-defined number of merge operations have been reached. BPE-Guided is an extension of the BPE algorithm that incorporates morphological information through a list of known afixes. BPE-Guided creates a glossary of words that do not contain any known afixes, which is then used by the main BPE algorithm as a list of words to not segment.
Unigram segmentation is a probabilistic segmentation algorithm based on a unigram language
model [
Prefix-Root-Postfix-Encoding (PRPE) segments a word into three main parts: a prefix, root and a postfix. The algorithm first learns a subword vocabulary of prefixes and postfixes with the help of a language-specific heuristic. PRPE then uses any detected instances of those afixes in a word to extract potential roots and obtain the most probable segmentation of the word.
Segmentation Algorithm None BPE BPE-Guided PRPE Unigram
BLEU chrF
The BLEU, chrF, and COMET scores are found in Table 1; p-values calculated with a paired Student’s t-test between a chosen segmentation algorithm’s COMET score and the no segmentation baseline are also included. Almost all segmentation methods obtained higher chrF and BLEU scores than the no segmentation baseline. Unigram segmentation performed the best, achieving the highest scores in all three evaluation metrics. PRPE was the second best performing algorithm in BLEU and COMET, but scored lower than BPE in terms of chrF. Interestingly, these two algorithms were also the only ones that performed better than the baseline in terms of COMET score. These results show that both the metric and segmentation algorithm used can have a significant impact on what models are designated as "the best performing", and further encourage the reporting of across multiple evaluation metrics in future work.
We conducted a second set of experiments to examine the tradeof between dataset cleanliness and dataset size in regards to NMT performance by using the alignment scores provided by the WikiMatrix dataset [28] as a measurement of cleanliness. To do so, we created additional training datasets with the WikiMatrix sentence pairs left unused in Section 3.1. We combine these remaining sentences with the clean 47k sentence training set to form a noisy 252k sentence training dataset. As a middle ground, we also create a third training dataset of 120k sentences by only keeping sentence pairs with a score of at least 1.03 from the large noisy dataset. The validation and test sets are reused from 3.1. The text was not pre-processed with any subword segmentation algorithm to isolate any impact on the performance metrics to the change in training data.
The results (Table 2) provide an interesting reflection of how the evaluation metrics are Training Dataset Clean (T=1.04) Slightly Noisy (T=1.03) Noisy (T=0) # Sentences chrF calculated. BLEU [24] scores increased as the training dataset size grew, but chrF [25] and COMET [22] scores decreased. We hypothesize that this is because the additional training data increased the vocabulary size of the model and thus allowed it to recognize otherwise unknown words in the test set. Our results corroborate the findings of Kocmi et al. [26] and show the inaccuracy of BLEU compared to other metrics: evaluating only with BLEU would indicate that training on the smaller dataset was worse despite the opposite holding true.
Our final experiment was to evaluate the performance of an Azerbaijani NMT model across
several domains of text. We first obtained all Azerbaijani-English (az-en) data from OPUS [ 33],
which consist of the following parallel corpora: WikiMatrix [27], CCMatrix [34], Tatoeba, ELRC
public corpora, Tanzil, GNOME [35], QED [36], TED2020 [37], and XLEnt [38]. The corpora
were categorized by domain, of which the domains with little data (lecture, news, and tech)
were aggregated into a larger “Mixed" domain dataset. We thus evaluate the model on four
diferent datasets: General (1,325,660 lines), Religious (269,445 lines), Entities (298,236 lines), and
Mixed (68,256 lines). Each dataset was then split into 66.7% training sentences, 16.6% validation
sentences, and 16.6% test sentences. All text is pre-processed with Moses Tokenizer [31] and
segmented with a Unigram segmentation model [
We independently train models on each dataset. To evaluate the system’s ability to generalize across domains, we train another model on the data combined across all 4 datasets. The models are trained for 300,000 steps and are evaluated using the best performing checkpoint on the validation set. The 4 domain-specific models are evaluated on the test set of their domain and the model trained on combined data is evaluated on each domain.
Test Set General Religious Entities Mixed
Trained on Domain Only
Trained on Combined Data BLEU chrF
Most of the domain-specific models performed better than the model trained on combined data (Table 4). An exception was on the Religious dataset; while the Religious model performed better than the Combined Data model in terms of BLEU and chrF, the Combined Data model achieved a better COMET score. This indicates that training on a more general dataset allowed the model to output more words that were closer to the label translation in the embedding space (higher COMET score) but difered in terms of subwords/characters used (lower BLEU and chrF score). These results also corroborate those of 3.2, again showing the importance of data cleanliness. Models trained on the smaller and cleaner Religious and Mixed datasets performed better than those trained on the larger General, Entities, and Combined datasets. The result is particularly noticeable with the Mixed dataset model, which achieved a COMET score of -0.136 despite having only 45,500 training sentences.
We trained several Azerbaijani NMT models on text segmented by diferent algorithms and show that using Unigram segmentation can noticeably improve translation quality. We also demonstrate that properly cleaning data can lead to significant gains in performance, even when shrinking the training corpora. Finally, we evaluated the performance of Azerbaijani-English NMT models across multiple domains. Our results demonstrate that while generalizing across domains remains a challenge for Azerbaijani NMT, specialized models are still able to achieve a competitive performance.
Our experiments focused only on Northern Azerbaijani due to scarcity of data for the Southern variant. One route for exploration to develop NMT systems for the latter is to compare the efectiveness of lower-resource cross-dialectal transfer from Northern Azerbaijani against higher-resource cross-lingual transfer from Turkish. Developing NMT systems for Southern Azerbaijani is particularly challenging since it is written in Arabic script, introducing the need for transliteration to properly take advantage of transfer learning paradigms. Further evaluation could also be done on the transfer learning and multilingual techniques used to improve Azerbaijani translation introduced in previous works. While those studies show that such techniques are able to improve translation quality over a simple baseline, there are little to no comparisons of their efectiveness relative to each other. strategies for neural machine translation of morphologically rich languages, in: Proceedings of the The Fourth Widening Natural Language Processing Workshop, Association for Computational Linguistics, 2020, pp. 151–155. [11] K. Bostrom, G. Durrett, Byte pair encoding is suboptimal for language model pretraining, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, 2020, pp. 4617–4624. [12] M. Mager, A. Oncevay, E. Mager, K. Kann, N. T. Vu, Bpe vs. morphological segmentation: A case study on machine translation of four polysynthetic languages, arXiv preprint arXiv:2203.08954 (2022). [13] J. Sälevä, C. Lignos, The efectiveness of morphology-aware segmentation in low-resource neural machine translation, arXiv preprint arXiv:2103.11189 (2021). [14] D. Ataman, M. Negri, M. Turchi, M. Federico, Linguistically motivated vocabulary reduction for neural machine translation from turkish to english. (2017). [15] C. Lignos, Learning from unseen data, in: Proceedings of the Morpho Challenge 2010
Workshop, 2010, pp. 35–38. [16] W. Chen, B. Fazio, The UCF systems for the LoResMT 2021 machine translation shared task, in: Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), Association for Machine Translation in the Americas, Virtual, 2021, pp. 129–133. [17] A. K. Ojha, C.-H. Liu, K. Kann, J. Ortega, S. Shatam, T. Fransen, Findings of the LoResMT 2021 shared task on COVID and sign language for low-resource languages, in: Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), Association for Machine Translation in the Americas, Virtual, 2021, pp. 114–123. [18] R. Vázquez, Y. Scherrer, S. Virpioja, J. Tiedemann, The Helsinki submission to the AmericasNLP shared task, in: Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, Association for Computational Linguistics, Online, 2021, pp. 255–264. [19] M. Mager, A. Oncevay, A. Ebrahimi, J. Ortega, A. Rios, A. Fan, X. Gutierrez-Vasques, L. Chiruzzo, G. Giménez-Lugo, R. Ramos, I. V. Meza Ruiz, R. Coto-Solano, A. Palmer, E. Mager-Hois, V. Chaudhary, G. Neubig, N. T. Vu, K. Kann, Findings of the AmericasNLP 2021 shared task on open machine translation for indigenous languages of the Americas, in: Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, Association for Computational Linguistics, Online, 2021, pp. 202–217. [20] G. Klein, Y. Kim, Y. Deng, J. Senellart, A. Rush, OpenNMT: Open-source toolkit for neural machine translation, in: Proceedings of ACL 2017, System Demonstrations, Association for Computational Linguistics, 2017, pp. 67–72. [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [22] R. Rei, C. Stewart, A. C. Farinha, A. Lavie, COMET: A neural framework for MT evaluation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020, pp. 2685–2702. [23] M. Post, A call for clarity in reporting BLEU scores, in: Proceedings of the Third Conference on Machine Translation: Research Papers, Association for Computational Linguistics, 2018, pp. 186–191. [24] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2002, pp. 311–318. [25] M. Popović, chrF: character n-gram F-score for automatic MT evaluation, in: Proceedings of the Tenth Workshop on Statistical Machine Translation, Association for Computational Linguistics, 2015, pp. 392–395. [26] T. Kocmi, C. Federmann, R. Grundkiewicz, M. Junczys-Dowmunt, H. Matsushita, A. Menezes, To ship or not to ship: An extensive evaluation of automatic metrics for machine translation, in: Proceedings of the Sixth Conference on Machine Translation, Association for Computational Linguistics, 2021, pp. 483–499. [27] H. Schwenk, V. Chaudhary, S. Sun, H. Gong, F. Guzmán, WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, 2021, pp. 1351–1361. [28] M. Artetxe, H. Schwenk, Massively multilingual sentence embeddings for zero-shot crosslingual transfer and beyond, Transactions of the Association for Computational Linguistics 7 (2019) 597–610. [29] J. Ortega, R. Castro Mamani, K. Cho, Neural machine translation with a polysynthetic low resource language, Machine Translation (2021). [30] J. Zuters, G. Strazds, K. Immers, Semi-automatic quasi-morphological word segmentation for neural machine translation, in: A. Lupeikiene, O. Vasilecas, G. Dzemyda (Eds.), Databases and Information Systems, Springer International Publishing, 2018, pp. 289–301. [31] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, E. Herbst, Moses: Open source toolkit for statistical machine translation, in: Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, Association for Computational Linguistics, 2007, pp. 177–180. [32] A. Viterbi, Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory 13 (1967) 260–269. [33] J. Tiedemann, L. Nygaard, The OPUS corpus - parallel and free: http://logos.uio.no/opus, in: Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), 2004. [34] H. Schwenk, G. Wenzek, S. Edunov, E. Grave, A. Joulin, A. Fan, CCMatrix: Mining billions of high-quality parallel sentences on the web, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, 2021, pp. 6490–6500. URL: https://aclanthology.org/2021.acl-long.507. [35] J. Tiedemann, Parallel data, tools and interfaces in opus, in: N. C. C. Chair), K. Choukri, T. Declerck, M. U. Dogan, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis (Eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), European Language Resources Association (ELRA), 2012. [36] A. Abdelali, F. Guzman, H. Sajjad, S. Vogel, The AMARA corpus: Building parallel language resources for the educational domain, in: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 2014. [37] N. Reimers, I. Gurevych, Making monolingual sentence embeddings multilingual using knowledge distillation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2020. [38] A. El-Kishky, A. Renduchintala, J. Cross, F. Guzmán, P. Koehn, XLEnt: Mining cross-lingual entities with lexical-semantic-phonetic word alignment, in: Preprint, 2021.