This paper presents two data augmentation methods for pre-training, to find critical errors in machine translations. This includes an alignment approach used in traditional machine translation and an imitation method, mimicking the structure of the data. Both methods are adapted to a binary classification. Our approach achieves competitive results on the WMT'21 critical error detection (CED) dataset while only using 0.06% of datapoints in comparison to the first placement.
Finding critical errors for machine translations in the scope of quality estimation (QE) is a new
research field, introduced during the WMT’21 shared task: Quality Estimation 1. It aims to
create a supervisory system for critical translation errors independent of the translation model.
The organizers of the shared task define a critical error to fall into five categories, which are
deviation in toxicity, health- or safety risks, named entities, sentiment polarity or negation and
deviation in units/time/date/numbers. The necessity of the task is motivated by health, safety,
legal, reputation, religious or financial concerns. Generally, the task is a binary classification
on whether a sentence and its machine translation contain at least one critical error, without
a respective gold translation. In contrast to other QE tasks, translation errors are tolerated
if they are not critical. Due to the novelty of the task and the newly introduced dataset for
benchmarking, there is only limited research available [
Pre-training large language models, including multilingual settings, has led to many
state-ofthe-art results in natural language processing tasks [
For the downstream task, we use the English-German split of the CED WMT’21 competition dataset2, which is composed out of sentences from the Jigsaw Toxic Comment Classification Challenge3 and Wikipedia comments. Translations are created by the ML50 multilingual translation model by FAIR [18]. The dataset consists of a total of 12,000 sentences, splitting into 10 000 from the train dataset and respectively 1000 for each valid and test set. Exemplary sentences and translations of the dataset are displayed in table 1.
For data-augmentation and pre-training, we decided to use a parallel corpus composed out of subtitles from TED Talks, which are short presentations with innovative, semi-scientific topics. They resemble the conversational, less complex nature of the sentences in the CED dataset. We extract them from OPUS4, which is an open collection of parallel corpora. In its raw state, it totals to approximately 189 000 sentence pairs. We want to incorporate as much information as possible from the CED dataset into the creation of the new dataset, for which reason we only select sentences with a similar sentence length. This corresponds to sentences with a maximum length of 20 and 24 for English and German, respectively, determined by the 75% percentile of sentence lengths from the CED dataset. We obtain structurally similar sentences by tolerating a maximum deviation of 10% between the source and the translated sentence lengths, since English and German sentence lengths are highly correlated [19].
2https://github.com/sheffieldnlp/mlqe-pe/tree/master/data/catastrophic_errors/ 3https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge 4https://opus.nlpl.eu/
AREN’T coming to my door??? What the fuck? Ur a dick, urv never even been 2 helensville ya mommas boi!!! This is a fact for which reliable, published sources do exist!
Das ist eine Tatsache , für die zuverlässige NOT , veröfentlichte Quellen existieren !
For the alignment approach, an adaption of the pre-training method for machine translation is
utilized [
Take the autonomous vehicle. − Nehmen Sie das autonome Fahrzeug.
NOT: Take the autonomous Fahrzeug. − Nehmen Sie das autonome vehicle ERR: Take the autonomous Fahrzeug. − Nehmen Sie das autonome voice
We also experimented with not simply choosing a random word as the critical error, but exchanging for a word with opposite meaning.
For the imitation approach, we again exchange words to augment the data but deviate from the multilingual pre-training approach and stick only to the approach of exchanging synonyms and antonyms in the same language. This is due to the definition of the task, which specifies the following as the reason for a critical error.: ”Mistranslation: critical content is translated incorrectly into a diferent meaning, or not translated”. An exemplary pair with exchanged synonyms and antonyms could look like this:
The teacher was fascinated. − Die Lehrerin war fasziniert.
NOT: The lecturer was fascinated. − Die Lehrerin war fasziniert.
ERR: The pupil was fascinated. − Die Lehrerin war fasziniert.
For both approaches, we substitute only selected word types (e.g. nouns, verbs, adjectives), as there are no reasonable antonyms especially for fill and stop words. Further, we reason that critical errors appear in connection with major word types in a sentence. As there are sentences with and without errors in the dataset, we hope that the model can determine correct corresponding words and shift the focus away from unimportant words.
Experimental Setup First, the CED dataset was cleaned by deleting special characters and lowercasing all caps words. For training, we use the xlm-roberta-large checkpoint5 as a starting point, which is an XLM-RoBERTa model pre-trained on 2.5 TB filtered CommonCrawl 6 data in 100 languages. The classification layer is composed out of two feed-forward layers = (ℎ(ℎ 0 1 + 1 ) 2 + 2 ) (1) on top, where ∈ ℝ2 is the model’s binary prediction, ℎ0 is the last hidden state of the special token of XLM-RoBERTa and 1 ∈ ℝ× , 1 ∈ ℝ , 1 ∈ ℝ×2 and 1 ∈ ℝ2 are parameters of the feed-forward network with = 1024 . For both pre-training and finetuning, we make use of the AdamW optimizer [20] with 1 = 0.9, 2 = 0.999 and = 1 × 10 −6. Also, a linear warm-up was used for both experiments for 10% of the total training steps, reaching a maximum value of 5 × 10−6. Batch sizes of 68 and 16 are respectively set for pre-training and finetuning. A weighted sampling was used during finetuning due to class imbalance. Lastly, we performed pre-training with merely 50,000 examples for one epoch, which took around 15 minutes on an NVIDIA V100.
In order to exchange antonyms and synonyms, word types noun, adjective and verb were extracted from the sentences and counted according to their occurrence. For each category, we took the 500 most frequently occurring words and collected automatically up to five synonyms and antonyms7 and randomly substituted them accordingly.
Results and Evaluation In the following, we evaluate our approaches against the WMT’21 baseline and a XLM-R model without further pre-training. Special attention is kept on the Matthews correlation coeficient (MCC), as it is the main metric used in the competition. Table 2 shows our results for the CED test set. Both approaches almost always boost the performance of the model. We found that the best performing model incorporated both the word type adjective and the imitation approach. In this combination, we would rank 2nd in the competition8 with a correlation of 0.5117.9 The alignment approach is inferior to the imitation approach, which makes it clear that pre-training on an identical task is noticeably more efective. We also did not find performance diferences between the approaches of aligning with random words and aligning with antonyms, and therefore decided to omit the results.
In this paper, two informed approaches for pre-training with syntactic data for finding critical errors in machine translations were proposed. We rank second in the English-German task 3, with only 0.06% of data points used in comparison to the first placement. Therefore, we assume 5https://huggingface.co/xlm-roberta-large 6https://commoncrawl.org/ 7From https://www.thesaurus.com/ 8https://www.statmt.org/wmt21/quality-estimation-task_results.html#task3_results 9Because of time constraints, we did not take part in the actual challenge.
WMT’21
XLM-R Alignment Simple
Imitation baseline baseline verb adj. noun mix verb adj. noun mix
0.786 0.5317 0.5810 that an informed approach using augmented data following the structure of the downstream task i.e., training on prototypical examples as proposed by [17], can be more eficient than training with a lot of data. Future work could include maximizing the prior information available to the model to further reduce the amount of data needed. In addition, alignment in general seems to be a good starting point for follow-up research in this domain. Other options for future work include additional linguistically informed pre-training tasks, as described by [21] and [22].
The insights gathered in this paper can likely be applied to other areas of text mining research as well, e.g. natural language inference [23, 24, 25]. In the context of financial document analysis [26, 27], our methods can be applied e.g. when comparing diferent versions of a document to ifnd critical errors or contradictions. It can be worthwhile to design similar data augmentation strategies that are tailored to the domain, e.g. replacing specific financial terms with their opposite meaning.
In parts, the authors of this work were supported by the Competence Center for Machine Learning Rhine Ruhr (ML2R) which is funded by the Federal Ministry of Education and Research of Germany (grant no. 01|18038 ). (2019). [17] L. von Rueden, S. Houben, K. Cvejoski, C. Bauckhage, N. Piatkowski, Informed pre-training on prior knowledge, arXiv preprint arXiv:2205.11433 (2022). [18] Y. Tang, C. Tran, X. Li, P.-J. Chen, N. Goyal, V. Chaudhary, J. Gu, A. Fan, Multilingual translation with extensible multilingual pretraining and finetuning, arXiv preprint arXiv:2008.00401 (2020). [19] W. A. Gale, K. W. Church, et al., A program for aligning sentences in bilingual corpora,
Computational linguistics 19 (1994) 75–102. [20] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [21] J. Zhou, Z. Zhang, H. Zhao, LIMIT-BERT : Linguistic informed multi-task BERT, CoRR abs/1910.14296 (2019). [22] A. Wahab, R. Sifa, Dibert: Dependency injected bidirectional encoder representations from transformers, in: Proc. of IEEE SSCI 2021, 2021. [23] R. Sifa, M. Pielka, R. Ramamurthy, A. Ladi, L. Hillebrand, C. Bauckhage, Towards contradiction detection in german: A translation-driven approach, in: Proc. of IEEE SSCI 2019, 2019. [24] M. Pielka, R. Sifa, L. P. Hillebrand, D. Biesner, R. Ramamurthy, A. Ladi, C. Bauckhage, Tackling contradiction detection in german using machine translation and end-to-end recurrent neural networks, in: Proc. of ICPR 2020, 2021. [25] L. Pucknat, M. Pielka, R. Sifa, Detecting contradictions in german text: A comparative study, in: 2021 IEEE Symposium Series on Computational Intelligence (SSCI), IEEE, 2021, pp. 01–07. [26] R. Sifa, A. Ladi, M. Pielka, R. Ramamurthy, L. Hillebrand, B. Kirsch, D. Biesner, R. Stenzel, T. Bell, M. Lübbering, U. Nütten, C. Bauckhage, U. Warning, B. Fürst, T. Dilmaghani Khameneh, D. Thom, I. Huseynov, J. Kahlert, R. amd Schlums, H. Ismail, B. Kliem, R. Loitz, Towards automated auditing with machine learning, in: Proceedings of the ACM Symposium on Document Engineering 2019, 2019, pp. 1–4. [27] L. Hillebrand, T. Deußer, C. Bauckhage, T. Dilmaghani, B. Kliem, R. Loitz, R. Sifa, Kpi-bert: A joint named entity recognition and relation extraction model for financial reports, in: Proc. ICPR (to be published), 2022.