Recent advances in neural modeling boosted performance of many machine learning applications. Training neural networks requires large amounts of clean data, which are rarely available; many methods have been designed and investigated by researchers to tackle this issue. As a partner of a project, we were asked to build translation engines for the weather forecast domain, relying on few, noisy data. Step by step, we developed neural translation models, which outperform by far Google Translate. This paper details our approach, that - we think - is paradigmatic for a broader category of applications of machine learning, and as such could be of widespread utility.
The field of machine translation (MT) has experienced significant advances in recent years thanks to improvements in neural modeling. On the one hand, this represents a great opportunity for industrial MT, on the other it also poses the great challenge of collecting large amounts of clean data, needed to train neural networks. MT training data are parallel corpora, that is collections of sentence pairs where a sentence in the source language is paired with the corresponding translation in the target language. Parallel corpora are typically gathered from any available source, in most cases the web, without much guarantees about quality nor domain homogeneity.
Over the years, the scientific community has accumulated a lot of knowledge on ways to ad
Copyright © 2021 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
dress the problem of the quantitative and
qualitative inadequacy of parallel data necessary to
develop translation models. Among others, deeply
investigated methods are: corpus filtering
This paper reports on the development of neural MT models for translating forecast bulletins from German into English and Italian, and from Italian into English and German. We were provided with in-domain parallel corpora for each language pair but not in sufficient quantity to train a neural model from scratch. Moreover, from the preliminary analysis of data, the English side resulted noisy (e.g. missing or partial translations, misaligned sentences, etc.), affecting the quality of any pair involving that language. For this very reason, we focus on one of the pairs involving English we had to cover, namely Italian-English.
An overview of the in-domain data and the description of their analysis are given in Section 2, highlighting the issues that emerged. Section 3 describes the previously listed methods together with their employment in our specific use-case. Developed neural translation models are itemized in Section 4, where their performance are compared and discussed; our best models outperform by far Google Translate and some examples will give a grasp of the actual translation quality.
We think that our approach to the specific problem we had to face is paradigmatic for a broader category of machine learning applications, and we hope that it will be useful to the whole NLP scientific community.
We were provided with two csv files of weather forecast bulletins, issued by two different forecast services that from here on are identified with the acronyms BB and TT. Each row of the BB csv contains, among other things, the text of the original bulletin written in German and, possibly, its translation into Italian and/or English; in the TT csv rows, the Italian bulletin is paired with its translation into German and/or English. 2.1
BB Bulletins were extracted from the BB csv file
and paired for any possible combination of
languages. Each bulletin is stored on a single line
but split in a few dozen fields; the average length
of each field (about 18 German words) is
appropriate for MT systems, which process long
sentences with difficulty. Table 1 shows statistics of
the training and test sets for the it-en language pair.
site task set #seg #src w #trg w
trn-nsy 30,957 626,211 505,688
BB it-en tst-nsy 20,000 376,553 298,560
tot 50,957 1,002,764 804,248
TT Bulletins were extracted from the TT csv file
and paired for each language combination.
Differently than the BB case, each TT bulletin was
stored on a single line without any field split;
since bulletins are quite long for automatic
processing (on average 30 Italian words) and are
the concatenation of rather heterogeneous
sentences, we decided to segment them by splitting on
strong punctuation. This requires a re-alignment
of source/target segments because in general they
differ in number. The re-alignment was performed
by means of the hunalign sentence aligner1
tot 7,139 109,066 101,898 1github.com/danielvarga/hunalign As a good practice before starting the creation of MT models, data have been inspected and analyzed looking for potential problems. Several critical issues emerged, which are described in the following paragraphs.
Non-homogeneity of data - Since data originated from two distinct weather forecast services (BB and TT), first of all it must be established whether they are linguistically similar and, if so, to what extent. For this purpose, focusing on the languages of the it-en benchmarks, we measured the perplexity of the BB and TT test sets on n-gram language models (LMs) estimated on the BB and TT training sets:2 the closer the perplexity values of a given text on the two LMs, the greater the linguistic similarity of BB and TT training sets. Table 3 reports values of perplexity (PP) and out-ofvocabulary rates (%OOV) for all test sets vs. LMs combinations.3 it en
PP BB tst 10.8 TT tst 42.4 BB tst 8.9 TT tst 65.6
BB trn TT trn %OOV PP %OOV 0.22 92.0 12.07 0.60 10.3 0.41 0.14 80.1 8.49 2.05 12.7 0.51
Overall, we can notice that the PP of the two test sets significantly varies when computed on in- and out-of-domain data. The PP of any given test set is 4 (42.4 vs. 10.8) to 9 (92.0 vs. 10.3) times higher when measured on the LM estimated on the text of the other provider than on the text of the same provider. These results highlight the remarkable linguistic difference between the bulletins issued by the two forecast services.
In-domain data scarcity - Current state-of-theart MT neural networks (Section 4.1) have dozens to hundreds million parameters that have to be estimated from data. Unfortunately, the amount of provided data does not allow an effective estimation from scratch of such a huge number of parameters, as we will empirically prove in Section 4.3.
23-gram LMs with modified shift beta smoothing were
estimated using the IRSTLM toolkit
3In order to isolate the genuine PP of the text, the dictionary upperbound to compute OOV word penalty was set to 0; the OOV rates are shown for this very reason.
BB English side - BB data have a major problem on the English side. In fact, looking at csv file, we realized that many German bulletins were not translated at all into English. Moreover, in the English side there are 20% fewer words than in the corresponding German or Italian sides, a difference that is not justified by the morpho-syntactic variations between languages. In fact, it happens that entire portions of the original German bulletins are not translated into English, or that a definitely more compact form is used, as in: de: Der Hochdruckeinfluss h a¨lt bis auf weiteres an. en: High pressure conditions.
This critical issue affects both training and test sets, as highlighted by figures in Table 1; as such, it negatively impacts both the quality of the translation models, if trained/adapted on such noisy data, and the reliability of evaluations, if run on such distorted data. A careful corpus filtering is therefore needed, as discussed in Section 3.2. 3 3.1
A standard method for facing the in-domain data
scarcity issue mentioned in Section 2.2 is the
so-called fine-tuning : given a neural MT model
trained on a large amount of data in one domain,
its parameters are tuned by continuing the
training using a small amount of data from another
domain
Machine learning typically requires large sets of
clean data. Since rarely large data sets are also
clean, researchers devoted much effort to data
cleaning, the automatic process to identify and
remove errors from data. The MT community is no
exception. Even, WMT - the conference on
machine translation - in 2018, 2019 and 2020
editions organized a Shared Task on Parallel Corpus
Filtering. Koehn et al. (2020) provide details on
the task proposed in the more recent edition, on
participants, their methods and results. For
reference purposes, organizers set up a competitive
baseline based on LASER (Language-Agnostic
SEntence Representations)4
site task set
trn-cln BB it-en tst-cln tot #seg 1,673 1,011 2,684 #src w 37,629 20,280 57,909 #trg w 40,256 21,657 61,913
The filtered bi-text does not suffer anymore from the imbalance number of words but it is 20 times smaller than the original one. Since the corpus filtering discussed in the previous section removes most of the original data, further exacerbating the problem of data scarcity, we tried to overcome this unwanted side effect by means of data augmentation methods. A widespreadly adopted data augmentation method is data selection. Data selection assumes the availability of a large general domain corpus and a small in-domain corpus; in MT, the aim is to extract parallel sentences from the large bilingual corpus that are most relevant to the target domain as defined by the small corpus.
On the basis of the bilingual cross-entropy
difference
The form of curves is convex, as usual in data selection. In our case, the best trade-off between the pertinence of data and its amount occur when something more than a million words is selected; therefore, we decided to mine from OPUS the bilingual text whose size is given in row DS of Table 5. Anyway, note that the lowest PP for selections is at least one order of magnitude greater than on LMs trained on in-domain training sets. task it-en set DS BT
#seg #src w #trg w 206,990 1,352,623 1,312,068
30,957 482,398 505,688
Another well-known data-augmentation method,
which somehow also represents an alternative
way to corpus filtering for dealing with the BB
English side issue, is back-translation.
Backtranslation
The MT engine is built on the ModernMT
framework6 which implements the
Transformer
The same Big model and its smaller variants, the Base with 50 million parameters and the Tiny with 20 million parameters, were also trained on in-domain data only for the sake of comparison. 4.2
We empirically compared the quality of
translations generated by various MT models: two
generic, three genuine in-domain of different size
and several variants of our generic model adapted
(Section 3.1) on in-domain data resulting from the
presented methods: filtering (Section 3.2), data
selection (Section 3.3.1) and back-translation
(Section 3.3.2). Performance was measured on the
BB and TT test sets in terms of BLEU
Scores are collected in Table 7. First, as expected (in-domain data scarcity paragraph of Section 2.2), it is not feasible to properly train a huge number of parameters with few data; in fact, the best performing pure in-domain model is the smallest one (Transformer tiny). Instead, the naive application of the MT state-of-the-art would have led to simply train a Transformer big model on the original in-domain data. This model would not have been competitive with GT on TT data (28.01 vs. 32.20 BLEU); it would have been on BB data if we had only considered the noisy test set (20.45 vs. 11.45) resulting in an important misinterpretation of the actual quality of the two systems; conversely, our preliminary analysis allowed us to discover the need of cleaning BB data, which guarantees a reliable assessment (24.73 vs. 32.59).
Data augmentation methods (DS, BT) are both effective in making available additional useful bitexts; for example, the BLEU score of the model BBtrn-cln+TTtrn increases by 3 absolute points 7BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a, TER+tok.tercom-nonorm-punct-noasian-uncased, chrF2+numchars.6+space.false (from 37.02 to 40.09) when DS and BT data are added to the adaptation corpus.
The fine-tuning of a Transformer big generic model to the weather forecast domain turned out to be more effective than any training from scratch using original in-domain data only: the top performing model - BBtrn-cln+TTtrn+DS+BT - definitely improves the Transformer tiny with respect to all metrics on the BB clean test set (40.09/45.28/.6617 vs 35.80/61.05/.5808), and to two metrics out of three on the TT test set (TER: 51.01 vs. 51.79, CHRF: .6803 vs. .6488). Moreover, all its scores are a lot better than those of Google Translate. 4.4
To give a grasp of the actual quality of automatic translations, Table 8 collects the English text generated by some of the tested MT models fed with a rather complex Italian source sentence. The manual translations observed in BB data are shown as well: their number, their variety, some questionable/wrong lexical choices in them (“high” instead of “upper-level currents”, “South-western” instead of “Southwesterly”) and one totally wrong (“Weak high pressure conditions.”) prove the difficulty of learning from such data and the need to pay par
Italian source sentence:
Le correnti in quota si disporranno da sudovest avvicinando masse d’aria piu` umida alle Alpi. Manual English translations found in BB bulletins:
Weak high pressure conditions.
The high currents will turn to south-west and humid air mass will reach the Alps. Southwesterly currents will bring humid air masses to South Tyrol.
South-western currents will bring humid air masses to the Alps.
South-westerly upper level flow will bring humid air masses towards our region. More humid air masses will reach the Alps.
Humid air reaches the Alps with South-westerly winds.
Automatic English translations generated by some MT models: GT The currents at high altitudes will arrange themselves from the southwest, bringing more humid air masses closer to the Alps.
FBK Currents in altitude will be deployed from the southwest, bringing wet air masses closer to the Alps.
Transformer tiny
South-westerly upper level flow will bring humid air masses towards the Alps.
The upper level flow will be arranged from the southwest approaching more humid air masses to the Alps. ticular attention to the evaluation phase. Concerning translations, GT is able to keep most of the meaning of the source text but the translation is too literal to result in fluent English. FBK only partially transfers the meaning from the source and generates a rather bad English text. Transformer tiny provides a very good translation both from a semantic and a syntactic point of view, losing only the negligible detail that the “air masses” are “more humid”, not simply “humid”. Finally, BBtrn-cln+TTtrn+DS+BT, the model that on the basis of our evaluations is the best one, on this specific example works very well at the semantic level but rather poorly on the grammatical level.
This example shows that pure in-domain models, as expected, are “more in-domain” than generic models, though adapted, showing greater adherence to domain-specific language. On the other hand, according to scores in Table 7, adapted models should be better in generalization. Only subjective evaluations involving meteorologists can settle the question of which model is the best. 5
In this paper we described the development process that led us to build competitive customized translation models. Given the provided in-domain data, we started by analyzing them under several perspectives and discovered that they are few, noisy and heterogeneous. We faced these issues by exploiting a number of methods which represent established knowledge of the scientific community: adaptation of neural models, corpus filtering and data augmentation techniques such as data selection and back-translation. In particular, corpus filtering allowed us to avoid the misleading results observed on the original noisy data, while adaptation and data augmentation proved useful in effectively taking advantage of out-of-domain resources.