The Covid-19 pandemic has created a global demand for accurate and up-to-date information which often originates in English and needs to be translated. To train a machine translation system for such a narrow topic, we leverage in-domain training data in other languages both from related and unrelated language families. We experiment with different transfer learning schedules and observe that transferring via more than one auxiliary language brings the most improvement. We compare the performance with joint multilingual training and report superior results of the transfer learning approach.
A global crisis such as the current Covid-19 pandemic requires information to be spread as efficiently as possible. Working with data from different international resources in multiple languages can resolve possible inconsistencies and prevent misinformation. In an emergency situation, new data is released constantly and is communicated to the public not only via national news and authorities, but also foreign media, scientific journals or statements of international agencies. There are extensive data resources written in English which are not accessible for non-English speakers.
In order to quickly access the information in a foreign language, machine translation (MT) can be of great help. However, Covid-related texts use a specific terminology and MT models are known to struggle outside of the general domain.
More than a year after the Covid outbreak, there already is a significant amount of domain-specific multilingual text resources. Furthermore, Covid-related texts are a part of a broader medical domain which can provide additional authentic data for training. Thanks to the MLIA @ Eval1 initiative who gathered training data for MT and information retrieval related to the pandemic, we can successfully adapt an MT system to the Covid domain or even train it from scratch.
This paper gives an overview of possible methods to automatically translate Covid-related texts. Section 2 outlines different approaches to train a domain-specific MT system using multilingual corpora. Section 3 describes our training data and section 4 gives more details about our MT systems and presents the results. Section 5 concludes the paper. 2
In this section we outline several strategies applicable in the situation where we need to translate from English to multiple languages, we are confined within a specific domain and we have mid-size parallel corpora for every language pair of interest.
Firstly, we can train a standard MT system from scratch for each language separately, possibly resorting to some data augmentation method, e.g. back-translation. Secondly, we can use transfer learning to transfer from a pretrained MT system in one language to another. Finally, we can train a multilingual MT system which learns jointly from all available data.
In our experiments, we do not consider any additional monolingual resources. Although monolingual data are generally easier to obtain, we remain constrained by the datasets provided by the organizers of MLIA @ Eval which are described in Section 3. We also do not evaluate a transfer from a large MT model pretrained on texts from the general domain which would be a promising strategy as well. 2.1
When neural machine translation (NMT) became the
dominant paradigm in MT [
An NMT system directly trained to translate in the
Covid domain serves as our baseline.
Back-translation is a crucial method in NMT used to
augment training data by translating an existing monolingual
corpus [
When using a bidirectional model (sharing the
encoder and decoder for both translation directions),
backtranslation can be performed on-the-fly. During training,
the model switches between the training and the inference
mode to produce batches of synthetic sentence pairs and
learn from both authentic and synthetic samples in each
training step. As the system improves, the quality of the
generated samples improves as well. This approach was
originally proposed for training an unsupervised MT
system [
In our systems, we translate only the target sentences and generate a synthetic source side on the fly. We do not use any additional monolingual data for back-translation. 2.3
The first strategy to utilize multilingual in-domain training
corpora is transfer learning. It can be used to transfer from
a different domain [
A trivial transfer learning approach was proposed by
Kocmi and Bojar [
Before training the parent model, it is necessary to
designate some vocabulary entries for the new language.
Otherwise the model would be forced to completely re-learn
its subword embeddings and their connections and would
lose its ability to transfer. Kocmi [
In this work, we experiment with several transfer learning schedules. We repeat the transfer procedure several times with the child becoming the parent for either a completely new language (e.g. German ! English ! Spanish ! . . . ) or for the original parent (e.g. German ! English ! German ! . . . ), as illustrated in Figure 1. We always generate the vocabulary from the concatenation of the parent and its "first child". When adding a third (or fourth) language, the joint BPE vocabulary has to be modified by replacing the original parent vocabulary entries with the new child ones. The schedules and their results are described in Section 4. 2.4
The second strategy to utilize multilingual in-domain training corpora is joint multililngual training.
Multilingual translation systems are either trained with
full parameter sharing [
In this work, we rely on full parameter sharing and use the same architecture as our bilingual systems, while training it to translate from English into three languages (French, Italian and Spanish) at once. During inference, the target language is determined from indicated language embeddings of the target sentence. We selected these three languages for their similarity which could help the model re-use and share some knowledge. The BPE vocabulary was extracted from the concatenation of all four corpora, using only unique English sentences to reach a comparable corpus size. 3
Covid-19 MLIA @ Eval organized a community evaluation effort aimed at accelerating the creation of resources and tools for improved Multilingual Information Access (MLIA). A part of this initiative is a competition to develop the best MT system translating from English to several European languages: German, Modern Greek, French, Italian, Spanish and Swedish. The competition is incremental and so far only the first round was concluded.
The parallel training data provided by the organizers for the first round and used in this paper is summarized in Table 1.2 It was created based on existing corpora from the medical domain, enriched with sentences directly about 2The development test set used for the final model selection was obtained by cutting 500 sentences off of either the train set or the development set, depending on the original development set size.
English → German dev BLEU
EN → DE
20.76
English → Italian dev BLEU
EN → IT 30.97
EN → ES EN → DE
21.26 EN → ES EN → IT 31.68
EN → DE EN → ES EN → DE
22.60 EN → DE EN → ES EN → IT 32.10
EN → DE EN → SV EN → DE
22.55 EN → DE EN → ES EN → IT EN → ES EN → IT 33.07
EN → DE EN → SV EN → DE EN → SV EN → DE 22.50
Train
Fine-tune
Fine-tune
Fine-tune
Fine-tune
Train
Fine-tune
Fine-tune
Fine-tune
Fine-tune
Covid, mostly harvested through web crawling and
parallel sentence mining [
All data was segmented into BPE units [
We participated in the MT shared task of the Covid-19
MLIA @ Eval initiative and trained a model for translation
into each of the six languages listed in Section 3. The
results of our submitted systems are summarized in the
preliminary report [
We experimented with three training strategies compared against one baseline BASE: 1. unidirectional training without back-translation (BASE); 2. bidirectional training with online back-translation (BT); 3. transfer learning (TRANSFER); 4. multilingual training (MULTILING).
For all our MT models we use a 6-layer Transformer
[
For each language pair we first trained a bidirectional back-translation model described in Section 2 and compared it to a standard unidirectional model without backtranslation. Online back-translation improved the score by 0.5–2.9 BLEU points, depending on the language, but surprisingly caused a decrease of 0.4 BLEU in the case of the English–Modern Greek model. The reason for this drop is likely in the bidirectionality of the model rather than the data augmentation itself. The results are summarized in Table 2.
We experimented with a dropout of 0.1 and 0.2 and
concluded that higher dropout helps in most settings. This
observation is in line with Sennrich and Zhang [
We observed that it often helped to use the transfer incrementally, having the model converge on one parallel corpus, switch the target language, wait for convergence and switch again. We hypothesize that the model benefits from seeing a larger variety of sentences. For example transferring from German to Spanish to Italian (32.10 BLEU) performs better than transferring directly from Spanish to Italian (31.68 BLEU). The best combination is to even repeat the Spanish-Italian transfer twice (33.07 BLEU).
When translating from English to German, finetuning the en-de BT model on English!Spanish (or English!Swedish) and switching back to English ! German adds around 1 BLEU on top of the original BT model. All language combinations used in our transfer learning multilingual best transfer best base multilingual best transfer best base experiments are described in Table 3 and selected schedules are illustrated in Figure 1.
We observe that transfer learning improves the
performance in all cases but French, where the BASE model with
BT reaches 38.5 BLEU, which is 3 BLEU points more
than transfer learning. There is a significant overlap
between the training sets in different languages and it is
possible that French does not benefit from the transfer because
it does not provide enough new sentences. On the other
hand, the largest improvement is seen by the language pair
with the least amount of training data, English-Swedish,
where BLEU increases by 1.1 points on the dev set and
1.6 points on the test set.
We train a multilingual model for translation from English
to French, Italian and Spanish. The model has the same
architecture as our bilingual models, all parameters are
shared for all languages. Its encoder and decoder were
first pretrained on monolingual data in all three languages
and English using the MLM criterion [
Table 4 shows the comparison of the TRANSFER models with a multilingual model trained jointly. We observe that transfer learning yields superior results and is thus a more effective way to leverage multilingual data than joint multilingual training. However, there is an advantage of a joint model in terms of the training and storage cost. After three days of training, the multilingual model can be used for translation into all three languages. The initial BASE models can take between one (without BT) and five (with BT) days to train and fine-tuning on a child language pair adds around 6 hours.
Table 5 lists our task submissions and compares all approaches on the official Covid-19 MLIA @ Eval blind test set.5 5The BLEU scores in Table 4 and Table 5 cannot be directly com
We trained several MT systems specialized in translation of texts related to the topic of Covid-19 and the pandemic from English to six European languages.
We experimented with three training approaches and we conclude that there is not a universal winner that would work the best for all language pairs. However, transfer learning brings promising results across the board, especially when training data is limited. We observed an interesting phenomenon where incremental fine-tuning on multiple languages brings additional gains, as we expose the model to a larger variety of training sentences.
In our setting, transferring knowledge is a more efficient way to leverage multilingual data than joint training. For English!German, we observe that a transfer learning detour via Spanish or Swedish improves the parent model itself. For English!Modern Greek, transfer learning via German works well, despite the unrelatedness of the two languages. For English!French, on the other hand, a bidirectional model with back-translation beats both multilingual and transfer-based models.
This study was supported in parts by the grants CZ.07.1.02/0.0/0.0/16_023/0000108 (Operational Programme – Growth Pole of the Czech Republic), 19-26934X of the Czech Science Foundation, and by the SVV project number 260 575. pared as the dev scores were calculated by authors and test scores by the organizers.