English. Neural machine translation (NMT) recently redefined the state of the art in machine translation, by introducing deep learning architecture that can be trained end-to-end. One limitation of NMT is the difficulty to learn representations of rare words. The most common solution is to segment words into subwords, in order to allow for shared representations of infrequent words. In this paper we present ways to directly feed a NMT network with external word embeddings trained on monolingual source data, thus enabling a virtually infinite source vocabulary. Our preliminary results show that while our methods do not seem effective under large-data training conditions (WMT En-De), they instead show great potential for the typical low-resourced data scenario (IWSLT EnFr). By leveraging external embeddings learned on Web crawled English texts, we were able to improve a word-level En-Fr baseline trained on 200,000 sentence pairs by up to 4 BLEU points.
Italiano. La traduzione automatica con reti neurali (neural machine translation, NMT) ha ridefinito recentemente lo stato dell’arte nella traduzione automatica introducendo un’architettura di deep learning che puo` essere addestrata interamente, dall’input all’output. Una limitazione della NMT e` comunque la difficolta` di apprendere rappresentazioni di parole poco frequenti. La soluzione piu` adottata consiste nel segmentare le parole in sottoparole, in modo da consentire rappresentazioni condivise per parole poco frequenti. In questo lavoro presentiamo dei metodi per fornire ad una rete word embedding esterni addestrati su testi nella lingua sorgente, consentendo quindi un vocabolario virtualmente illimitato sulla lingua di input. I nostri risultati preliminari mostrano che i nostri metodi, pur non sembrando efficaci sotto condizioni di addestramento con molti dati (WMT EnDe), risultano invece promettenti per scenari di addestramento con poche risorse (IWSLT En-Fr). Sfruttando word embedding appresi da testi inglesi estratti dal Web, siamo riusciti a migliorare un sistema NMT basato a parole e addestrato su 200.000 coppie di frasi fino a 4 punti BLEU.
The latest developments of machine translation
have been led by the neural approach
One consequence of the scarcity of parallel data
is the occurrence of out-of-vocabulary (OOV) and
rare words. In fact, being NMT a statistical
approach, it cannot learn meaningful representations
for rare words and no representation at all for
OOV words. The solution up to this moment is
to segment words into sub-words
Neural machine translation is based on the
encoder-decoder-attention architecture
The embedding sequence is then processed by a
bi-directional RNN
The decoder takes as input the encoder outputs (or
states) and produces a sequence of target words
e1; e2; : : : ; el. The decoder works by
progressively predicting the probability of the next
target word ei given the previously generated target
words and the source context vector ci. At each
step, the decoder computes a word embeddings
yi 1 of the previous target word, applies one or
more recurrent layers, an attention model function
and a softmax layer. The recurrent layers produce
an hidden state si
where, g can be computed with one or more LSTM
or GRU layers. The output of the RNN is then
used by the attention model
ij =
exp(score(si; hj )) Pm
k=1 exp(score(si; hk)) The weights are used to compute a weighted average of the encoder outputs, which represents the source context
m ci = X j=1 ij hj The source context vector is then combined with the output of the last RNN layer in a new vector zi that is passed as input to the softmax layer to compute the probability for each word in the vocabulary to be the next word, such that:
p(e j ei 1; ci) / exp(o>zi) where zi is a column of Z, a matrix with the same size of the target-side embedding matrix. Let be the set of all the network parameters, then the objective of the training is to find parameter values maximizing the likelihood of the training set S, i.e.:
jej
X X log p(eije<i; ci; ) (f ;e)2S i=1
In order to achieve open-vocabulary translation
with a limited vocabulary size, the words are
segmented into sub-words, and the words with shared
sub-words share part of their representation. The
most common segmenting approach was
introduced by Sennrich et al. (2015b) and exploits only
statistical information, but there are promising
research lines trying to use linguistically motivated
segmentations
The method we propose is based on the training of word embeddings from source-language monolingual data. We use these embeddings as an input to the network, and we remove the source-side embedding matrix. As the external embeddings have been learned for a task that is not machine translation, we introduce a feed-forward layer to map the embeddings into a new space that is more useful for the translation task:
x~j = tanh(xj>W + b) for j = 1; : : : ; m where xj is the external embedding for the word j and the vectors x~i are used merged with the internal embeddings.
In this work we experimented three different settings: (1) only external, (2) mix sum, (3) mix gate. While only external is the setting we have just described above, the other two settings combine the external embeddings with the internal NMT embeddings. The mix sum setting inserts a vector sum between the embeddings and the RNN which simply sums the internal embedding for the word fj and the mapped external embedding for the same word:
x^j = xj + x~j In the mix gate setting, we let the network learn parameters to combine the internal and the external embeddings. A gate is a function that produces a vector of the same dimensionality of the input, with all elements between 0 and 1 to represent the proportion of the corresponding input element that is propagated to the following layer:
zj = ([xj ; x~j ]>Wz + bz) where zj is the output of the gate and is the sigmoid function. The new vector is produced by combining linear transformations of the inputs with the gate zj : x^j = tanh(zj f f1(xj ) + (1 zj ) f f2(x~j )) Where f f is a feed-forward layer. In this setting the network has more parameters to learn for combining the internal and external embeddings in an effective way. 4
Baseline 25.37 Only External Crawl 26.13 Mix Sum Crawl 29.45
Mix Gate Crawl 27.10
We performed our experiments on two tasks representing two different training conditions: large data and small data. The first task is the 2017 WMT News translation task, from English to German, which provides a substantial amount of parallel data. For this experiment, we use all the available training data, about 5 million sentence pairs1, newstest2013 and 2014 as a validation set and newstest2015 (NEWS-15) and newstest 2016 (NEWS-16) as test sets. The second task in the 2016 IWSLT TED Talk translation task, from English to French, for which we only deployed a small in-domain data set consisting of 200,000 sentence pairs, dev and test sets from 2010 to 2013 as a validation sets and the test set 2014 as test set (TED-14) 2.
We used two sets of pre-trained English
wordembeddings. The first is the Common Crawl set
available from the GloVe website3, which
contains 1:9M word embeddings (dim=300) trained
with Glove
plied Adam
Results show that our approach is greatly beneficial in our small data condition (table 1), improving up to 4 bleu scores with the simple strategy of summing the external and internal word embeddings. For the large-data condition (table 2) the picture is instead very different, as none of the settings using external embeddings reaches the results of the baseline.
In order to verify our hypothesis that external embeddings help to extend the vocabulary, we firstly counted the number of OOV words with respect to the internal and external vocabularies for each test set, and also the number of words that are unknown in both of them. The results listed in table 3 show that in the case of TED, the number of OOVs in both vocabularies is 39% smaller than in the internal vocabulary, but at the same time in the external vocabulary it is more than 5 times larger.
In all the experiments, the embeddings trained on Gigacrawl have many more OOVs than the interso I was trained to become a gymnast for two years in Hunan , China in the 1970s .
J’ai e´te´ entraˆıne´e pour devenir gymnaste pendant 2 ans , dans la province d’ Hunan en Chine dans les anne´es 1970 . J’ai e´te´ forme´e pour devenir gymnaste , pendant deux ans au Texas, en Chine dans les anne´es 1970 . J’ai donc e´te´ forme´ pour devenir une gymnaste pendant deux ans en UNK, en Chine dans les anne´es 70. Egyptologists have always known the site of Itjtawy was located somewhere near the pyramids of the two kings [...] . les e´gyptologues avaient toujours pre´sume´ qu’ Itjtawy se trouvait quelque part entre les pyramides des deux rois [...] . Nous avons toujours connu le site de Londres , situe´ quelque part pre´s des pyramides des deux rois [...] Et on sait toujours que le site de UNK e´tait situe´ quelque part pre`s des pyramides des deux rois [...]. nal counterpart, and the difference is particularly large in newstest15 and 16. This can be a reason for degradation of representations, unless the network learns to correct the noise coming from the external side.
To have a glimpse of the degradation, we also counted the number of generated unknown words for each test set. The results are listed in table 4.
What we can observe is a slightly reduced number of unknown tokens in newstest when using only the external embeddings, but in a setting where the target side uses subwords. In all the other cases, the number of unknown words during translations increases dramatically. The increase is from 5 to 22 times in WMT and about 2 times in TED. Now we want to understand if this is due to a corrupted representation of words, which mixes good embeddings with the external embedding for the unknown token, or the reason is to find somewhere else. This is particularly true because of the contemporary improvement in BLEU score.
To verify the correction capabilities of the
network, we check some translations where one word
is missing in one of the two vocabularies. Two
example translations are shown in table 5. In the first
example, the word “China” exists only in the
internal vocabulary, but it’s correctly translated also
by the mix-gate system. Furthermore, the baseline
translates the OOV word “Hunan” with “Texas”,
while our system translates it with an unknown
token. The second behavior is surely one of the main
reasons of the increased number of generated
unknown words using external embeddings, and it is
also preferable as there are methods for replacing
the unknown tokens in a postprocessing step.
In the second example, “Egyptologists” is OOV for the internal vocabulary. Lacking the subject, the baseline resorts to the first person plural, and it also adds a subordinate sentence that change s the meaning with respect to the source. Moreover, again an unknown word for a location is translated with another word that is related with the source only because it is another location (in this case the system translates with “Londres”, which is the French word for “London”). By contrast, in absence of more information about the subject, the mix-gate uses the impersonal form and the grammar of its translation is better in general.
In the large-data setting, the best system using external embeddings is the mix-gate with data from the news domain. From table 3, we can relate the improvement also to the reduced number of external OOV words, but the improvement is so small that we suppose that using better corpora is not a path to follow. Moreover, our results lower than the baseline are an empirical proof that pre-trained embeddings are not useful when there are large parallel data available. 6
In this paper we propose three methods to extend the input word embeddings to an NMT network in order to leverage a word representation coming from a big monolingual corpus. Our results show that this approach greatly improves over an NMT baseline in a low-resource scenario, while it is not helpful for better-resourced tasks.
Using monolingual data for improving NMT is a problem also in the latter case, thus our future work will focus on how to integrate models larger than word embeddings, and trained on monolingual data, to improve word and sentence representations.
This work has been partially supported by the ECfunded projects ModernMT (H2020 grant agreement no. 645487) and QT21 (H2020 grant agreement no. 645452). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPUs used for this research.