=Paper= {{Paper |id=Vol-2253/paper63 |storemode=property |title=Online Neural Automatic Post-editing for Neural Machine Translation |pdfUrl=https://ceur-ws.org/Vol-2253/paper63.pdf |volume=Vol-2253 |authors=Matteo Negri,Marco Turchi,Nicola Bertoldi,Marcello Federico |dblpUrl=https://dblp.org/rec/conf/clic-it/NegriTBF18 }} ==Online Neural Automatic Post-editing for Neural Machine Translation== https://ceur-ws.org/Vol-2253/paper63.pdf

Online Neural Automatic Post-editing for Neural Machine Translation

Matteo Negri1 Marco Turchi1 Nicola Bertoldi1,2 Marcello Federico1,2
1
Fondazione Bruno Kessler - Trento, Italia
2
MMT Srl - Trento, Italia
[negri,turchi,bertoldi,federico]@fbk.eu

Abstract et al., 2007). Cast as a problem of “monolin-
gual translation” (from raw MT output into im-
English. Machine learning from user cor- proved text in the same target language), APE
rections is key to the industrial deploy- has followed a similar evolution to that of MT.
ment of machine translation (MT). We in- As in MT, APE research received a strong boost
troduce the first on-line approach to auto- from shared evaluation exercises like those orga-
matic post-editing (APE), i.e. the task of nized within the well-established WMT Confer-
automatically correcting MT errors. We ence on Machine Translation (Chatterjee et al.,
present experimental results of APE on 2018). In terms of approaches, early MT-like
English-Italian MT by simulating human phrase-based solutions (Béchara et al., 2011; Rosa
post-edits with human reference transla- et al., 2013; Lagarda et al., 2015; Chatterjee et
tions, and by applying online APE on MT al., 2015) have been recently outperformed and re-
outputs of increasing quality. By evaluat- placed by neural architectures that now represent
ing APE on generic vs. specialised and the state of the art (Junczys-Dowmunt and Grund-
static vs. adaptive neural MT, we address kiewicz, 2016; Chatterjee et al., 2017a; Tebbi-
the question: At what cost on the MT side fakhr et al., 2018; Junczys-Dowmunt and Grund-
will APE become useless? kiewicz, 2018). From the industry standpoint,
APE has started to attract MT market players in-
Italiano. L’apprendimento automatico terested in combining the two technologies to sup-
dalle correzioni degli utenti è fonda- port human translation in professional workflows
mentale per lo sviluppo industriale della (Crego et al., 2016).
traduzione automatica (MT). In questo
Focusing on this industry-oriented perspective,
lavoro, introduciamo il primo approccio
this paper makes a step further on APE research
on-line al post-editing automatico (APE),
by exploring an online neural approach to the
ovvero il compito di correggere automati-
task. The goal is to leverage human feedback
camente gli errori della MT. Presentiamo
(post edits) to improve on-the-fly a neural APE
risultati di online APE su MT da inglese
model without the need of stopping it for fine-
a italiano simulando le correzioni umane
tuning or re-training from scratch. Online learn-
con traduzioni manuali già disponibili e
ing capabilities are crucial (both for APE and
utilizzando MT di qualità crescente. Val-
MT) in computer-assisted translation scenarios
utando l’APE su MT neurale generica op-
where professional translators operate on sugges-
pure specializzata, statica o adattiva, af-
tions provided by machines. In such scenarios, hu-
frontiamo la domanda di fondo: a fronte
man corrections represent an invaluable source of
di quale costo sul lato MT l’APE diventerà
knowledge that systems should exploit to enhance
inutile?
users’ experience and increase their productivity.
Towards these objectives we provide two contri-
1 Introduction butions. One is the first online approach to neural
APE. Indeed, while MT-like online learning tech-
Automatic Post-editing for MT is a supervised niques have been proposed for phrase-based APE
learning task aimed to correct errors in a machine- (Ortiz-Martı́nez and Casacuberta, 2014; Simard
translated text (Knight and Chander, 1994; Simard and Foster, 2013; Chatterjee et al., 2017b), nothing
has been done yet under the state-of-the-art neural selection mechanism that updates the model by
paradigm. In doing this, the other contribution is learning from previously collected triplets that are
the first evaluation of neural APE run on the output similar to the input test item (see lines 2-5 in Al-
of neural MT (NMT). So far, published results re- gorithm 1);
port significant gains1 when APE is run to correct (2) After post-editing, by means of a model adap-
the output of a phrase-based MT system. To our tation procedure that learns from human revisions
knowledge, the true potential of APE with higher of the last automatic correction generated by the
quality NMT output has not been investigated yet. system (lines 8-10).
The last observation introduces a more general Similar to the methods proposed in (Chatter-
discussion on the relation between MT and APE. jee et al., 2017b) and (Farajian et al., 2017),
Since, by definition, APE’s reason of being is the the instance-selection technique (first update step)
sub-optimal quality of MT output, one might won- consists of two components: i) a knowledge base
der if the level of current MT technology still justi- (KB) that is continuously fed with the processed
fies efforts on APE. Along this direction, our third triplets, and ii) an information retrieval engine
contribution is an analysis of online neural APE that, given the (src, mt) test item, selects the most
applied to the output of NMT systems featuring similar triplet (lines 2-3). The engine is simulta-
different levels of performance. Our competitors neously queried using both src and mt segments
range from a generic model trained on large paral- and it returns the triplet that has the highest co-
lel data (mimicking the typical scenario in which sine similarity with both (Top(R)). If the similar-
industry users – e.g. Language Service Providers ity is above a threshold τ , a few training iterations
– rely on web-based services or other black-box are run to update the model parameters (line 5).
systems) to highly customized online models (like Depending on the application scenario, KB can be
those that LSPs would desire but typically cannot pre-filled with the APE training data or left empty
afford). Our experiments in this range of condi- and filled only with the incoming triplets. In our
tions aim to shed light on the future of APE from experiments, the repository is initially empty.
the industry standpoint by answering the question:
At what cost on the MT side will APE become Algorithm 1: Online neural APE
useless? Require M: Trained APE model
Require Ts: Stream of test data
2 Online neural APE Require KB: Pool of (src, mt, hpe) triplets
1: while pop (src, mt) from Ts do
APE training data usually consist of (src, mt, hpe) 2: R ← Retrieve ((src, mt), KB)
3: (srctop , mttop , hpetop ) ← Top (R)
triplets whose elements are: a source sentence 4: if Sim ((srctop , mttop , hpetop ), (src, mt)) > τ do
(src), its translation (mt) and a human correction 5: M∗ ← Update (M,(srctop , mttop , hpetop ))
6: ape ← APE (M∗ ,(src, mt))
of the translated sentence (hpe). Models trained 7: hpe ← HumanPostEdit ((src, ape))
on such triplets are then used to correct the mt el- 8: KB ← KB ∪ (src,mt,hpe)
ement of (src, mt) test data. Neural approaches 9: M∗∗ ← Update (M∗ ,(src, mt, hpe))
10: M ← M∗∗
to the task have shown their effectiveness in batch 11: end while
conditions, in which a static pre-trained model is
run on the whole test corpus. When moving to an
Once the hpe has been generated, the second up-
online setting, instead, APE systems should ide-
date step takes place (line 9) by running few train-
ally be able to continuously evolve by stepwise
ing iterations on the (src, hpe) pair. When training
learning from the interaction with the user. This
using only one single data point, the learning rate
means that, each time a new post-edit becomes
and the number of epochs have a crucial role be-
available, the model has to update its parameters
cause too high/small values can make the training
on-the-fly in order to produce better output for the
unstable/inefficient. To avoid such problems, we
next incoming sentence. To this aim, we extend a
connect the two parameters by applying a time-
batch APE model by adding the capability to con-
based decay learning rate that reduces the learning
tinuously learn from human corrections of its own
rate when increasing of the number of epochs (i.e.
output. This is done in two steps:
lr = lr/(1+num epoch)). In our experiments, this
(1) Before post-editing, by means of an instance
strategy results in better performance than setting
1
Up to 7.6 BLEU points at WMT 2017 (Bojar et al., 2017) a fixed learning rate.
3 Experiments translation, few training iterations adapt it to the
domain of the input document. The adaptation
We run our experiments on English-Italian data,
steps implement the same strategies of the online
by comparing the performance of different neural
APE system (see §2). This setting represents the
APE models (batch and online) used to correct the
situation in which our LSP has access to the inner
output of NMT systems of increasing quality.
workings of a competitive online NMT system.
3.1 Data Specialized (S). This model is built by fine-tuning
(Luong and Manning, 2015) G on the in-domain
To train our NMT models we use both generic
training data (416K). It reflects the condition in
and in-domain data. Generic data cover a vari-
which our LSP has access both to customer’s data
ety of domains. They comprise about 53M par-
and to the inner workings of a competitive batch
allel sentences collected from publicly-available
NMT engine. The adaptation routine, however, is
collections (i.e. all the English-Italian parallel cor-
limited to the standard approach of performing ad-
pora available on OPUS2 ) and about 50M sen-
ditional training steps on the in-domain data.
tence pairs from proprietary translation memories.
Specialized Online (SO). This model is built by
Generic data, whose size is per se sufficient to
combining the functionalities of GO and S. It uses
train a competitive general-purpose engine, are
the in-domain training data for fine-tuning and the
used to build our basic NMT model. On top of it,
incoming (src, hpe) pairs for online adaptation to
in-domain (information technology) data are used
the target domain. This setting represents the sit-
in different ways to obtain improved, domain-
uation in which our LSP has access to: i) cus-
adapted models. In-domain data are selected to
tomer’s in-domain data and ii) the inner workings
emulate the online setting of industrial scenarios
of a competitive online NMT engine.
where input documents are processed sequentially
All the models are trained with the ModernMT
on a sentence-by-sentence basis. They consist in a
open source software,3 which is built on top of
proprietary translation project of about 421K seg-
OpenNMT-py (Klein et al., 2017). It employs
ments, which are split in training (416K segments)
an LSTM-based recurrent architecture with atten-
and test (5,472) keeping the sentence order. Post-
tion (Bahdanau et al., 2014) using 2 bi-directional
edits are simulated using references.
LSTM layers in the encoder, 4 left-to-right LSTM
To train the APE models we use the English-
layers in the decoder, and a dot-product attention
Italian section of the eSCAPE corpus (Negri et al.,
model (Luong et al., 2015). In our experiments
2018). It consists of about 6.6M synthetically-
we used an embeddings’ size of 1024, LSTMs of
created triplets in which the mt element is pro-
size 1024, and a source and target vocabulary of
duced with phrase-based and neural MT systems.
32K words, jointly trained with the BPE algorithm
3.2 NMT models (Sennrich et al., 2016). The fact that ModernMT
already implements the online adaptation method
Our NMT models feature increasing levels of
presented in (Farajian et al., 2017) simplified our
complexity, so to represent a range of conditions
tests with online neural APE run on the output of
in which a user (say a Language Service Provider)
competitive NMT systems (GO and SO).
has access to different resources in terms of MT
technology and/or data for training and adaptation. 3.3 APE models
Our systems, ranked in terms of complexity with
respect to these two dimensions are: We experiment with two neural APE systems:
Generic (G). This model is trained on the large Generic APE. This batch system is trained only
(103M) multi-domain parallel corpus. It repre- on generic data (6.6M triplets from eSCAPE) and
sents the situation in which our LSP entirely re- is similar to those tested in the APE shared task
lies on an off-the-shelf, black-box MT engine that at WMT. The main difference is that the training
cannot be improved via domain adaptation. data are neither merged with in-domain triplets nor
Generic Online (GO). This model extends G with selected based on target domain information.
the capability to learn from the incoming human Online APE. This system is trained on the generic
post-edits (5,472 test items). Before and after data and continuously learns from human post-
edits of the test set as described in §2.
2
http://opus.lingfil.uu.se dump of mid June
3
2017. http://github.com/ModernMT/MMT.
MT Type MT Generic Online Online APE achieves significant4 improve-
APE APE
Generic (G) 40.3 39.0 47.1† ments not only over the output of G (+6.8) and
Gen. Online (GO) 45.6 41.9 48.1† its online extension GO (+2.5), but also over the
Specialized (S) 52.1 45.5 53.5† specialized model S (+1.4). The gain over GO is
Spec. Online (SO) 55.0 47.4 54.8 particularly interesting: it shows that even when
APE and MT use the same in-domain data for on-
Table 1: APE performance on NMT outputs of dif- line adaptation, the APE model is more reactive to
ferent quality (“†” denotes statistically significant human feedback. Though trained on much smaller
differences wrt. the MT baseline with p<0.05). generic corpora (6.6M triplets versus 103M paral-
lel sentences), the possibility to leverage richer in-
The two systems are based on a multi-source formation in the form of (src, mt, pe) instances at
attention-based encoder-decoder approach simi- test time seems to have a positive impact. A deeper
lar to (Chatterjee et al., 2017a). It employs a exploration of this aspect falls out of the scope of
GRU-based recurrent architecture with attention this paper and is left as future work.
and uses two independent encoders to process the Also with online APE, the gains become
src and mt segments. Similar to the NMT systems, smaller by increasing the MT quality, reaching
it is trained on sub-word units by using BPE, with a point where the system can only approach the
a vocabulary created by selecting to 50K most fre- highest MT performance of SO (with a non-
quent sub-words. Word embedding and GRU hid- significant -0.2 BLEU difference). This confirms
den state sizes are set to 1024. Network param- that correcting the output of competitive NMT en-
eters are optimized with Adagrad (Duchi et al., gines is a hard task, even for a dynamic APE sys-
2011) with a learning rate of 0.01. A develop- tem that learns from the interaction with the user.
ment set randomly extracted from the training data However, besides improving its performance by
is used to set the similarity threshold used by the learning from user feedback acquired at test time
online model for the first update step (τ =0.5) as (similar to the APE system), SO also relies on
well as the learning rate (0.01) and the number of previous fine-tuning on a large in-domain corpus
epochs (3) of both adaptation steps. (similar to S). To answer our initial question (“At
what cost on the MT side will APE become use-
4 Results and discussion less?”) it is worth remarking that leveraging in-
domain training/adaptation data is a considerable
APE results computed on different levels of trans- advantage for MT but it comes at a cost that should
lation quality are reported in Table 1. Looking not be underestimated. In terms of the data itself,
at the NMT performance, all the adaptation tech- collecting enough parallel sentences for each tar-
niques yield significant improvements over the get domain is a considerable bottleneck that limits
Generic model (G). The large gain achieved via the scalability of competitive NMT solutions. In
fine-tuning on in-domain data (S: +11.8 BLEU) is addition to that, the technology requirements (i.e.
further increased when adding online learning ca- having access to the inner workings of the NMT
pabilities on top of it to create the most competi- engine) and the computational costs involved (for
tive Specialized Online system (SO: +14.7). fine-tuning the generic model) are constraints that
As expected, the batch APE model trained on few LSPs are probably able to satisfy.
generic data only (that is, without in-domain in-
formation) is unable to improve the quality of 5 Conclusion
raw MT output. Moreover, although APE results
increase with higher translation quality, also the We introduced an online neural APE system,
performance distance from the more competitive which is trained on generic data and only exploits
NMT systems becomes larger (from -1.3 to -7.6 user feedback to improve its performance, and
points respectively for G and SO). These results evaluated it on the output of NMT systems fea-
confirm the WMT findings about the importance turing increasing complexity and in-domain data
of domain customization for batch APE (Bojar et demand. Our results show the effectiveness of
al., 2017), and advocate for online solutions ca- current APE technology in the typical setting of
pable to maximize knowledge exploitation at test 4
Statistical significance is computed with paired bootstrap
time by learning from user feedback. resampling (Koehn, 2004).
most LSPs while, in terms of resources (especially Patrice Brunelle, Aurelien Coquard, Yongchao
in-domain data) and technical expertise needed. Deng, et al. 2016. SYSTRAN’s Pure Neu-
ral Machine Translation Systems. arXiv preprint
We also conclude that developing MT engines that
arXiv:1610.05540.
make APE useless is still a prerogative of few.
John Duchi, Elad Hazan, and Yoram Singer. 2011.
Adaptive Subgradient Methods for Online Learning
References and Stochastic Optimization. Journal of Machine
Learning Research, 12:2121–2159, July.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-
gio. 2014. Neural Machine Translation by Jointly M. Amin Farajian, Marco Turchi, Matteo Negri, and
Learning to Align and Translate. arXiv preprint Marcello Federico. 2017. Multi-Domain Neural
arXiv:1409.0473. Machine Translation through Unsupervised Adapta-
Hanna Béchara, Yanjun Ma, and Josef van Genabith. tion. In Proceedings of the Second Conference on
2011. Statistical Post-Editing for a Statistical MT Machine Translation, pages 127–137, Copenhagen,
System. In Proceedings of the 13th Machine Trans- Denmark, September.
lation Summit, pages 308–315, Xiamen, China,
September. Marcin Junczys-Dowmunt and Roman Grundkiewicz.
2016. Log-linear Combinations of Monolingual and
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Bilingual Neural Machine Translation Models for
Yvette Graham, Barry Haddow, Shujian Huang, Automatic Post-Editing. In Proceedings of the First
Matthias Huck, Philipp Koehn, Qun Liu, Varvara Conference on Machine Translation, pages 751–
Logacheva, Christof Monz, Matteo Negri, Matt 758, Berlin, Germany, August.
Post, Raphael Rubino, Lucia Specia, and Marco
Turchi. 2017. Findings of the 2017 Conference on Marcin Junczys-Dowmunt and Roman Grundkiewicz.
Machine Translation (WMT17). In Proceedings of 2018. Microsoft and University of Edinburgh at
the Second Conference on Machine Translation, Vol- WMT2018: Dual-Source Transformer for Auto-
ume 2: Shared Task Papers, pages 169–214, Copen- matic Post-Editing. In Proceedings of the Third
hagen, Denmark, September. Conference on Machine Translation, Brussels, Bel-
gium, October.
Rajen Chatterjee, Marion Weller, Matteo Negri, and
Marco Turchi. 2015. Exploring the Planet of Guillaume Klein, Yoon Kim, Yuntian Deng, Jean
the APEs: a Comparative Study of State-of-the-art Senellart, and Alexander Rush. 2017. OpenNMT:
Methods for MT Automatic Post-Editing. In Pro- Open-Source Toolkit for Neural Machine Transla-
ceedings of the 53rd Annual Meeting of the Associa- tion. In Proceedings of ACL 2017, System Demon-
tion for Computational Linguistics), pages 156–161, strations, pages 67–72, July.
Beijing, China, July.
Kevin Knight and Ishwar Chander. 1994. Auto-
Rajen Chatterjee, M. Amin Farajian, Matteo Ne- mated Post-Editing of Documents. In Proceedings
gri, Marco Turchi, Ankit Srivastava, and Santanu of AAAI, volume 94, pages 779–784.
Pal. 2017a. Multi-source Neural Automatic Post-
Editing: FBK’s participation in the WMT 2017 APE Philipp Koehn. 2004. Statistical Significance Tests for
shared task. In Proceedings of the Second Con- Machine Translation Evaluation. In Proceedings of
ference on Machine Translation, Volume 2: Shared the Empirical Methods on Natural Language Pro-
Task Papers, pages 630–638, Copenhagen, Den- cessing, pages 388–395, Barcelona, Spain, July.
mark, September.

Rajen Chatterjee, Gebremedhen Gebremelak, Matteo Antonio L. Lagarda, Daniel Ortiz-Martı́nez, Vicent Al-
Negri, and Marco Turchi. 2017b. Online Automatic abau, and Francisco Casacuberta. 2015. Translat-
Post-editing for MT in a Multi-Domain Translation ing without In-domain Corpus: Machine Transla-
Environment. In Proceedings of the 15th Confer- tion Post-Editing with Online Learning Techniques.
ence of the European Chapter of the Association for Computer Speech & Language, 32(1):109–134.
Computational Linguistics: Volume 1, Long Papers,
pages 525–535, Valencia, Spain, April. Minh-Thang Luong and Christopher D Manning.
2015. Stanford Neural Machine Translation Sys-
Rajen Chatterjee, Matteo Negri, Raphael Rubino, and tems for Spoken Language Domains. In Proceed-
Marco Turchi. 2018. Findings of the WMT 2018 ings of the International Workshop on Spoken Lan-
Shared Task on Automatic Post-Editing. In Pro- guage Translation (IWSLT’15), pages 76–79, Da
ceedings of the Third Conference on Machine Trans- Nang, Vietnam, December.
lation, Brussels, Belgium, October. Association for
Computational Linguistics. Minh Thang Luong, Hieu Pham, and Christopher D
Manning. 2015. Effective Approaches to Attention-
Josep Crego, Jungi Kim, Guillaume Klein, Anabel Re- based Neural Machine Translation. arXiv preprint
bollo, Kathy Yang, Jean Senellart, Egor Akhanov, arXiv:1508.04025.
Matteo Negri, Marco Turchi, Rajen Chatterjee, and
Nicola Bertoldi. 2018. eSCAPE: a Large-scale
Synthetic Corpus for Automatic Post-Editing. In
Proceedings of the Eleventh International Confer-
ence on Language Resources and Evaluation (LREC
2018), Miyazaki, Japan, May.
Daniel Ortiz-Martı́nez and Francisco Casacuberta.
2014. The New THOT Toolkit for Fully-Automatic
and Interactive Statistical Machine Translation. In
Proceedings of the 14th Annual Meeting of the Eu-
ropean Association for Computational Linguistics,
pages 45–48, Gothenburg, Sweden, April.
Rudolf Rosa, David Marecek, and Ales Tamchyna.
2013. Deepfix: Statistical Post-editing of Statistical
Machine Translation Using Deep Syntactic Analy-
sis. In Proceedings of the 51st Annual Meeting
of the Association for Computational Linguistics,
pages 172–179, Sofia, Bulgaria, August.

Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural Machine Translation of Rare Words
with Subword Units. In Proceedings of the 54th An-
nual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany, August.

Michel Simard and George Foster. 2013. PEPr: Post-
edit Propagation Using Phrase-based Statistical Ma-
chine Translation. In Proceedings of the XIV Ma-
chine Translation Summit, pages 191–198, Nice,
France, September.
Michel Simard, Cyril Goutte, and Pierre Isabelle.
2007. Statistical Phrase-Based Post-Editing. In
Proceedings of the Annual Conference of the North
American Chapter of the Association for Computa-
tional Linguistics, pages 508–515, Rochester, New
York, April.
Amirhossein Tebbifakhr, Ruchit Agrawal, Rajen Chat-
terjee, Matteo Negri, and Marco Turchi. 2018.
Multi-source Transformer with Combined Losses
for Automatic Post-Editing. In Proceedings of the
Third Conference on Machine Translation, Brussels,
Belgium, October.