ITAI: Adaptive Neural Machine Translation Platform ITAI: Plataforma de traducción automática neuronal adaptativa Thierry Etchegoyhen,1 David Ponce,1 Harritxu Gete Ugarte,1 Victor Ruiz Gómez,1 1 Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) {tetchegoyhen,adponce,hgete,vruiz}@vicomtech.org Abstract: We describe an adaptive neural machine translation platform which integrates continuous learning and supports multiple use-cases in the translation in- dustry. The application is being developed and evaluated within the applied research project ITAI. Research within the project has shown the potential of the platform to cover the main identified use cases and provide rapid adaptation via continuous learning. Keywords: Neural machine translation, Continuous learning Resumen: Este artı́culo presenta una plataforma de traducción automática neu- ronal que integra aprendizaje continuo y da soporte a múltiples casos de uso en la industria de la traducción. La aplicación se está desarrollando y evaluando en el marco del proyecto de investigación aplicada ITAI. La investigación realizada re- salta el potencial de la plataforma para dar soporte a los casos de uso identificados y proveer una rápida adaptación de las traducciones mediante aprendizaje continuo. Palabras clave: Traducción automática neuronal, Aprendizaje continuo 1 Introduction Secondly, MT systems are usually only up- Neural Machine Translation (NMT) (Bah- dated periodically when significant volumes danau, Cho, and Bengio, 2015; Vaswani et of new training data become available and, al., 2017) has brought significant gains in Ma- therefore, do not provide timely adaptation chine Translation (MT) quality and has be- of MT output corrections generated via post- come the dominant paradigm in both aca- editing. This limitation can result in a loss of demic research and commercial exploitation. productivity and increased frustration on the This technology is being increasingly inte- part of translators tasked to repeatedly cor- grated in the translation industry to support rect identical errors over time when querying growing translation needs in the digital era. MT engines. Continuous learning (CL) ad- Providing adequate support to the trans- dresses this issue via continuous updates of lation industry requires taking two main as- MT models on the basis of post-edited ma- pects into account. chine translation output fed back to model First, actual practices in the industry fea- training processes. In NMT, CL usually takes ture a wide array of scenarios depending on the form of Online Learning (OL), where the IT infrastructure at hand and the net- each new pair of source sentence and post- work of translators working for a specific edited translation is used to update the cor- company in the field. Translation may thus responding model (Peris and Casacuberta, be performed via computer-assisted transla- 2019; Turchi et al., 2017; Wuebker, Simianer, tion (CAT) tools such as SDL Trados Stu- and DeNero, 2018; Domingo et al., 2019), dio or Wordfast, to name two of the main although CL could also be performed via ones, in Content Management System (CMS) micro-batches with slightly delayed integra- environments, or directly performed in doc- tion of user feedback. ument editors such as Libre Office or MS One key aspect in CL is determining the Word. This disparity makes it difficult to proper trade-off between rapid adaptation of bring MT technology to a significant portion the models from user corrections and model of the translation industry. stability over time, a topic which has only Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 49 been partially explored so far. Optimal inte- continuous learning to produce incremental gration of CL methods remains a matter of improvements of the models. active research and is of key importance to The ITAI UI is implemented in Angular provide useful adaptive machine translation and the back-end components in Go. We de- technology. scribe the main components and functional- In this paper, we describe ITAI, an ity in more detail in the following sections. adaptive neural machine translation plat- form which integrates continuous learning 2.1 Front-ends and supports multiple use-cases in the trans- As noted in the introduction, translation ac- lation industry. The platform is being de- tivities in the industry cover a wide array of veloped within the applied research project usage. To provide support for the main iden- ITAI, partially supported by the Department tified use cases, the application supports dif- of Economic Development of the Basque ferent entry points. Government (Spri). The project started in We first addressed the most commonly April 2019 and will finalise in December 2021. used frameworks for multilingual content It is carried over by the following consor- generation. CAT tools are an important part tium: MondragonLingua1 (project coordina- of the translation ecosystem, and we devel- tor), Emun2 , iAmetza3 , Mixer4 , Tai Gabe5 oped a specific plugin to connect the pop- and Vicomtech6 . The project takes into ac- ular SDL Trados environment to the appli- count the translation requirements from each cation, similarly to Domingo et al. (2019). company and continuous translators’ feed- Other frameworks such as Wordfast Classic back across development cycles. have also been configured and tested within the project. Other CAT environments with 2 ITAI support for custom MT can be easily con- The architecture of ITAI is described in Fig- figured to interact with ITAI via its REST ure 1. The application consists of the follow- API. Additionally, ITAI supports integration ing main elements: within CMS environments, and specific de- velopments are being carried out within the • Front-ends, from which users may inter- project for the Ubiquo7 environment. act with the application. The front-ends We also developed a Web-based user inter- include a web-based user interface (UI), face to provide an additional access point to plugins for specific CAT tools, and sup- the functionalities of the system. Such an en- ports CMS integration. vironment was identified as necessary for two main reasons. First, translation is also being • A REST API, which exposes the func- carried out professionally outside dedicated tionality of the back-end and handles environments such as CAT tools and with- user authentication. out any technological support to improve pro- • A back-end, which includes the required ductivity. Secondly, some proprietary CAT components to perform machine trans- tools do not support the transfer of post- lation, manage the data generated from edited translations to external applications the use of the system, and manage the and users’ feedback cannot be reflected in the training and selection of continuously MT models. In either case, users are thus updated NMT models. limited in their interaction with supporting MT technology. The core workflow involves users request- To address these issues, the ITAI UI of- ing machine translation of texts or docu- fers a full-fledged access to the MT technol- ments, post-editing the automated transla- ogy supported by the back-end. Users with tions as needed, and sending validated trans- little or no access to MT technology may thus lations to the system. These validated trans- upload documents and retrieve automatically lations are fed back to the NMT models via translated documents maintaining the origi- 1 https://www.mondragonlingua.com/en/ nal format. The translated documents can 2 https://www.emun.eus/en/ then be post-edited in an external environ- 3 https://iametza.eus/ ment and the resulting validated translations 4 http://www.mixer.com.es/es/mixer/ uploaded via the UI, where the content will 5 https://www.naiz.eus/ 6 7 https://www.vicomtech.org https://www.ubiquo.me/ 50 Machine translation Document translation MARIANNMT Pre/post-processing Text translation UI Translated data Data management Translate text | document Content extraction Alignment Corpora Filtering Post-edit and validate Data selection CAT API Validated data Model management Online learning Batch learning Models CMS Evaluation Model selection FRONTENDS BACKEND Figure 1: ITAI Architecture be extracted to feed the MT models. Dual of documents in a variety of formats (odt, use-cases are also supported, where users docx, xlsx, pptx, html or xliff, among others). may work with a CAT tool that supports the It performs content extraction, text transla- integration of ITAI MT services, and upload tion via itzuli-translator, and document re- translation memories or documents contain- construction with format preservation. ing their post-edited data via the ITAI UI. Itzuli is a validated platform which sup- Additionally, the UI provides a simple ed- ports large-scale translation services and pro- itor where users can directly post-edit ma- vides ITAI with robust MT functionality.8 chine translated output that has been auto- All NMT models currently deployed in matically segmented and filtered, as a default ITAI are Transformer models (Vaswani et environment to correct and validate transla- al., 2017), trained on large volumes of par- tions prior to sending them for integration in allel, comparable and synthetic data. Al- the MT models. though the platform is agnostic in terms of The user interface also provides dash- language pairs and domains, special empha- boards to monitor volumes of translated and sis is placed within the project on transla- validated data, and list content that is pend- tion between Basque and English, French or ing validation to dissociate use of MT from Spanish, to contribute to improving language the provision of post-edited data, as priorities technology for the Basque language. usually differ for these two activities. 2.2.2 Data management 2.2 Back-end As one of the main goals of the project is to The ITAI back-end provides support for three gradually increase the quality of MT models main types of functionality, which we de- via continuous learning from user-generated scribe in turn in the next sections. corrections and validations, data manage- 2.2.1 Machine translation ment is a key functionality of the platform. Since the platform allows for the provision Machine translation is carried out with Vi- of post-edited data via documents, in addi- comtech’s Itzuli MT toolkit, via its two main tion to the provision of segment-level data, components: itzuli-translator and itzuli- the component supports sentence alignment doctrans. with a combination of the metrics generated The former performs text translation and can be deployed in scalable Kubernetes mode 8 It notably supports the internal and pub- or as a standalone platform in a dedicated lic MT services of the Basque Government server. It integrates MarianNMT (Junczys- (https://www.euskadi.eus/traductor/), Mondrag- Dowmunt et al., 2018) to perform efficient onLingua’s commercial MT services for Basque (https://lingua.eus/eu/itzultzailea) and domain- NMT inference and training. specific translation, and Vicomtech’s public platform Document translation is done via itzuli- for the improvement of Basque translation technology doctrans, a robust component for translation (https://www.batua.eus/). 51 by the HunAlign (Varga et al., 2005) and potential of the platform to cover the main STACC (Etchegoyhen and Azpeitia, 2016) identified use cases and provide rapid adap- aligners. It also performs several types of fil- tation via continuous learning. It also un- tering, to identify misaligned or noisy data, covered the need to further explore contin- via alignment scores and regular expression- uous learning for neural machine translation based filters. Data selection is then per- to reach an optimal balance between rapid formed to determine relevant data for con- adaptation and model stability over time. tinuous learning, given previous history. Finally, the component also generates References translation memories from validated aligned Bahdanau, D., K. Cho, and Y. Bengio. data, which users can download as a by- 2015. Neural machine translation by product of the data management processes. jointly learning to align and translate. In 2.2.3 Model management Proc. of ICLR. Previously unseen data validated by users Domingo, M., M. Garcı́a-Martı́nez, A. Es- reach the model management component, tela Pastor, L. Bié, A. Helle, Á. Peris, where continuous learning takes place. New F. Casacuberta, and M. Herranz Pérez. pairs, consisting of a source sentence and its 2019. Demonstration of a neural machine validated translation, are used to adapt the translation system with online learning for relevant models, with a single update for on- translators. In Proc. of ACL, pages 70–74. line learning using the appropriate learning Etchegoyhen, T. and A. Azpeitia. 2016. Set- rate for the selected optimiser. Automatic Theoretic Alignment for Comparable Cor- evaluation then takes place to measure the pora. In Proc. of ACL, pages 2009–2018. impact of the update on both the new pairs and static test sets for the models at hand. Junczys-Dowmunt, M., R. Grundkiewicz, Although online learning is a relevant T. Dwojak, H. Hoang, K. Heafield, framework to adapt MT models on the fly, T. Neckermann, F. Seide, U. Germann, it is still an open matter to determine an op- A. Fikri Aji, N. Bogoychev, A. F. T. Mar- timal balance between aggressive adaptation, tins, and A. Birch. 2018. Marian: Fast required for online learning to take effect on neural machine translation in C++. In the basis of single data points, and model sta- Proc. of ACL, pages 116–121, July. bility over time, necessary to maintain the Peris, Á. and F. Casacuberta. 2019. On- overall quality of the models. line learning for effort reduction in inter- Several experiments are being carried out active neural machine translation. Com- within the ITAI project to determine the ap- puter Speech & Language, 58:98–126. propriate configurations in this respect. Cur- Turchi, M., M. Negri, A. Farajian, and rent results tend to favour a hybrid approach, M. Federico. 2017. Continuous learning with online learning performed for rapid MT from human post-edits for neural machine adaptation useful to the users of the system, translation. The Prague Bulletin of Math- and batch fine-tuning over prior model train- ematical Linguistics, 108:233–244. ing checkpoints once the volumes of accumu- lated new data reach a significant threshold. Varga, D., L. Németh, P. Halácsy, A. Kor- Model management processes for continuous nai, V. Trón, and V. Nagy. 2005. Parallel learning will be adapted as necessary as fi- corpora for medium density languages. In nal conclusions are reached within the project Proc. RANLP, pages 590–596. regarding continuous learning for neural ma- Vaswani, A., N. Shazeer, N. Parmar, chine translation. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Atten- 3 Conclusions tion is all you need. In Advances in Neu- In this paper, we described a neural machine ral Information Processing Systems, pages translation platform which supports contin- 6000–6010. uous learning and multiple use cases in the Wuebker, J., P. Simianer, and J. DeNero. translation industry. The application is al- 2018. Compact personalized models for ready operative within the applied research neural machine translation. In Proc. of project ITAI and will be finalised in 2021. EMNLP. Research within the project has shown the 52