The paper presents a novel approach to developing Ukrainian-English machine translation technology using neural machine translation. A comparative analysis of the developed software product with analogues such as Google Neural Machine Translation, Microsoft Translator and OpenNMT is conducted. The study focuses on the analysis of their main advantages and disadvantages. Here, we introduce an effective solution for the typical architecture of the intelligent Ukrainian-English machine translation system based on recurrent neural networks. The main functional requirements for similar systems are determined, and the technical task is developed. The neural network model test results have shown that the confirmation loss stopped decreasing after 25 epochs.
Machine translation (MT) is widely used in various areas of our lives as it speeds up natural language translation and improves the traditional translation process. Its relevance is due to the constant demand for translation as a type of information activity and the rapid increase of information exchange value. The digital translation still needs improvement, but the text obtained from an electronic translator allows you to understand the document’s meaning in most cases. Further, the user may edit the document with basic knowledge of a foreign language and a command of the subject area the translated information belongs to. Due to the development of machine learning technologies, it is possible to achieve better results than ever. Neural Machine Translation (NMT) is a recently proposed approach to machine translation that differs from traditional statistical machine translation as it focuses on building a single artificial neural network that can be jointly configured to maximise translation performance.
This work aims to study the use of recurrent neural networks for machine learning; to determine the main features and principles of the sequence-to-sequence model for deep learning in machine translation. The subject of research is a recurrent neural network for machine translation. The object of research is machine translation.
The study material is a set of bilingual pairs of sentences with tab separators taken from the open
database of sentences and translations Tatoeba [
Unlike other approaches, such as neural/rule-based/statistical machine translation uses an extensive neural network that applies artificial intelligence to work like the human brain. It is the most advanced form of machine translation available, with tremendous progress made in recent years through AIpowered self-learning, huge data collection and deep learning. Modern neural machine translation engines are a basis for developing professional translations. Recent technological advances have enabled an increasing number of multinational institutions to use NMT engines to support internal and external communication.
Advantages of using neural machine translation:
High accuracy: drawing from ever-extending data sets and language modelling, NMT engines
can mine the broader context of words and phrases to create more accurate and fluent
translations, which can be improved over time. In contrast, conventional phrase-based MT only
considers the context of a few words next to the translated word.
Fast training: neural networks can be trained quickly using automated processes instead of the
expensive and largely manual methods required for rule-based MT.
Simple integration and flexibility: the advantage that NMT carries over its statistical
predecessor includes its integration via an application programming interface (API) and
software development kit (SDK) into any software and applied to many content files formats.
Customisation: to improve the results, the user can set up the NMT output and update the model
with terminological databases, brand-specific glossaries, and other data sources.
Cost-efficiency: human translation can be expensive, especially in projects that involve long
texts and many languages. NMT produces highly accurate and fast translation systems at low
costs. If needed, the user may address human translators for post-editing.
Scalability: when your translation needs to scale up, neural machine translation can help meet
increased demand quickly and efficiently [
In recent years, the technology of neural machine translation, based on artificial neural networks
with representation learning (deep learning), has rapidly developed. That is why many companies have
decided to use this technology for their products. Despite several reviews in the literature that address
the importance of neural machine translation, none of the recently published articles has
comprehensively examined the critical roles of these technologies in the process of Ukrainian-English
translation based on a recursive neural network. In our research, we will first study and discuss these
products (Fig. 1) [
Google Neural Machine Translation (GNMT) is a neural machine translation (NMT) system
developed by Google. The GNMT network can undertake interlingual machine translation by encoding
the semantics of the sentence rather than by memorising phrase-to-phrase translations. Next, we will
consider the main benefits and drawbacks of the GNMT system [
The benefits of Google neural machine translation can be summed up as follows: Google Translate is free. An experienced professional translator can sometimes be costly, but the quality balances the expenses spent.
Google Translate is quick. One of the main advantages of Google Translate is that it is speedy. A
human translator(s) cannot compete with the speed nor, as a result, the number of translations that
Google Translate can perform. In an average workday, an experienced translator can translate about
2,000 words maximum (300-400 words/hour), depending on the difficulty of the text. In contrast,
Google Translate can produce a translation with the same number of words in seconds [
The main drawbacks include the following: Google Translate may “lose” the word meaning because there is no way to incorporate context.
The complexity of the text and any context that cannot be interpreted without accurate language knowledge make the error probability higher. Direct translation is standard with Google Translate and often results in nonsensical literal translations, while professional translators make much effort to ensure this does not happen. They use well-established online glossaries, back translation methods, proofreaders and reviewers to achieve this.
The quality of translation relies on the language pair. The source and target languages involved also affect the quality of the translation. Since Google’s web-based translation database is built primarily from existing online translations, standard translations for languages, e.g. German or English, tend to be more accurate. The translations for other languages that are not available in Google’s database are less likely to be precise.
Google Translate often translates with significant grammatical errors. Since Google’s translation
system uses a method based on language pair frequency, grammatical rules are not considered [
Microsoft Translator is the main competitor of Google Translate. The advantages of the Microsoft
Translator App include the following features [
The Microsoft Translator App also has the following disadvantages: Limited Languages. Interpretations of One Type. Not Always Accurate.
Open-Source Neural Machine Translation (OpenNMT) is an open-source ecosystem for neural
machine translation and neural sequence learning [
It provides implementations in two popular deep learning frameworks:
1. OpenNMT-py: user-friendly and multimodal, benefiting from PyTorch ease of use.
2. OpenNMT-tf: modular and stable, powered by the TensorFlow ecosystem [
In summary, we may conclude that the advantage of neural machine translation systems is that they are end-to-end models that do not have a pipeline of specific tasks. The disadvantage is the need for a bilingual set and the ongoing problem of processing rare words.
The Python programming language and the Jupyter programming environment are used to develop the neural machine translation system. A set of Keras tools is employed for the natural process of building the model.
Python. A brief review of the programming language itself is worth our attention. Python is the most common programming language for artificial neural networks. It has a low entry threshold: no one writes neural networks in Python from scratch because it is time-consuming. There are libraries for Python neural networks already written by experts. Thus, a whole neural network community has formed around Python.
The advantages of using Python include the following main features: Conciseness and interoperability: The language allows you to develop complex algorithms in a short time. It is distinguished by simplicity, conciseness and expressiveness. In addition, it has a powerful interoperability mechanism with C\++, which allows for fast calculations.
Because of this, engineers can create simple and complex neural networks in Python. Flexibility: Neural networks are primarily small programs, but there is a need to change them often, choosing the best architecture, data processing, and other parameters. Therefore, there are practically no difficulties with legacy code, but there is a need for rapid development. Creating and building neural networks in Python is an option that meets these requirements better than using C++ or Java.
Keras. The next stage of our study is to consider the toolkit built based on TensorFlow Keras. It is
a powerful platform that can scale to large clusters of GPUs or an entire TPU module [
Figure 2 shows the popular open-source NMT toolkits on GitHub.
Jupyter Notebook integrates code and its output into a single document that combines
visualisations, narrative text, mathematical equations, and other important media. We assume it is a
single document where users can run code, display the output, add explanations, formulas, and charts,
and make their work more transparent, understandable, repeatable, and shareable [
Using Notebooks is now a significant part of the data science workflow at companies across
the globe. If the software developer’s goal is to work with data, using a Notebook will speed up their
workflow and make communicating and sharing the results more accessible. Another essential feature
of the programme is that as part of the open-source Project Jupyter, Jupyter Notebooks are entirely free.
The software can be downloaded independently or as part of the Anaconda data science toolkit [
NMT Model. The general architecture of our model is explained below. It is a
sequence-tosequence (Seq2Seq) neural network model with inner layers of recurrent neural networks [
Each Seq2Seq model contains two main components:
Encoder outputs some value for each word in the input sentence. For every input word, the
encoder outputs a vector and a hidden state and uses the hidden state for the next input word
[
Long Short-Term Memory (LSTM) is a type of RNN capable of learning long-term
dependencies. All recurrent neural networks have the form of a chain of recurrent neural network
modules. LSTMs also have this chain architecture, but the repeated module has a different structure.
Instead of having one layer of a neural network, four interact exceptionally [
The visualisation of the architecture of our neural machine translation model is shown in Fig. 5 [
There are several ways we can formulate the task of training an RNN to write text, in this case,
Ukrainian-English translation. However, we choose to introduce it as a many-to-one sequence mapper
[
Use case diagram is the initial conceptual model of the system in the process of its design and development (Fig. 6). It consists of actors, use cases, and relationships between them. The following interfaces are used in this diagram: A diagram of components of the neural machine translation system model (ITP) is an interface for transferring downloaded text data to the input of the word processor. Interface Input Text Processing (IITP) – an interface for transmitting user-entered text to a primary word processing module. Interface Machine Translation (IMT) is used for loading the most effective stored machine translation model for further use in the process of neural machine translation of pre-processed text entered by the user. Interface Machine Translation Model Training (IMTMT) transmits the original processed (cleaned, normalised, tokenised, digitised) training data to the machine translation model training input. Interface Processing-Machine Translation (IPMT) transfers the output (from the text translation module) processed text data, entered by the user to be translated, to the input module of the direct NMT. Interface User-Application (IUA) is the interface for transferring the resulting translation of the user’s text to the User Interface module for its final output.
Activity diagram is another crucial behavioural diagram in UML diagram to describe dynamic aspects of the system. An activity diagram is an advanced flow chart that models the flow from one activity to another in the system [25-27].
An activity diagram is a great tool for achieving the following goals: to write an algorithm logic; to illustrate a business process or workflow between users and the system; to simplify and improve any process by identifying complex use cases; to model software architecture elements such as method, function, and operation. The first thing we need to do is import the necessary libraries.
Next, we will write a function for reading data in Jupyter Notebook [
Then, another function for dividing the text into English-Ukrainian pairs separated by a symbol’ \n’
is defined. Further, we correspondingly split these pairs into English and Ukrainian sentences [
Now we can use these functions to read text into an array of the necessary format [
The actual data consists of over 150,000 sentence pairs. However, we will only use the first 50,000
per item to reduce the training time of the model [
ua_eng = ua_eng [: 50000 ,:]
Next, we pre-process the text. It is an essential step in any project, especially in NLT. The data we
process is often unstructured, so there are certain things we need to take care of before moving on to
the model-building stage [
a) Text cleaning First, we analyse our data as it helps us decide which pre-processing steps to take: ua_eng
The next step is to remove punctuation and convert text into lowercase. # Remove punctuation ua_eng[:,0] = [s.translate(str.maketrans('', '', string.punctuation)) for s in ua_eng[:,0]] ua_eng[:,1] = [s.translate(str.maketrans('', '', string.punctuation)) for s in ua_eng[:,1]] for i in range(len(ua_eng)): # convert text to lowercase ua_eng[i,0] = ua_eng[i,0].lower() ua_eng[i,1] = ua_eng[i,1].lower() b) Converting text into a sequence
The Seq2Seq model requires us to convert input and output sentences into fixed-length integer
sequences. Before doing that, we visualise the length of the sentences by collecting the length of all
sentences in two separate lists for English and Ukrainian languages [
for i in ua_eng[:,1]:
ua_l.append(len(i.split())) length_df = pd.DataFrame({'eng':eng_l, 'ua':dua_l}) length_df.hist(bins = 30) plt.show()
We may assume that the maximum length of English sentences is six words, and there are eight in
Ukrainian sentences. Next, we vectorise our text data using the Keras Tokenizer() class. It turns out
sentences into sequences of integers. We can then pad these sequences with zeros to make all rows the
same length. And afterwards, the lexemes for both Ukrainian and English sentences are prepared [
Then we proceed to build our model. First, we divide the data into training and test sets for model
training and its evaluation [
The next step is to code our sentences. We encode Ukrainian sentences as input sequences and English sentences as target sequences. It is done for both training and testing datasets.
Subsequently, we define the architecture of the model. As mentioned earlier, we use an embedding
layer and an LSTM layer for the encoder; for the decoder, we use another LSTM layer and then a dense
layer [
We use the RMSprop optimiser in this model because it shows positive results in dealing with
recurrent neural networks [
# model compilation (with 512 hidden units) model = build_model(ua_vocab_size, eng_vocab_size, ua_length, eng_length, 512) rms = optimizers.RMSprop(lr=0.001) model.compile(optimizer=rms, loss='sparse_categorical_crossentropy') We should mention that we use ‘sparse_categorical_crossentropy’ as the loss function. It is reasoned by the function’s ability to use the target sequence in its pure state instead of a one-time encoded format. Using such a vast vocabulary, the former encoding of target sequences can fully exhaust the system’s memory. We train our model for 30 epochs, with a batch size of 512 and a validation distribution of 20%. The model is trained with 80% of the data; the rest is left for evaluation.
Finally, we can download the saved model and make predictions on unseen data – testX [
The obtained sequences of integers representing predictions are converted into an English text: Next, we place the original English sentences in the test data set and the predicted sentences in the data frame:
pred_df = pd.DataFrame({'UA': test[:,1], 'actual_ENG' : test[:,0], 'predicted_ENG' : preds_text}) pd.set_option('display.max_colwidth', 200) Finally, we can run a benchmark test and review the results (Fig. 10 - 11).
pred_df.head(15)
pred_df.tail(15)
Now we would like to describe the model training statistics. For this, we apply a function called ModelCheckpoint() [28]. Application of the ModelCheckpoint():
The following arguments of the ModelCheckpoint function are used: filename : string or PathLike, path to save the model file. The directory of the filepath should not be reused by any other callbacks to avoid conflicts [28].
monitor : the metric name to monitor.
Note: Prefix the name with “val_” to monitor validation metrics. Use “loss” or “val_loss” to monitor the model’s total loss [28].
verbose : verbosity mode, 0 or 1. Mode 0 is silent, and mode 1 displays messages when the callback takes action [28].
save_best_only: if save_best_only=True, it only saves when the model is considered the “best” and the latest best model according to the quantity monitored will not be overwritten.
mode : one of {‘auto’, ‘min’, ‘max’}. If save_best_only=True, the decision to overwrite the current save file is made based on either the maximisation or the minimisation of the monitored quantity. For val_loss, this should be min [28].
So, we will build a graph and compare the model’s training loss and validation loss. plt.plot(history.history['loss']) plt.plot(history.history['val_loss']) plt.legend([‘train’, ‘validation’]) plt.show()
As we can see in the diagram above, the validation loss ceased decreasing after 25 epochs.
In the course of work, we developed a project, the objective of which was to design a system of Ukrainian-English translation using the means of neural machine translation. First, we discussed essential considerations in developing the research question, defined objectives and tasks, constructed the subject and object of research, scientific novelty and practical value of the developed project.
The first stage of our study compared the developed product with analogues such as Google Neural Machine Translation (GNMT), Microsoft Translator and OpenNMT were carried out, determining their advantages and disadvantages. Then, a system analysis of the developed product was conducted, and UML diagrams were constructed (use case diagram, components diagram and activity diagram). After that, the research hypothesis and the technical task were formulated.
The next stage of our study dealt with selecting methods and means of the developed product. We proved the advantages of Python, the Keras toolkit and the Jupyter Notebook development environment as the necessary tools for our work. The general architecture of the Seq2Seq model using LSTM layers was also presented. Having analysed the software process and its structure, we performed system testing to evaluate system specifications and describe the results. Then we made the training statistics of the model with a description of its outcome. All things considered, it seems reasonable to assume that future research should explore improving the translation’s system accuracy and increasing the model’s speed, which will involve exploring additional or alternative architectural solutions.
Part of this paper was written within the project which received funding from the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I0303-V01-00118. 7. References [24] Dive into Deep Learning. Machine Translation and the Dataset. URL: https://d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset.html [25] Xuanming Zhang. Analysis of State of the Art Deep Learning based Techniques for Medical
Natural Answer Generation. URL: https://www.billyzhang.me/uploads/billyzhang_thesis.pdf [26] Deepika Singh. Natural Language Processing - Machine Learning with Text Data. URL: [27] https://www.pluralsight.com/guides/nlp-machine-learning-text-data [28] ModelCheckpoint. URL: https://keras.io/api/callbacks/model_checkpoint/. [29] Callbacks API. URL: https://keras.io/api/callbacks/ [30] M. Garcarz, Legal Language Translation: Theory behind the Practice, CEUR Workshop
Proceedings, Vol-3171 (2022) 2-2. [31] N. Hrytsiv, I. Bekhta, M. Tkachivska, V. Byalyk, Sylvia Plath’s I felt-Narrative Label of The Bell
Jar in Ukrainian Translation, CEUR Workshop Proceedings, Vol-3171 (2022) 240-255. [32] M. Bekhta-Hamanchuk, H. Oleksiv, T. Shestakevych, Y. Shyika, Quantitative Parameters of J.
London's Short Stories Collection “Children of the Frost” and its Translation, CEUR Workshop Proceedings, Vol-3171 (2022) 697-710. [33] K. S. Mandziy, U. V. Yurlova, M. P. Dilai, English-Ukrainian Parallel Corpus of IT Texts:
Application in Translation Studies, CEUR Workshop Proceedings, Vol-3171 (2022) 724-736. [34] N. Hrytsiv, T. Shestakevych, J. Shyyka, Corpus Technologies in Translation Studies: Fiction as
Document, in: CEUR Workshop Proceedings, Vol-2917 (2021) 327-343. [35] A. Lutskiv, R. Lutsyshyn, Corpus-Based Translation Automation of Adaptable Corpus Translation
Module, CEUR Workshop Proceedings, Vol-2870 (2021) 511-527. [36] V. Lytvyn, P. Pukach, V. Vysotska, M. Vovk, N. Kholodna, Identification and Correction of Grammatical Errors in Ukrainian Texts Based on Machine Learning Technology. Mathematics 11 (2023) 904. https://doi.org/10.3390/math11040904 [37] A. Kopp, D. Orlovskyi, S. Orekhov, An Approach and Software Prototype for Translation of Natural Language Business Rules into Database Structure, CEUR Workshop Proceedings 2870 (2021) 1274-1291. [38] S. Kubinska, R. Holoshchuk, S. Holoshchuk, L. Chyrun, Ukrainian Language Chatbot for Sentiment Analysis and User Interests Recognition based on Data Mining, CEUR Workshop Proceedings, Vol-3171 (2022) 315-327. [39] A. Dmytriv, S. Holoshchuk, L. Chyrun, R. Holoshchuk, Comparative Analysis of Using Different Parts of Speech in the Ukrainian Texts Based on Stylistic Approach, CEUR Workshop Proceedings, Vol-3171 (2022) 546-560.