To find correct translation of an input sentence in Machine Translation is not an easy task of Natural language processing (NLP). The hybridization of different translation models has been found to handle this problem in an easy way. This paper presents an approach that takes advantage of various translation models by combining their outputs with statistical machine translation (SMT) and transformer method. Firstly, we achieve Google Translator and Bing Microsoft Translator outputs as external system outputs. Then, outputs of those models are fed into SMT and Transformer. Finally, the combined output is generated by analyzing the Google Translator, Bing, SMT and Transformer output. Prior work used system combination but no such approach exist which tried to combine the statistical and transformer system with other translation system. The experimental results on English-Hindi and Hindi-English language have shown significant improvement.
Machine Translation is the main area of
Natural Language Processing. There are
various translation approaches each with its
pros and cons. One of the recent and existing
approaches of Machine Translation (MT) is
Statistical Machine Translation (SMT). The
Statistical system is [
The combining these approaches into one is
not an easy task. By using either SMT or
Transformer does not give the solution to all
issues. NMT has a problem of over-translates
and under-translates to some extent. Also long
distance dependency, phrase repetitions,
translation adequacy for rare words and word
alignment problems are observed in neural
based system. As SMT [
The paper is framed as following: In Section 2, a brief introduction of hybrid approaches proposed for Machine Translation. Section 3 elaborates our proposed approach. The experiments undertaken in this study have been discussed along with the results obtained in Section 4. In Section 5, the conclusion is presented.
Many methods have been presented in
literature for machine translation. Researchers
combined different translation techniques [
The Example-based, Knowledge-based and
Lexical transfer system combined using chart
manager in [
Minimum Baye’s risk system combination
(MBRSC) method [
The lattice-based system combination model
[
Multi-engine machine translation
amalgamated output of several MT systems
into single corrected translation [
The Transformer model [
The attention mechanisms:- Both selfattention mechanism and cross-lingual attention are computed as follows: Attention (Q, K, V) = softmax ( ) V (3)
Here, Q represent Query vector, V and K represent as Value and Key vector of both encoder and decoder respectively and dmodel is the size of this key vector. The product of Query and Key represents the similarity between each element of Q and K and it is converted to a probability by using the softmax function, which can be treated as weights of attention of Q to K.
The self-attention captures the degree of association between words in the input sentence by using the Q, K and V in the encoder. In similar way, the self-attention in the decoder captures the degree of association between words of output sentence by using Query, Key and Value in the decoder. The cross-lingual attention mechanism computes the degree of association between a word in source and target language sentence by using Query of decoder and output of last layer of encoder as Key and Value. In the multiple head self-attention with h number of heads, Q, K, and V are linearly projected to h subspaces, and then the attention function is used in parallel on each subspace. The concatenation of these heads is projected to a space with the original dimension. MultiHead (Q, K, V) = Concat (head1, …. , headh)WO (4) headi = Attention (Q , K , V ) (5) where, ∈ , ∈ , ∈ , WO ∈ , are weight matrices and Concat is a function that concatenates two matrices. Multiple head attention learns information from representation spaces at different positions. The Transformer uses position encoding (PE) to encode the position related information of each word in a sentence because the Transformer does not have any recurrent or convolution structure. PE is calculated as follows: PE (pos, 2i) = sin (pos/ PE (pos, 2i + 1) = cos(pos/ )
(6) ) (7) where, i is dimension or size, and pos is absolute position of the word.
Fig 1. Translation System Architecture
The transformer system inputs are the translated outputs of different or external translation methods and same source sentence as shown in Fig 1. Then, we used three encoders: one for Google output text, another uses Bing and source language encoder. The concatenation generated by the three encoders is fed into conventional Transformer decoder.
The Google output text encoder represented as X1 = (x11, x12, ..., x1N) input. The Bing output sentence represented as X2 = (x21, x22, ..., x2N) and third transformer encoder accepts a source language sentence X3 = (x31, x32, ..., x3N) as an input. Then, Transformer encoder generates the intermediate representation hg (t = 11,...., 1N), hb (p = 21,...., 2N), ht (t = 31,...., 3N), from the source language sentence X1, X2 and X3. The intermediate representation hg, hb, ht: hg = Transformer Encoder(X1) hb = Transformer Encoder(X2) ht = Transformer Encoder(X3) (8) (9) (10)
After the intermediate representations of
inputs, these are concatenated in the
composition layer [
h = Concat (hg, hb, ht) (11)
The decoder generated the combined target language output using the above expressions.
The phrase translation method or Baye’s Rule
forms the basis of Statistical Translation [
ebest = argmaxe P(e|f) = argmaxe [P(f|e) PLM(e)] (12) where, f is source sentence and e is target sentence. PLM(e) and P(f |e) are language model (LM) and the translation model (TM) , respectively. The input text f is partitioned uniformly into a sequence of T phrases . Each foreign phrase in is translated into english phrase. The translation model P(f|e) is disintegrated into:
P( | ) = d (αi – βi-1) (13)
The phrase translation is formed by probability distribution . The relative distortion probability distribution d (αi – βi-1) calculates the output phrases.
The Statistical combination approach uses three modules: Alignment module, Decoding and Scoring. The alignment is useful for string alignments of the hypotheses generated from different machine translation systems. A decoding step builds hypotheses using aligned strings from previous step by using beam search algorithm. The final scoring step helps in estimating the final hypotheses.
The single best outputs d1, d2, ….dm from each
of the m participating systems are taken into
consideration. We take sentence pairs di and dj,
and strings between the sentence pairs are
aligned. For m sentences,
possible
sentence pairs are required to be aligned. The
string w1 in sentence di aligned to string w2 in
sentence dj following two conditions:- Firstly,
w1 and w2 are same. Then, w1 and w2 have Wu
and PaLMer [
In decoding, best outputs are combined of participating systems to form a set of hypothesis. The first word of a sentence is used to start the hypothesis. At any moment of time, the search can be shifted to a different sentence or addition of the new words continued using words from the previous sentence. Let a word w is added to the hypothesis, taken from the best output di or shift to different output dj. On shifting, the first left over word from best output sentence is added to next hypothesis. With the help of this method, a hypothesis can be made using various system outputs. If a hypothesis made up of at most single word from each set of aligned words, there is less possibility of occurrence of duplication.
The search space is easily switched through sentences, and thus maintaining adequacy and fluency is difficult. Therefore, hypothesis length, language model probability and number of n-gram matching between individual system’s output and hypothesis, features are used for complete hypothesis.
In this search, the number of equal length
hypotheses is assigned to beam. The
hypotheses are recombined by feature state,
history of the hypothesis appropriate to the
length requested by features and search space
hashing. Then, pointers are maintained of
recombined hypotheses that are packed into
single hypothesis. Therefore, it enabled
extraction of k-best.
3.4.4. SCORING
In the output of decoding steps, the m-best lists
are generated. Language Model Probability
[
The proposed approach is tested on
HindEnCorp [
We trained and tested our approach on one more dataset from ILCC (Institute for Language, Cognition and Computation) for English to Hindi language which contains 43,396 sentences. For Hindi to English translation, we used TDIL-LC (Indian Language Technology Proliferation and Deployment Centre) dataset divided into tourism (25k sentences) and health (25k sentences) domain. Therefore, Hindi to English language pair trained and tested on 50k sentences.
We train Statistical Machine Translation
with KneserNey smoothing [
The Transformer1 system contains encoder
and decoder six layers, eight attention heads,
and 2048 feed-forward inner-layer size or
dimensions with dropout = 0.1. The hidden
state and word embedding dimension dmodel is
512. We limit maximum sentence length to
256, and input and the output tokens per batch
are limited to 2048. We used Adam optimizer
[
The Bilingual Evaluation Understudy
(BLEU) [
The results on different test sets are obtained for Hindi to English (Hi-Eng) and EnglishHindi (Eng-Hi) language pairs. It is evident from Table1 that translation system combination shows better results than individual system i.e. SMT with Google and Bing improved approximately 4 bleu scores than SMT alone in all language pairs. The output from individual system contains some erroneous or un-translated words. But the selection of best phrase among different translated outputs (Google and Bing) generated by participating systems makes the target sentence more accurate. 1 https://github.com/tensorflow/tensor2tensor
BLEU Score Eng-Hi (ILCC)
We also observe in the Table1 that by increasing number of MT system does not help in improving accuracy i.e. Bleu scores of SMT, Transformer, Google and Bing together achieved 2 points less than SMT, Google and Bing. The BLEU score achieved by using SMT, Bing Microsoft and Google translator together are highest. Also the scores of Transformer, Google and Bing are better than using all translation models and bleu scores improved by 1 point. The scores retrieved using TDIL-DC and ILCC dataset are lesser than HindiEnCorp because the size of dataset is very less. The overall accuracy of our translation output using reverse language pair is improved by combining the better parts of outputs. But, the Bleu scores are not improved much in our approach. The main reason is that the error occurred in external machine translation systems, would also reflect in the combination approach. Hence, by removing these errors, we will try to achieve better results in future.
We investigated the approach of combining the various translation outputs with statistical machine translation and Transformer which improve the final translation in this paper. The proposed method increased the complexity of the overall system. Experimentation on Hindi to English and from English to Hindi shows that incorporating different system output achieves better result than individual system. In future, we extend this approach for translation of other language pairs and tasks like text abstraction, sentence compression. We will also try to incorporate BERT model into the neural based English to Hindi translation and will also explore the graph based encoder-decoder translation methods.