A huge amount of valuable resources is available on the web in English, which are often translated into local languages to facilitate knowledge sharing among local people who are not much familiar with English. However, translating such content manually is very tedious, costly, and time-consuming process. To this end, machine translation is an e cient approach to translate text without any human involvement. Neural machine translation (NMT) is one of the most recent and e ective translation technique amongst all existing machine translation systems. In this paper, we apply NMT for English-Tamil and Hindi-Tamil language pair. We propose a novel neural machine translation technique using pre-trained subword embeddings (BPEmb) to develop an e cient translation system that overcomes the OOV (Out Of Vocabulary) problem for languages which do not have much translations available online. We use the BLEU score for evaluating the system performance. We got the 26.97% Precision,37.98% Recall for English to Tamil and 25.18% Precision and 27.24% Recall for Tamil to Hindi respectively.
Big countries such as India and China have several languages which change by
regions. For instance, India has 23 constitutionally recognized o cial languages
(e.g., Hindi, Tamil, and Panjabi) and several hundreds uno cial local languages.
Despite Indian population is approximately 1.3 billion, only approximately 10%
of them English speak English. Some studies say that out of these 10% English
speakers only 2% can speak, write, and read English well, and rest 8% can merely
understand simple English and speak broken English with an amazing variety
of accents [
There are many challenges in machine translation for Indian languages. For instance, (i) the size of parallel corpora and (ii) di erences amongst languages, mainly the morphological richness and word order di erences due to syntactical divergence are two of the major challenges. Indian languages (IL) su er both of these problems, especially when they are being translated from English. There are only a few parallel corpora for English and Indian languages. Moreover, Indian languages such as Tamil di er from English in word order as well as in morphological complexity. For instance, English has Subject-Verb-Object (SVO) whereas Tamil has Subject-Object-Verb (SOV). Moreover, English is a fusional whereas Tamil is agglutinative languages. While syntactic di erences contribute to di culties of translation models, morphological di erences contribute to data sparsity. We attempt to address both issues in this paper. Since both the languages (Tamil and Hindi) are known for its morphologically richness, that makes the task much more di cult to work.
Though much work is being done on machine translation for foreign and
Indian languages but apart from foreign languages most of works on Indian
languages are limited to conventional machine translation techniques. We observe
that the techniques like word-embedding along with Byte-pair-encoding (BPE)
are not applied on many Indian languages which have shown a great
improvement in natural language processing. Thus, in this paper, we apply a neural
machine translation technique (torch implementation) with pre-trained subword
embeddings [
Main contributions of our work are as follows: { This is the rst work to apply BPE along with pre trained word embedding on Indian language pair English-Tamil and Hindi-Tamil with NMT technique. { We achieve comparable accuracy with a simpler model in less training time rather then training on deep and complex neural network which requires much time to train. 2
In this study, we present a neural machine translation technique using BytePair-Encoding along with pre trained word embedding to develop an e cient translation system, called MIDAS translator that overcomes the OOV (Out Of Vocabulary) problem for languages which do not have much translations available online. Thus, rst, we provide an overview of neural machine translation, attention model, word embedding, and Byte Pair Encoding. Next, we present the framework of our MIDAS translator. 2.1
Neural Machine translation is a technique that is based on neural networks and
the conditional probability of translated sentence from the source language to
target sentences [
Sequence to Sequence Architecture Sequence to sequence architecture is basically used for response generation whereas in machine translation models it is used to nd the relationship between two di erent language pairs. It consists of two parts, an encoder and a decoder. The encoder takes the input from source and the decoder generates the output based on encoding vector and previously generated words. Assume A be the source sentence and B be a target sentence. The encoder converts the source sentence a1; a2; a3:::; an into vector of xed dimensions and the decoder outputs word by word using conditional probability. Here, A1; A2; :::; AM in the equation are the xed size encoded vectors. Using chain rule, the Eq. 1 is converted to the Eq. 2.
P (B=A) = P (BjA1; A2; A3; :::; AM )
P (BjA) = P (bijb0; b1; b2; :::; bi 1; a1; a2; a3; :::; am)
While decoding, next word is predicted using previously predicted word vectors
and source sentence vectors in Eq. 1. Each term in the distribution is represented
with a softmax over all the words in the vocabulary.
(1)
(2)
Attention Model In a basic encoder-decoder architecture, encoder reads the
whole sentence, memorizes it and store it in the nal activation layer, then the
decoder network generates the target translation. This architecture works quite
well for short sentences, so we might achieve a relatively high BLEU score,
but for very long sentences, maybe longer than 30 or 40 words, the performance
degrades. Using attention1 mechanism with a basic encoder-decoder architecture
is a solution for that. It translates similar to humans by looking at part of the
sentence at a time. The mechanism decides how much attention should be paid
to a particular word while translating the sentence. The mechanism is shown
in Fig. 2. The Encoder generates the attention vectors h1; h2; h3::::::ht from the
inputs A1; A2; A3At. Then, context vector Ci is calculated using concatenation
of these vector for each output time step. Then Using the context vector Ci
hidden state Si and previously predicted words, decoder generates the softmax
output Bi.
Word Embedding Word embedding is a way of representing words on a vector
space where the words having same meaning have similar vector representations.
Each word from vocabulary is represented in hundreds of dimensions. Normally
pre-trained word embeddings are used and with the help of transfer learning
words from vocabulary are converted to vector [
Byte Pair Encoding BPE [
BPEmb Subword embeddings is collection of both Byte-Pair-encodding and Word embeddings.In our model we used pre-trained subword embeddings which was trained on Wikipedia. Subwords allow guessing the meaning of unknown / out-of-vocabulary words and Byte-Pair Encoding gives a subword segmentation that is often good enough, without requiring tokenization or morphological analysis. We used BPEmb with 2000 merge operations and 50 dimension vectors and then apply it on train test and validation data for both source and target. BPE helped in compound splitting and su x, pre x separation which is used for creating new words of English Tamil and Hindi language. 2.2
We tried various models to get a better intuition on how parameter tuning along with di erent techniques a ects on Indian language pair. Our rst model architecture consists of 2 layer Bi-directional LSTM encoder and 2 layers LSTM decoder of 50 dimensions for both source and target. First we tried SGD optimization method, Luong attention with a dropout (regularization) of 0.3, and learning rate 1.0. Secondly, we changed the optimization method to Adam and attention to Bahdanau with the learning rate of 0.001. We got our best results with a BPE vocabulary size of 2000 with 2 Layer Bi-directional encoderdecoder, Adam optimization with a learning rate of 0.001, Bahdanau attention, and BPEmb with the dimension of 50. We used GPU (Nvidia GeForce GTX 1080) for the training of di erent models which increase the computation speed. 3 3.1
The BLEU score or bilingual evaluation under study is a method to measure the di erence between machine and human translations 8. The approach works by counting and matching n-grams in result translation to n-grams in the reference text, where unigram would be each token and a bigram comparison would be each word pair and so on. The comparison is made regardless of word order. This method is a modi cation of a simple precision method. 2 https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md
In pattern recognition, information retrieval and binary classi cation, precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances, while recall (also known as sensitivity) is the fraction of relevant instances that have been retrieved over the total amount of relevant instances. Both precision and recall are therefore based on an understanding and measure of relevance. 3.3
We used the datasets obtained from VPT-IL. After preprocessing and splitting it to train and validation, our nal dataset contains 1223 English-Tamil training corpus and 200 validation corpus. For Hindi to tamil we used 1772 training corpus and 200 validation corpus as provided by the VPT-IL organization. The data used is encoded in UTF-8 format. 3.4
We got the following results after training on the 1223 English-Tamil training corpus with 200 validation corpus which were splitted from 1443 corpus and 1772 Hindi to tamil training corpus and 200 validation corpus which were splitted from 1992 corpus as provided by the VPT-IL Organization.
Model % Precision % Recall English-Tamil 26.97 37.98 Tamil-Hindi 25.18 27.24
Table 1. Test Score for Our Model 4
The similar work is given in the paper 9. In this paper the Phrase-based and Hierarchical models trained after morphological preprocessing using neural machine translation. In paper 10 model is trained after su x separation and Compound Splitting. Di erent models were also tried for the same task and achieved a good result on their respective dataset as given in 11. We analyze that morphological preprocessing, su x separation and compound splitting can be overpass by using Byte-Pair-Encoding and produce similar or even better translation without making the model complex.
In this paper, we applied NMT to two most di cult language pairs EnglishTamil and Tamil-Hindi. Since amount of data highly a ects the neural network models, hence we got the comparable results. We believe that our model will perform much better on larger datasets and pre-trained subword embeddings will performs better than complex translation techniques on Indian languages. We can explore the possibility of using above techniques for various English Indian language translation. In future, we would also like to employ machine translation in detecting o ensive languages from code-switched languages too 12.