Native language identi cation (NLI) focuses on determining the native language of the author based on the writing style in English. Indian native language identi cation is a challenging task based on users comments and posts on social media. To solve this problem, we present a hybrid gated LSTM-CNN model to solve this problem. The nal vector of a sentence is generated at hybrid gate by joining the two distinct vector of a sentence. Gate seeks the optimum mixture of the LSTM and CNN level outputs. The input word for LSTM and CNN are projected into high-dimensional space by embedding technique. We obtained 88.50% accuracy during training on the provided social media dataset, and 17.10% is reported in the nal testing done by Indian native language identi cation (INLI) workshop organizers.
CNN Glove.
It was publicly used for prediction of native language of learner based on his/her writing style. It was released in 2002 and updated in 2009.
In the following sections; we mentions related work in Section 2, we describe the proposed model and training procedure in Section 3, we show the result and analysis in Section 4 and nally, draw conclusion in Section 6. 2
Native language identi cation is a new and signi cant problem. Language
learners are prone to make mistakes similarly if machines can learn the same
tendencies of making mistakes then it may help in developing systems for educational
domain. Several researchers worked on this problem and similar problem like
second language acquisition. One of the earliest work, Tomokiyo and Jones (2001)
tried to discriminate non-native statements from native statements, written in
English by Nave Bayes [
Kochmar et al. (2011) has performed experiments on prediction of the native
languages of Indo-European learners. He treated this problem as binary classi
cation and use linear kernel SVM. The features used for prediction were n-grams
and words. Also, The errors were tagged manually within the corpus [
The model architecture is showed in gure 1.
Each document includes m sentences and each sentence within a document consists of n words. The word level input wi n projected into high-dimensional vector space with the help of pretrained glove English word embedding, wi n2<k, where k is word vector's dimension, i and n represents sentence and word respectively. The word level input is converted into sentence level input by using a bidirectional LSTM. This is achieved by linearly combining the last hidden state of forward and backward LSTM.
xswtord = Wsf hsft + Wsrhrst + bs
Where, st 2 S, S is the total sentences in a document and hsft; hrst 2 <k are
the forward and backward LSTM's end hidden states, respectively. wf ; wr
s s 2
<k k and bs 2 <k are trainable parameters and xswtord 2 <k is the nal vector
representation of the sentence st. Similarly, with the help of bidirectional LSTM,
we will retrieve the single vector representation of the document. It will take
vector representation of the sentences [
xdwtord = Wdf hfdt + Wdrhrdt + bd
Where dt2D, D is the number of documents which is composed of the
combination of the last hidden state of the forward and backward LSTM, represented by
hfdt; hrdt 2 <k respectively. wdf ; wdr 2 <k k and bd 2 <k are trainable parameters
and xdwtord 2 <k is the nal vector representation of the sentence dt.
Convolutional neural network [
A convolution operation involves a lter and the size of this lter size shows h-gram of words. The lter moves over the entire document fw1:h; w2:h+1; :::: ; wn m h+1:n mg and generate features fc1; c2; ::::; cn m h+1g. These generated features produce a feature map c.
c = [c1; c2; ::::; cn m h+1] (4) Now we want to extract most important feature from feature map, so we apply max-over-pooling on this feature map that gives maximum value ^c= maxfcg as the feature crossponding to this particular lter. This process extracts one feature from one lter. This model uses multiple lters with di erent h-gram to extract multiple features. These di erent extracted features concatenated and atten before passing in the dense layer. We have D documents, for each document the entire process is repeated. The nal vector representation of the sentence dt is xcdhtar2<k where dt2D.
The assumption is that, bidirectional LSTM can capture the entire document features regarding language model into xdwtord and n-gram features for convolutional neural network into xcdhtar. If both combine, then it may help to distinguish the Indian languages. The gated word-char layer combines these two types of vector generation from bidirectional LSTM and convolutional neural network. The process of mixing generated vectors xdwtord and xcdhtar is done by gate gdt, where gdt is also a dense layer with sigmoid activation function.
gdt = (vgT xdwtord + bg)
xdt = (1
gdt)xdwtord + gdtxdt
char
Where vg2<d and bg2< are, weight vector and bias respectively which are
trainable parameters. Miyamoto and Cho (2016) applied this approach at word
level on language model to handle out-of-vocabulary and unusual words [
p = sof tmax(vsT xdt + bs) Where vs and bs are, weight vector and bias respectively which are trainable parameters. (5) (6) (7) 4 4.1
We have evaluated our model on the dataset provided by the task organizers. The dataset contains English comments of Facebook users. English comments are written in native language by users whose native language includes Bengali (BE), Hindi (HI), Kannada (KA), Malayalam (MA), Tamil (TA) and Telugu (TE). The statistics of the train dataset are summarized in Table 1. We divided the training dataset into training and validation data in the ratio of 90:10.
The given dataset is preprocessed to remove multiple occurrences of punctuation symbols which occurred in between the sentences or at the end of the sentences. Periods, question marks and exclamation marks are some of the punctuation symbols that occurred in the available dataset. Punctuation symbols present at the end of the sentence de nes the sentence boundary hence such kinds of repeated symbols are replaced by single symbol. There were some occurrences of emojis which are also removed. After preprocessing the obtained list of words is lemmatized4 to obtain the root word using the grammatical rules of the concerned language. There is variation in the occurrence of number of sentences and number of words in a sentence for di erent language document. To overcome this issue, we perform padding.
Each word is represented in 100 dimensions vector using pre-trained glove embedding. Glove embedding 5 is trained on 6 billion tokens of the combination of Gigaword5 + Wikipedia2014 dumps. Some words are not present in glove embeddings, which are represented by the glove vector dimension of random uniform distribution range from -1 to +1. We have set 20 as maximum document length and 300 as maximum sentence length. Number of hidden units is xed as 100 for each directional layer at sentence level and document level. The convolutional neural network uses three convolutional layer whose lter size is 2, 3 and 5 respectively however stride size is xed as 1 for all convolutional layer. A dense layer of the convolutional neural network has 100 hidden units that represents a document. We have 6 prediction classes (HI, BE, KA, MA, TA, TE), so number of hidden units in nal prediction layer will be based on number of prediction classes. 4 https : ==www:nltk:org=modules=nltk=stem=wordnet:html 5 https://nlp.stanford.edu/projects/glove/
The entire network is trained by Adam optimizer with epoch and mini-batch size of 15 and 10 respectively. Model is implemented on GeForce 840 GPU using keras 6 neural network library. 5
The task organizers evaluated the model on di erent metrics such as precision, recall and F-score for each language with overall accuracy. Two di erent test sets are provided for evaluating the model. The number of document in test set1 and test set2 are 783 and 1185 respectively. The obtained result is given in Table 2. We obtain the overall accuracy of 15.3% for test set1 and 17.1% for test set2 Indian native language identi cation. Our model retrieved more relevant documents for Tamil language as compared to other languages during testing phase. For Hindi and Tamil language, our model achieves highest F1-score for Testset1 data.
We obtained accuracy of 88.5% on training dataset however there is signi cant drop in the result during the testing phase which indicates over- tting as a probable reason. There is signi cant di erence in the number of documents shared for the training and testing phase. Considering the size of trainable data used during the training phase, our model was unable to predict more generalized features due to this large number of random word embeddings were formed which inturn re ects poor results during the testing phase. 6 https://keras.io/ The purposed hybrid gated LSTM-CNN model for identifying the Indian native languages, namely Bengali, Hindi, Kannada, Malayalam, Tamil and Telugu, posted on social media in English by the native speaker of those languages. Each document is represented in a vector by combining the output obtained using bidirectional-LSTM and convolutional neural network. We obtained better accuracy in test set2 data as compared to the accuracy obtained in the test set1 data. The proposed system can be improved by applying hybrid gated LSTMCNN model at word level instead of document level with dropouts.