Native Language Identi cation (NLI) is the task which involves identi cation of native language (L1) of an individual based on his/her language production in a learned language (L2). It is basically a classi cation task where we are classifying L1 into a number of di erent languages. In this task I have to identify an individual's native language (L1) among the following six Indian languages: Bengali, Hindi, Kannada, Malayalam, Tamil, and Telugu using their Facebook comments written in English language (L2). In this paper I propose to use machine learning models such as classi cation models together with N-grams as features and T df as vectorizer.
Native Language Identi cation (NLI) is the task of classifying the native language (L1) of an individual into given di erent languages based on his/her writing in another language (L2). NLI tasks involves identifying language use patterns that are common to certain groups of speakers that share the same native language. The native language of an individual in uences the usage of words as well the errors that a person makes when writing in another language. The task is usually considered as a classi cation problem where a machine learning algorithm is trained in a supervised manner which can then be used for predicting the native language of user text.
Predicting the native language of a writer has applications in di erent elds. It can be used for authorship identi cation, forensic analysis, tracing linguistic in uence in potentially multi-author texts and naturally to support Second Language Acquisition research. In the eld of cyber security, NLI can be used to determine the native language of an author of a suspicious or threatening text. In the eld of academics NLI can be used for educational applications such as developing grammatical error correction systems which can personalize their feedback and model performance to the native language of the user.
In my research here I have used classi cation models such as Logistic Regression, Linear SVC, Naive Bayes to name a few, together with features such as N-Grams both at character level and word level to nd the best working model which can e ciently classify L1 of a person to di erent sets of languages.
NLI research has mostly been focused on texts where both lexical and
syntactic features were used. Models formed try to extract patterns that speakers with
di erent native language will have in terms of di erent misspellings,
mispronunciations or usage frequency of particular words. Also some languages have speci c
linguistic styles, like Japanese is much more formal in nature while French and
Spanish are way more romantic in nature because of their gentler vocabulary
whereas Russian and German can be classi ed as harsh because of their string
vocabulary. Kumar et al.[
Dataset provided by task organizers contains information collected from English speakers of six di erent native Indian languages namely Tamil, Telugu, Kannada, Malayalam, Bengali and Hindi. Data was collected through social networking site Facebook. The distribution of class and training instances can be seen in Table 1.
Language Training Instances
I have tried to model this task as a classi cation task. I have divided the given dataset into three parts Training set (75%) , Testing set(12.5%) and Validation set(12.5%). Validation set was used to determine several hyper parameter values. For determining whether to use stop words or not I used Count Vectorizer as a sample vectorizer to calculate token counts, Logistic Regression as a sample classi cation model. Two classi cation models were trained one " with stop words" other " without stop words". Both were trained using training set data while accuracy was calculated on validation set. I used Python's NLTK stop words as sample. The results from both models were plotted (see Fig1.a). Validation Set Accuracy being represented on Y-axis and Number of features(maximum no of words in vocabulary) on X-axis.
Accuracy for category "without stop words" was found to be higher when number of features considered are large.Similar to this Validation Set was also used to determine which N-gram are giving the best results.Unigrams, Bigrams, Trigrams were under consideration. So I trained three sample Logistic regression model one for every N-gram using training set data. Results from all three models were plotted(see Fig1.b).Validation Set Accuracy being represented on Y-axis and Number of features (maximum no of words in vocabulary) on X-axis.
Accuracy for Unigrams were found to be highest when large number of features are considered. Hence it is Unigrams which are able to best capture inherent features of a language. Lastly I also determined which vectorizer to use. In my research I considered TfIdf Vectorizer and Count Vectorizer as two di erent candidates. Count Vectorizer basically converts a collection of text documents to a matrix of token counts whereas TfIdf Vectorizer converts a collection of raw documents to a matrix of T df features in simpler terms it is basically product of term frequency(no of times a particular word appears in a single document) and inverse document frequency(log of number of docs in your corpus divided by the number of docs in which this term appears). In total six sample Logis(a) StopWords Or Not (b) N-grams tic regression models were trained using training set data while accuracy was calculated using validation set.
Accuracy for T df vectorizer for Unigrams were found to be highest. Now after determining the hyper parameter values I considered several classi cation models. Each model was trained on training set while accuracy was calculated using Validation set and compared with null accuracy (accuracy calculated from model which always predicts the label which appeared maximum number of times in training set). Table 3 summarizes the results.
Based on the accuracy on Validation Set Passive-Aggressive classi er showed the best results. Accuracy on Test dataset for Passive-Aggressive classi er was found to be 82.89%. 5
Test Dataset was released on much later date and our nal model was tested on it. Test Dataset provided by organizers consisted Facebook comments for various Indian Languages.
On test set1 accuracy achieved was 41.4% while accuracy on test set2 which was released on a later date accuracy achieved was 31.9%. Accuracy is much lower than that achieved during training due to lack of large training dataset due to which model was not able to extract useful patterns of di erent languages e ciently.
I would like to express my special thanks to Soumil Mandal for his guidance and constant supervision as well as providing necessary information. Also I would like to extend my gratitude towards Team WebArch for providing this opportunity.