Native Language Identification (NLI) aims at identifying the native language (L1) of users writing in another or later learned language or speech (L2). NLI is an important task that has many applications in different areas such as social-media analysis, authorship identification, second language acquisition and forensic investigation. In forensic analysis [7], NLI helps to glean information about the discriminant L1 cues in an anonymous text. Second Language Acquisition (SLA) [12] studies the transfer effects from the native languages on later learned language. In education, automatic correction of grammatical errors is an important application of NLI [14]. NLI can be used as a feature in authorship identification task [6], which aims at assigning a text to one of the predefined list of authors. Authorship identification is used for terrorists communications investigation [1] and digital crime investigation [4].
Supervised approaches using machine learning algorithms have been used for NLI by many researchers. Jarvis et al. [9], used SVM classification algorithm to create a model for NLI and reported an accuracy of 83.6%. They used features such as n-grams of words, Part-of-Speech (PoS) tags and lemmas. Combining multiple classifier systems to enhance the final output, such as ensemble classifier was used for NLI by Tetreault et al. [15]. Bykh and Meurers [3] applied a tuned and optimized ensemble classifier on NLI 2013 shared task dataset and achieved an accuracy of 84.82%.
Given a comment I =<w 1 ,w 2 , . . . ,w N > where each w i , i = 1..n is either an English language word or a word of native language written in English (or transliterated to English language) for an individual social media user, the objective of the task is to identify the native language of the user. The comment may include English words in addition to the words of any one native language written in English. The task considers six Indian languages, namely Tamil (TA), Hindi (HI), Kannada (KA), Malayalam (MA), Bengali (BE) and Telugu (TE). Considering the languages as a set of classes C = {T A,HI ,KA,MA,BE,T E} and comments as individual instances I = {I 1 , I 2 , . . . , I n } we have formulated the task as a classification problem that assigns one of the six predefined classes of C to a new unlabelled instance I u .
The data sets provided for this task are a collection of comments from different regional newspaper's facebook pages during April-2017 to July-2017. Training and test sets contain 1233 and 783 files respectively. Each training and testing file consists of a set of comments. Table 1 shows a brief statistics about training set.
In this section, we will describe the two systems proposed for Indian Native Language Identification (INLI) [10] task submissions. The general frame work of classifier for both systems is shown in figure 1. First phase of our systems is data preprocessing, also known as corpus cleaning. This phase is important where we exclude non-informative tokens and phrases. Second phase comprises of constructing vector space model for the comments (input data). These two phases are common for both the systems. The next phase is creating a model using a machine learning algorithm. Support Vector Machine (SVM) and Ensemble learning are used for the first and second submission respectively. Details of each phase is given below. In this phase, we tokenized each comment I j into a set of words or tokens and removed uninformative tokens as follows to get bag of tokens:-
Emoji is a small image used as a visual presentation to express emotion. The first step in removing unrelated information is to remove Emojis as they are not important for the identification of native language.
Digits and special characters such as #, %, ... are the characters which appear frequently in the comments of all the languages. As such characters do not contribute to the identification of native language they are removed.
Stop words are the words which appear frequently and do not contribute to the identification of native language. Hence, to remove stop words we used a union of different stop words lists, namely, (1) stop words list extracted from nltk.corpus 1 package.
(2) stop words list extracted from stop_words 2 package.
(3) Manually written stop words. (The complete list of manually written stop words is given in Appendix A) 1 www.nltk.org/nltk_data/ 2 pypi.python.org/pypi/stop-words
After preprocessing, the comments will be represented as vector space model. If <t 1 ,t 2 , . . . ,t k > are the unique tokens/terms in a comment I j , the vector space model for the comment I j will be represented as <w j1 ,w j2 , . . . ,w jk > where w ji is the weight of the token/term t i in comment I j . For term weights, we used Term Frequency/Inverse Document Frequency (TF/IDF) calculated as follows:-
where t f j is the total number of occurrences of term t j in the current comment, d f j is the number of comments in which the token/term t j occurs and N is the total number of comments.
SVM is a binary classifier which creates a hyperplane that discriminates between the two classes [5]. SVM can be extended to multi-class problems by creating several binary SVMs and combining them using a one-vs-rest method or one-vs-one method [8].
We implemented a six class SVM corresponding to six classes TA, HI, KA, MA, BE and TE, as per the framework shown in figure 1 for comment identification using Stochastic Gradient Descent (SGD) for optimizing the parameters of SVM model. SGD algorithm updates the value of parameter θ of the objective function W (θ ) as
where η is step size and E[W (θ )] is the cost function. Ensemble learning is a classification technique, which uses a set of different heterogenous and diverse classifiers as base classifiers and combines the output of them in different approaches to get the final output [13]. Ensemble technique tries to overcome the weakness of some classifiers using the strength of other classifiers. Figure 2 shows the framework of ensemble learning.
We have used 3 base classifiers, namely, multinomial Bayes, SVM and random forest tree classifiers and combined the results by weighted voting. Multinomial Bayes classifier is an instance of Naive Bayes classifier that captures word frequency information in documents [11]. Random forests classifier is a supervised classifier which comprises of multiple decision trees and each tree depends on independently sampled random vector [2]. The base classifiers are designed as per the framework shown in figure 1.
Performance evaluation of INLI task is measured as the accuracy of the system in addition to class-wise accuracy which is calculated using Precision (P), Recall (R) and F1 measure 3 . For each class, P is the measure of the number of comments correctly classified over the total number of comments that system classified as same class. R is the measure of the number of comments correctly classified over the actual number of comments of the class. F1 measure is the harmonic mean of P and R, which can be calculate as follow:-
The class wise accuracy of first submission using SVM based on SGD algorithm to determine the parameters of the model is shown in Table 2 in terms of P, R and F1 measure. The overall accuracy of this submission is 47.60% and it ranks second among all the submissions. Table 3 shows the performance evaluation of the second submission where we used Ensemble approach to combine output of different models. Overall accuracy of this submission is 47.30% and it ranks third among all the submissions. Results of both submissions illustrates that the performance of identifying Hindi is the worst. The reason may be most of the other languages' natives have knowledge of Hindi. Our systems depend essentially on the effective words for each language.
In this work, SVM and Ensemble classifier have been used for INLI. SVM outperforms the Ensemble classifier which combines different three classifiers. Our Support Vector Machine (SVM) submission secured second rank respectively over all submissions for the task.
The following is the full list of stopwords used in our system:-{ a, about, above, across, after, afterwards, again, against, all, almost, alone, along, already, also, although, always, am, among, amongst, amoungst, amount, an, and, another, any, anyhow, anyone, anything, anyway, anywhere, are, around, as, at, back, be, side, since, sincere, six, sixty, so, some, somehow, someone, something, sometime, sometimes, somewhere, still, such, system, take, ten, than, that, the, their, them, themselves, then, thence, there, thereafter, thereby, therefore, therein, thereupon, these, they, thick, thin, third, this, those, though, three, through, throughout, thru, thus, to, together, too, top, toward, towards, twelve, twenty, two, un, under, until, up, upon, us, very, via, was, we, well, were, what, whatever, when, whence, whenever, where, whereafter, whereas, whereby, wherein, whereupon, wherever, whether, which, while, whither, who, whoever, whole, whom, whose, why, will, with, within, without, would, yet, you, your, yours, yourself, yourselves }