Native Language Identification has played an important role in forensics primarily for author profiling and identification. In this work, we discuss our approach to the shared task of Indian Language Identification. The task is primarily to identify the native language of the writer from the given XML file which contains a set of Facebook comments in the English language. We propose a hierarchical ensemble approach which combines various machine learning techniques along with language agnostic feature extraction to perform the final classification. Our hierarchical ensemble improves the TF-IDF based baseline accuracy by 3.9%. The proposed system stood 3rd across unique team submissions..
• Computing methodologies → Classification and regression trees; Support vector machines; Neural networks; Bagging; Feature selection;
Native Language Identification (NLI) is primarily the task of automatically identifying the native language of an individual based on their writing or speech in another language. The underlying assumption here is that an author’s native language (mother tongue) will often have an influence on the way they express themselves in another language. Identifying such common patterns across a group of people can be used to determine their native language.
Identifying the native language of an author has various
applications, primarily in forensics. In forensics, author profiling and
identification using their native language is an important feature
[
Researchers have experimented with a range of machine learning algorithms, with Support Vector Machines having found the most success. However, some of the most successful approaches have made use of classifier ensemble methods to further improve performance on this task.
In this shared task [
As we explore in the next section, prior work has primarily dealt with statistical machine learning algorithms including SVMs and representation methods such as tf-idf. Our approach combines these various state of the art algorithms using a hierarchical ensemble. We’ve also experimented with two diferent types of feature extraction strategies. They are explored further in Section 3.1 2
Most of the related NLI work can be categorized into 2 domains: text based and speech based. 2.1
The 2013 Native Language Identification Shared Task [
[
Starting from the shared task in 2013, quite a few approaches used ensembling techniques to combine multiple base classifiers to improve the performance. 3 3.1
We observe from the dataset that people often use words and phrases which belong to their native language transliterated into English. Some common examples are "Jai ho", "vadi koduthu" etc. We also expect that people who have the same native language would have some topics/concerns which would not be shared by people who have a diferent native language. For example an issue which revolves around Tamil Nadu would resonate more with Tamil speaking people as compared to others.
For our classification system we created two diferent feature sets from our data. In the first feature set we take raw sentences as inputs. The sentences are tokenized to create vocabulary of tokens. This vocabulary is then used to create term frequencyinverse document frequency features for each sample point which are then used as feature input in the classification step. One benefit of term frequency inverse document frequency over simple bag of words approach is it mitigates the efect of common words and thus making inputs easier to discriminate. We refrained from using higher n-grams features due to limited amount of data.
In the second feature set we leverage our observations stated above to filter the relevant information. First, for each sentence we collect words which do not belong to the English vocabulary. The sentences were tokenized using tweetokenize package1 and we check whether the word belongs to English vocabulary by using the English dictionary provided in enchant 2. These words are extracted for capturing usage of native language in inputs. We then tested our hypothesis that speakers of common native language would have topics/concerns which are not shared as strongly by others. To this end we collected all the documents of native speakers of each language and extracted topics from it using Latent Dirichlet Allocation. We observed a good deal of topics which were specific to speakers of common native language. We think this is a result of regional and cultural proximity between speakers of common native 1https://www.github.com/jaredks/ tweetokenize 2https://pypi.python.org/pypi/pyenchant/ language. Most of these topics were expressed in the noun forms and hence to extract this information we collected noun chunks which are present in the sentences. Noun chunks are extracted using spacy3. Now we follow procedure similar to first feature set. We collect these two features for each sentence and then create a vocabulary for it. This vocabulary is then used to create term frequency inverse document frequency features which are then used as inputs for classification. 3.2
We perform the classification separately for both feature sets
described above. The training data set in the competition was small
hence, instead of creating separate train and development set, we
performed 10-fold cross validation. On each fold, a model was
trained and the predictions were collected on the remaining dataset.
We calculated mean of accuracy over 10 fold for each type of
classiifer. We also observed the performance of each classifier on points
which were harder to classify i.e those points for which the
decisions were incorrect for majority of classifiers. After evaluation
selected four classifiers, namely LogisticRegression, MLPClassifier,
LinearSVC and RidgeClassifier of sklearn [
We created a hierarchical ensemble model for this task, consisting of two layers of ensembles. First layer consists of two ensemble. First one consists of four classifiers selected in the previous section mentioned. These classifiers were trained on feature set 1 ( term frequency inverse document frequency features on raw input sentences). Second ensemble also consists of same four classifiers but were trained on feature set 2, which had term frequency inverse document frequency features computed using noun chunk and non English words extracted from each sentence. Each ensemble predicts the output using the majority vote. We limited the decision to majority vote as complex weighted voting would have caused overiftting. Final classification is predicted using a combination of two ensembles described above. If they output same class, we present that class as prediction. If they difer, we calculate the confidence of each ensemble using count of classifiers in the ensemble which support its decision. Fig 1. depicts our system.
We can see from table 1. that all four classifiers perform quite well on both the extracted feature sets especially considering the classification problem involves six classes. This suggest that the dataset points are easier to discriminate. We further see that the accuracy increases significantly on feature set 2, suggesting that features such as native language words and regional/local topics are important for identification of native language.
We presented three submissions. Submission 1 is the output of ifnal classifier(see Fig 1). Submission 2 is the output of Ensemble 1, which was trained on raw sentences. Submission 3 was generated using ensemble 2 trained on feature set 2 (non-English phrase and noun chunks). We can see that Submission 3 outperform other two classifiers strengthening our belief on importance of native language phrases and shared topics in identifying native language of speaker.
This paper studies couple of approaches for identification of native language. First approach measures the power of tf-idf features for the purpose of classification. Second approach identifies certain features which separate diferent native language speakers from each other and utilizes those for better ac curacies of overall system. We have seen improvement in accuracy due to identification of discriminating features, however extending this procedure is time consuming and requires language expertise. Recent studies have shown use of deep neural networks can be a possible alternate to
Class BE HI KA MA TA TE OverAll creating manually hand-crafted features and can provide better performance.
We would like to thank the organisers of the FIRE-2017 Shared Task on Native language identification, for providing the data, the guidelines and timely support.