Corplab INLI@FIRE-2018: Identi cation of Indian Native Language using Pairwise Coupling

Abstract

We are going to describe the techniques and methodology used during the implementation of Indian Native Language Identi cation (INLI) that was organized at FIRE 2018. Native Language Identication (NLI) is a process of identifying the native language of a writer by analyzing their text written in another language, which is in this case English. The following six di erent Indian languages were considered in this task: Bengali, Hindi, Kannada, Malayalam, Tamil, and Telugu. We have used state of the art TF-IDF feature vectors with linear-SVM as a classi er in our analysis. The classi er has been used with three di erent strategies (i.e. One-vs-the-rest and Pairwise Coupling strategies as described in [7]) and achieved an accuracy of 42.1% for test TestSet-1 and 31.8% for test TestSet-2.

Keywords

• Native Language Identi cation Linear SVM TF-IDF Feature One Vs Rest Classi er Pairwise Coupling

-

Study on the in uence of native language while using a second language has been studied since 1950s in the linguistic literature domain. This has motivated research in NLI that aims to automatically identify the native language of a user by analyzing the text written in another language. NLI works on the assumption that the user's linguistic background will lead them to use native language (mother tongue) phrases/styles more frequently in their learned languages.

NLI is useful in di erent areas such as cyber-forensic, authorship identi cation, analysis on social media, and second language acquisition. NLI helps to identify the author's native language by analyzing text from web-blogs to monitor terrorist communications and for digital crime investigation [ 1, 9 ].

NLI is a nontrivial pattern recognition challenge especially with a small corpus for training which has several useful applications. Therefore it is gaining popularity and competitions are being organized in various conferences/events in recent years [ 11, 5 ]. Most commonly used features for NLI tasks are character n-grams, misspellings, mispronunciations, frequency patterns of particular words, POS n-grams, content words, function words, Term Frequency-Inverse Document Frequency (TF-IDF), Continuous Bag of Words (CBOW), etc. [ 4, 6 ], whereas irrespective of the feature vector used, the most dominant choice of classi er is Support Vector Machine (SVM) [ 10 ]. The corpus of INLI-2018 contains English comments/opinions of anonymous users that featured in regional newspapers, which was taken from Facebook. Users understood one of the following six native languages: Malayalam (MA), Bengali (BE), Kannada (KA), Telugu (TE), Hindi (HI), and Tamil (TA). The underlying assumption of this process is that only native speakers will read the native language newspapers. The distribution of the training instances with respect to the classes can be seen in Table 1. There are two test sets provided by the organizers (TestSet-1 and TestSet-2). TestSet-1 is the same test set that was used in INLI-2017, and TestSet-2 is a new test set given at INLI-2018 [ 3 ]. In total TestSet-1 and TestSet-2 has 783 and 1185 instances respectively. Detailed description about the training set and the TestSet-1 can be found in [ 2 ]. We followed a multi-class classi cation approach as described in [ 7 ] for this task. The detailed description of our approach has been given below.

Data preprocessing

The given corpus collected from social media includes a signi cant number of non-ASCII characters like emojis. In general, user-generated social media texts are very noisy and contains irrelevant textual information for the classi cation process. Therefore, we remove such non-ASCII characters before feature extraction process. We have also replaced multiple occurrences of some characters like "......" or "sorryyyyyyy" with "." or "sorry". 2.2

Feature extraction

TF-IDF is a well-known weighting algorithm that measures the importance of a word in a document, given a collection of documents. The rationale behind this algorithm is that, if a word appears in a document very frequently, then it should be a signi cant word for that document and should be given a higher weight. However if that word appears in many other documents, it is not a unique identi er and therefore it should be given lower weight. We have used this feature in our approach.

TF-IDF is a product of two di erent measures, Term Frequency (TF) and Inverse Document Frequency (IDF): tf idf (t; d; D) = tf (t; d) idf (t; D), where t denotes the words/terms; d denotes a given document; D = d1; d2; : : : dn denotes the collection of the documents. The rst part of this formula tf (t; d) equates the number of times each word appeared in each document and it excludes stop words such as "a", "the" etc. . The second part of the formula is jDj idf (t; D) = log 1+jfd2D:t2dgj . Note that frequency of a term in the document does not a ect IDF, and 1 is added to the denominator to avoid division by zero. 2.3

Classi er models and prediction

In the rst run, we have used one-vs-rest classi er settings. In the second run we have randomly arranged the set of classes into pairs and for each pair (i.e. i and j) based on the classi er's probability, we determine if the data ts better into class i or class j, and then proceed to the next round of the scheme [ 7, 8 ]. In the third run, for each class i, we randomly choose k other classes and determine the mean score for class i when comparing the test data in k pairwise comparisons with the randomly chosen classes. The class having the highest total score is selected as the identi ed class [ 7, 8 ]. We would like to mention that in all of the above approaches we have used linear-SVM with l2 regularization as a choice of the classi er. 3

Result

We report our achieved results using three evaluation metrics and it's overall T P T P accuracy. The three measures are: P recision = T P +F P , Recall = T P +F N and F measure = 2 PP rreecciissiioonn+RReeccaallll where TP is True Positive, FP is False

Run

Class BE HI

KA Run-1 MA

TA TE Overall Accuracy BE HI

KA Run-2 MA

TA TE Overall Accuracy BE HI

KA Run-3 MA

TA TE Overall Accuracy Positive, and FN is false negative predicted values. Table 2 shows the achieved results obtained during di erent runs of the two test sets. We can clearly observe very poor performance for HI and good performance for BE irrespective of the test set. We also observe that the overall performance for TestSet-2 is poor when compared to the TestSet-1.

We observe that Run-1 performed better than other runs. We believe that Run-2 and Run-3 techniques could perform well when we have many more classes present for identi cation. 4

Conclusion

We have discussed our methodology used to solve the task given at INLI-2018 and have derived some insights from the achieved results. The achieved results (i.e. 42.1% for test TestSet-1 and 31.8% for test TestSet-2) are moderate when compared to the other techniques used in this task. We believe that improving the feature set could improve the results.

References

1. Abbasi , A. , Chen , H.: Applying authorship analysis to extremist-group web forum messages . IEEE Intelligent Systems 20 ( 5 ), 67 { 75 ( 2005 )

2. Anand-Kumar , M. , Barathi-Ganesh , H.B. , Singh , S. , Soman , K.P. , Rosso , P. : Overview of the inli pan at re-2017 track on indian native language identi cation . In: CEUR Workshop Proceedings . vol. 2036 , pp. 99 { 105 ( 2017 )

3. Anand-Kumar , M. , Barathi-Ganesh , H.B. , Soman , K.P. : Overview of the inli@ re2018 track on indian native language identi cation . In: CEUR Workshop Proceedings ( 2018 )

4. Gebre , B.G. , Zampieri , M. , Wittenburg , P. , Heskes , T. : Improving native language identi cation with tf-idf weighting . In: BEA@ NAACL-HLT ( 2013 )

5. Malmasi , S. , Evanini , K. , Cahill , A. , Tetreault , J. , Pugh , R. , Hamill , C. , Napolitano , D. , Qian , Y. : A report on the 2017 native language identi cation shared task . In: 12th Workshop on Building Educational Applications Using NLP ( 2017 )

6. Mechti , S. , Abbassi , A. , Belguith , L.H. , Faiz , R.: An empirical method using features combination for arabic native language identi cation . In: 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA) . pp. 1 { 5 ( 2016 )

7. Mondal , S. , Bours , P. : Person identi cation by keystroke dynamics using pairwise user coupling . IEEE Transactions on Information Forensics and Security 12 ( 6 ), 1319 { 1329 ( 2017 )

8. Mondal , S. , Bours , P.: A continuous combination of security & forensics for mobile devices . Journal of Information Security and Applications 40 , 63 { 77 ( 2018 )

9. Perkins , R.: New Threats and Countermeasures in Digital Crime and Cyber Terrorism, chap. Native Language Identi cation (NLID) for Forensic Authorship Analysis of Weblogs , pp. 213 { 234 . IGI Global ( 2018 )

10. Stehwien , S. , Pad , S.: Generalization in native language identi cation { learners versus scientists . In: Proceedings of CLiC-IT . pp. 264 { 268 . Trento , Italy ( 2015 )

11. Tetreault , J. , Blanchard , D. , Cahill , A. : A report on the rst native language identi cation shared task . In: In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications ( 2013 )