NAYEL@APDA: Machine Learning Approach for Author Pro ling and Deception Detection in Arabic Texts

Abstract

In this paper, we describe the methods and experiments that have been used in development of our system for Author Pro ling and Deception Detection in Arabic shared task. There are two tasks, Author Pro ling in Arabic Tweets and Deception Detection in Arabic Texts. We have submitted three runs for each task. The proposed system depends on classical machine learning approaches namely Linear Classi er, Support Vector Machine and Multilayer Perceptron Classi er. Bag-of-Word with range of n-grams model has been used for feature extraction. Our submissions for the rst task achieved the second, seventh and third ranks. For the second task, one of our submissions outperformed all other submissions developed by other teams.

Keywords

• Arabic NLP
• Author Pro ling
• Deception Detection

1 Introduction

The tremendous usage of social platforms makes analysing shared contents a crucial task. One of the key tasks is Author Pro ling (AP), which aims at predicting author attributes such as native language, gender, or political attitude [ 14 ]. AP has gained a lot of interest, due to it's applications in di erent areas such as Ecommerce, Cyber-Security and forensics. In E-commerce, companies may analyze online reviews to improve targeted advertising. Analysing online reviews helps companies to improve their marketing strategy by knowing the demographics of people (gender and age) whose liked or disliked their products [ 14 ]. In CyberSecurity, AP can be used for detection of di erent crimes such as phishing, Cyber-blackmailing and Cyber-bullying. In forensics, pro le of authors could be used as valuable additional evidence in criminal investigations [ 18 ].

Arabic is an important language having a huge number of native and nonnative speakers. The research in Natural Language Processing for Arabic language is continuously increasing. Applying NLP tasks for Arabic is a challenge due to di erent aspects of Arabic such as orthography, morphology, dialects, short vowels and word order [ 1 ]. AP has been studied for for English, Spanish and Arabic in [ 14 ], Indian languages [ 6 ] and Russian [ 5 ].

In this paper, we describe the model submitted for Author Pro ling and Deception Detection in Arabic (APDA) shared task [ 16 ]. Shared task comprises of two tasks AP in Arabic Tweets and Deception Detection in Arabic texts. The rst task identi es three attributes of Arabic Twitter users namely, age, gender and language variety. The second task detects the deception in Arabic texts.

2 Related Work

Author pro ling is an important task that involves a lot of challenges and hitches. Many research works have been done on author pro ling in di erent languages. The following are a brief about some of these works in recent years.

Di erent research areas such as psychology, linguistics and NLP have studied the relation between linguistics features and pro le of the corresponding authors. The relation between language use and the personality traits has been studied by Pennebaker et al. [ 15 ]. They studied how variations of linguistic features in a text can provide information regarding the pro le of its author. Author pro ling task at PAN 2013 aimed at identifying age and gender of the author [ 17 ]. A large corpus collected from social media both in English and Spanish has been used for PAN 2013. In PAN 2014[ 13 ], a compiled a corpus of four di erent genres, namely social media, blogs, Twitter, and hotel reviews has been used. Rangel et al. [ 12 ], organized the third author pro ling task for age, gender and personality prediction. English, Spanish, Dutch and Italian languages were considered in this task. Di erent features have been used by participants in model design such as BoW, n-grams, frequencies and punctuations.

Nayel and Shashirekha [ 9, 10 ] have been designed a model for Native Language Identi cation for Indian languages. They used Term Frequency/Inverse Document Frequency (TF/IDF) with range of n-grams as feature extraction approach. They investigated di erent classi cation algorithms such as SVM, multinomial Naive Bayes, ANN-based classi er and ensemble based classi er.

The research works that have been done for age and gender identi cation in the Arabic are rare [ 19 ]. Estival et al. [ 4 ] studied the age and gender identi cation problem as well as level of education in English and Arabic emails. For Arabic, they collected 8,028 emails from 1,030 native speakers of Egyptian Arabic. Several classi ers, such as SVM, KNN and decision trees combined with chi-square and information gain, have been tested to develop the Text Attribution Tool (TAT). They achieved accuracies of 72.10% and 81.15% for gender and age identi cation respectively.

Alsmearat et al [ 2 ] investigated gender identi cation in 500 articles collected from well-known Arabic newsletters. Articles written in Modern Standard Arabic (MSA) have been collected from writers with similar academic pro les and experience in journalistic writings. They applied di erent classi cation algorithms using BoW, sentiments and emotions as a feature set to train the classi ers.

In this work, we applied an e ective ML-based approach using a simple TF/IDF features for the APDA shared task.

3 Task Description and Corpora

APDA shared task consists of two main tasks, the following are the descriptions of both tasks, Task 1. Author Pro ling in Arabic Tweets which aims at identifying author personality such as gender, age and language variety of Arabic Twitter users. In this subtask given a twitter written in Arabic, the system predict the age range, gender and language variety of twitter writer. There are three categories of age, under 25, between 25, and 35 and above 35. For language variation, 15 Arabic varieties have been considered namely Algeria (AL), Egypt (EG), Iraq (IR), Kuwait (KW), Lebanon-Syria (LS), Libya (LI), Morocco (MO), Oman (OM), Palestine-Jordan (PJ), Qatar (QA), Saudi Arabia (SA), Sudan (SU), Tunisia (TU), United Arab Emirates (UAE), Yemen (YE).

The training corpus for this task consists of tweets in Arabic, labeled with age, gender and language variety. This corpus is divided into ve sub-corpora: dz-ag-iq (for AL,EG and IR), kw-lbsy-ly (for KW, LS and LI), ma-om-psjo (for MO, OM and PJ), qa-sa-sd (for QA, SA and SU) and tn-uae-ye (for TU, UAE and YE).

Task 2. Deception Detection in Arabic Texts which detects the deception in Arabic. The text is annotated with credible or non-credible label. There are two genre of data Twitter and news headlines.

The training corpus consists of this task contains two di erent genres. The rst one is Twitter, a set of tweets written in Arabic collected and annotated with credible and non-credible labels. The second genre is news headlines, some news headlines are collected from news agencies and labeled with credible and non-credible labels.

4 Approaches

A detailed description of our model and the classi cation algorithms have been used are given in this section.

4.1 Problem Formulation

Given a set of segments of text such as a tweet, comment or news headline S = fs1; s2; :::; sng and each segment is composite of a set of tokens or words si = fw1; w2; :::; wkg. Consider a set of 15 language varieties as described above L = fAL; EG; IR; KU; LS; LI; M O; OM; P J; QA; SA; SU; T U; U AE; Y Eg.

Assume that, the set A = fU N; BE; ABg represents the age categories under 25 (UN), between 25, and 35 (BE) and above 35 (AB). In addition, the set G = fM; F g represents male and female respectively. Then, we can formalize each subtask as follows: Task 1. It has been formalized as a multi-label classi cation problem. A multilabel classi cation is a classi cation problem where, the instance can be assigned with multiple class labels. Given an instance sk 2 S, we have to assign the triple < g; a; l > such that, g 2 G, a 2 A and l 2 L.

Task 2. It has been formalized as a simple binary classi cation problem. Given a text, the model should decide whether this text is deception or not.

4.2 Model

Our model consists of the following steps:

A. Preprocessing

Preprocessing is a key step in building models for Arabic language. In this step, each tweet sk has been tokenized into a set of words or tokens to get n-gram bag of words. the following processes have been implemented to each tweet: Punctuation Elimination We removed punctuation marks such as f'+', ' ', '#', '$'.. g, which are increasing the dimension of feature space with redundant features. Example of redundancy, the following tokens f ½ËAÓQË@# , ½ËAÓQË@ g pronounced "Al Zamalek (a famous football team in Egypt)", are the same with extra # which produces redundant features. Tweet Cleaning Twitter users usually do not follow the standard rules of the language especially Arabic language. A common manner of users is to repeat a speci c letter in a word. Cleaning the tokens from this redundant letters helps in feature space reduction. In our experiments, the letter is assumed to be redundant if it is repeated more than two times. For example the words " éêêêêêêë" ("hahahah" i.e. giggles) and "Ég. @@@@@@A«" (i.e. "urgent") containing redundant letter and will be reduced to " éë" and "Ég. @A«" respectively.

B. Feature Extraction

TF/IDF with range of n-grams has been used to represent tweets as vectors. If <w1; w2; : : : ; wk> are the tokenized words in a tweet Tj, the vector associated to the tweet Tj will be represented as <vj1; vj2; : : : ; vjk> where vji is the weight of the token wi in tweet Tj which is calculated as:vji = tfji log

N + 1 dfi + 1 where tfji is the total number of occurrences of token wi in the tweet Tj, dfi is the number of tweets in which the token wi occurs and N is the total number worth loss (the City refers to Manchester City football team) ) has following set of features f"úaeJ‚Ë@", "€ñ®jJ‚ AÓ", " èPA‚mÌ'@", "€ñ®jJ‚ AÓ úaeJ‚Ë@", " èPA‚mÌ'@ €ñ®jJ‚ AÓ"g.

C. Training the Classi er Three classi ers have been trained for our model namely Linear classi er, SVM and Multilayer Perceptron [ 20 ]. Linear classi er uses a set of linear discriminant functions to distinguish between different classes [ 20 ]. Linear classi er is a simple and computationally e ective approach. SVM is a linear classi er which uses training samples or vectors close to the boundaries of classes as support vectors. SVM implemented for di erent NLP tasks e ectively [ 11, 8 ]. Multilayer Perceptron (MLP) is a feedforward ANN that characterized by several layers of input nodes connected as a directed graph between the input and output layers. MLP uses back propagation for training the network. MLP is a deep learning method.

4.3 Performance Evaluation 1. Task 1.

Accuracy is used to evaluate the performance of systems for this task. Individual accuracies will be calculated for each subtask (age, gender, language variety). Systems will be ranked by the joint accuracy (when all subtasks are properly identi ed together). 2. Task 2.

The performance of systems developed for this task will be evaluated using the macro-averaged measures (precision, recall and F1-score) and systems will be ranked by F1-score.

5 Experiments and Results

We have designed a model and used di erent classi cation algorithms for the different submissions. The classi cation algorithms are linear classi er, SVM and MLP classi ers. Stochastic gradient descent optimization algorithm has been used for linear classi er. Linear kernel has been applied for SVM kernel. The number of neurons in hidden layer is 20 neurons and the logistic function has been used as activation function for MLP classi er.

Separate models have been trained for each subtask (age, gender and language variety), then the output have been combined. While training the classiers, 5-fold cross-validation technique has been used. The cross validation accuracies of all classi cation approaches for task 1 and task 2 are given in Table 1 and Table 2 respectively.

In Table 1, we highlighted the best reported accuracies for each sub-corpus and subtask. It is clear that, linear classi er reported the best accuracies for the majority of subtasks and sub-corpora. While, MLP gives best accuracy for the age subtask of tn-uae-ye corpus.

Among 28 submissions of task 1, our submissions achieved 2nd, 3rd and 7th ranks as shown in Table 3. It is clear that linear classi er reported the best performance among all of our submissions. There are 25 submissions for task 2 and our submissions achieved 1st, 2nd and 6th ranks as shown in Table 3. Our submission based on SVM approach outperforms all 25 submissions of all teams.

References

1. Alayba , A.M. , Palade , V. , England , M. , Iqbal , R.: Improving sentiment analysis in arabic using word representation . 2018 IEEE 2nd International Workshop on Arabic and Derived Script Analysis and Recognition (ASAR) (Mar 2018 ). https://doi.org/10.1109/asar. 2018 .8480191

2. Alsmearat , K. , Shehab , M. , Al-Ayyoub , M. , Al-Shalabi , R. , Kanaan , G.: Emotion analysis of arabic articles and its impact on identifying the author's gender . In: 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA) . pp. 1 { 6 (Nov 2015 ). https://doi.org/10.1109/AICCSA. 2015 .7507196

3. Cappellato , L. , Ferro , N. , Jones , G.J.F. , SanJuan , E. (eds.): Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum , Toulouse, France, September 8- 11 , 2015 , CEUR Workshop Proceedings, vol. 1391 . CEUR-WS.org ( 2015 ), http://ceur-ws. org/ Vol-1391

4. Estival , D. , Gaustad , T. , Pham , S.B. , Radford , W. : Pro ling for english emails ( 2007 )

5. Litvinova , T. , Gudovskikh , D. , Sboev , A. , Seredin , P. , Litvinova , O. , Pisarevskaya , D. , Rosso , P. : Author gender prediction in russian social media texts . In: van der Aalst, W.M.P. , Khachay , M.Y. , Kuznetsov , S.O. , Lempitsky , V.S. , Lomazova , I.A. , Loukachevitch , N.V. , Napoli , A. , Panchenko , A. , Pardalos , P.M. , Savchencko , A.V. , Wasserman , S. , Ignatov , D.I . (eds.) Supplementary Proceedings of the Sixth International Conference on Analysis of Images, Social Networks and Texts (AIST 2017 ), Moscow, Russia, July 27 - 29 , 2017 . CEUR Workshop Proceedings , vol. 1975 , pp. 105 { 110 . CEUR-WS.org ( 2017 ), http://ceur-ws. org/ Vol-1975/paper12.pdf

6. Majumder , P. , Mitra , M. , Mehta , P. , Sankhavara , J . (eds.): Working notes of FIRE 2017 - Forum for Information Retrieval Evaluation, Bangalore , India, December 8- 10 , 2017 , CEUR Workshop Proceedings, vol. 2036 . CEUR-WS.org ( 2018 ), http://ceur-ws. org/ Vol-2036

7. Mehta , P. , Rosso , P. , Majumder , P. , Mitra , M. (eds.): Working Notes of FIRE 2018 - Forum for Information Retrieval Evaluation, Gandhinagar , India, December 6- 9 , 2018 , CEUR Workshop Proceedings, vol. 2266 . CEUR-WS.org ( 2018 ), http://ceurws.org/Vol-2266

8. Nayel , H.A. , Shashirekha , H.L. : Improving NER for clinical texts by ensemble approach using segment representations . In: Bandyopadhyay, S . (ed.) Proceedings of the 14th International Conference on Natural Language Processing, ICON 2017 , Kolkata, India, December 18-21 , 2017 . pp. 197 { 204 . NLP Association of India ( 2017 ), https://aclweb.org/anthology/papers/W/W17/W17-7525/

9. Nayel , H.A. , Shashirekha , H.L. : Mangalore-university@inli- re-2017: Indian native language identi cation using support vector machines and ensemble approach . In: Majumder et al. [6] , pp. 106 { 109 , http://ceur-ws. org/ Vol-2036/T4-2.pdf

10. Nayel , H.A. , Shashirekha , H.L. : Mangalore university inli@ re2018: Arti cial neural network and ensemble based models for INLI . In: Mehta et al. [7] , pp. 110 { 118 , http://ceur-ws. org/ Vol- 2266 / T2 -10.pdf

11. Nayel , H.A. , Shashirekha , H.L. , Shindo , H. , Matsumoto , Y. : Improving multiword entity recognition for biomedical texts . CoRR abs/ 1908 .05691 ( 2019 ), http://arxiv.org/abs/ 1908 .05691

12. Pardo , F.M.R. , Celli , F. , Rosso , P. , Potthast , M. , Stein , B. , Daelemans , W. : Overview of the 3rd author pro ling task at PAN 2015 . In: Cappellato et al. [ 3 ], http://ceur-ws. org/ Vol- 1391 /inv-pap12 -CR .pdf

13. Pardo , F.M.R. , Rosso , P. , Chugur , I. , Potthast , M. , Trenkmann , M. , Stein , B. , Verhoeven , B. , Daelemans , W. : Overview of the author pro ling task at PAN 2014 . In: Cappellato, L. , Ferro , N. , Halvey , M. , Kraaij , W. (eds.) Working Notes for CLEF 2014 Conference, She eld , UK, September 15-18 , 2014 . CEUR Workshop Proceedings , vol. 1180 , pp. 898 { 927 . CEUR-WS.org ( 2014 ), http://ceur-ws.org/Vol1180/CLEF2014wn-Pan-RangelEt2014.pdf

14. Pardo , F.M.R. , Rosso , P. , Montes- y-Gomez, M. , Potthast , M. , Stein , B. : Overview of the 6th author pro ling task at PAN 2018: Multimodal gender identi cation in twitter . In: Cappellato, L. , Ferro , N. , Nie , J. , Soulier , L . (eds.) Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum , Avignon, France, September 10-14 , 2018 . CEUR Workshop Proceedings , vol. 2125 . CEUR-WS.org ( 2018 ), http://ceur-ws. org/ Vol- 2125 /invited paper 15.pdf

15. Pennebaker , J.W. , Mehl , M.R. , Niederho er, K.G.: Psychological aspects of natural language use: Our words, our selves . Annual review of psychology 54(1) , 547 { 577 ( 2003 )

16. Rangel , F. , Rosso , P. , Char , A. , Zaghouani , W. , Ghanem , B. , Sanchez-Junquera , J. : Overview of the track on author pro ling and deception detection in arabic . In: Working Notes of Forum for Information Retrieval Evaluation FIRE 2019 , Kolkata, India, December 12-15 , 2019 . CEUR Workshop Proceedings, CEUR-WS.org ( 2019 )

17. Rangel , F. , Rosso , P. , Koppel , M. , Stamatatos , E. , Inches , G. : Overview of the author pro ling task at pan 2013 . In: CLEF Conference on Multilingual and Multimodal Information Access Evaluation . pp. 352 { 365 . CELCT ( 2013 )

18. Rosso , P. , Pardo , F.M.R. , Ghanem , B. , Char , A. : ARAP: arabic author pro ling project for cyber-security . Procesamiento del Lenguaje Natural 61 , 135 { 138 ( 2018 ), http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/5654

19. Rosso , P. , Rangel , F. , Far as, I.H. , Cagnina , L. , Zaghouani , W. , Char , A. : A survey on author pro ling, deception, and irony detection for the arabic language . Language and Linguistics Compass 12 ( 4 ), e12275 ( 2018 ), https://onlinelibrary.wiley.com/doi/abs/10.1111/lnc3.12275

20. Theodoridis , S. , Koutroumbas , K. : Chapter 3 - linear classi ers . In: Pattern Recognition (Fourth Edition) , pp. 91 { 150 . Academic Press, Boston, fourth edition edn. ( 2009 ), http://www.sciencedirect.com/science/article/pii/B9781597492720500050