BENHA@IDAT: Improving Irony Detection in Arabic Tweets using Ensemble Approach

Abstract

This paper describes the methods and experiments that have been used in the development of our model submitted to Irony Detection for Arabic Tweets shared task. We submitted three runs based on our model using Support Vector Machines (SVM), Linear and Ensemble classi ers. Bag-of-Words with range of n-grams model have been used for feature extraction. Our submissions achieved accuracies of 82.1%, 81.6% and 81.1% for ensemble based, SVM and linear classi ers respectively.

Keywords

• Irony Detection Arabic NLP Ensemble Based Classi ers SVM
• 3

1 Introduction

Analyzing social media is an important research area due to the huge amount of information streaming from online social networking and microblogging platforms such as Twitter, Facebook and Instagram. One of the attractive tasks is irony detection which can be de ned as the con ict of using the verbal meaning of a sentence and its intended meaning [ 15 ]. Twitter platform comprises of text communications with a high percentage of ironic messages. Devices and platforms monitoring the sentiment in Twitter messages are faced with the problem of wrong polarity classi cation of ironic messages [ 16 ].

Irony is studied by various disciplines, such as linguistics, philosophy, and psychology [ 18 ], but it is di cult to de ne it in formal terms. In computational linguistics, irony is often used as a concept of sarcasm, although some researchers di erentiate between irony and sarcasm, considering that sarcasm tends to be harsher, humiliating, degrading and more aggressive [ 4, 8 ].

Irony detection is not a straightforward problem, since ironic statements are used to express the contrary of what is being said [ 20 ], therefore it is di cult to be solved by current systems. Being a creative form of language, there is no agreement in the literature on how verbal irony should be de ned. Recently irony detection has been studied from a computational perspective as one of classi cation problem that separates ironic from non-ironic statements [ 16 ].

Arabic is an important natural language having a huge number of speakers. The research in Natural Language Processing (NLP) for Arabic is constantly increasing. However, there is still a need to handle the complexity of NLP tasks in Arabic. This complexity arises from di erent aspects, such as orthography, morphology, dialects, short vowels and word order [ 1 ]. Irony detection in Arabic is a challenging task. The following example : " ¼PAJ. Ó úae‚k I. jJ Kð úΫ YËAg Ñ«YK AJk@ " from Twitter illustrates how the author ironically employs a positive opinion word "Ñ«YK" (which means support) towards Khaled Ali, the former presidential candidate of Egypt to express a negative opinion and take opposite action " ¼PAJ. Ó úae‚k I. jJ K " (which means then elect Hosni Mubarak).

In this paper we have developed a model for detecting ironic Arabic tweets using Machine Learning (ML) approach. The proposed model classi es a given tweet as either ironic or non-ironic. The paper is organized as follows: section 2 introduces the related work and section 3 contains a description of our model. Section 4 gives a brief overview about dataset and the performance evaluation metric is given in section 5. Results and future work are tackled in section 6 and section 7 respectively.

2 Related

Work

Irony detection is a very challenging task that encountered a lot of development through the years. There are many research works that have been done on English language and fewer research on other natural languages specially the Arabic language. Here are some of the recent research works that contribute into the problem.

Reyes and Rosso [ 14 ] have focused on identifying the key components to detect irony in English customer reviews via computational point of view. Reviews that were posted by means of an online viral e ect have been selected. They have designed a model with six categories namely, n-grams, POS ngrams, funny pro ling, positive/negative pro ling, a ective pro ling, and pleasantness pro ling to represent irony from di erent linguistic layers. They achieved good results using three di erent classi ers in terms of accuracy and F-score.

In [ 19 ], some patterns associated to ironic statements in Brazilian Portuguese have been analyzed and implemented. A common ground between the author of the tweets and their audience is required in order to establish some background information on the text. Features like the city, time, and genre have been considered while analysing the patterns. Results showed that patterns related to symbolic language, such as laughter marks and emoticons are the best hints to irony and sarcasm. In addition, results illustrated that heavy punctuations are clues to ironic statements and patterns related to static expressions are bad search choices that have given low output results.

Barbieri and Saggion [ 2 ] casted the automatic detection of irony as a classication problem. They have proposed a model capable of detecting irony in the social network Twitter based on lexical features. Tweets that have hashtag irony and some other topics have been selected to create a linguistically motivated set of features. The features take into account frequency, written/spoken di erences, sentiments, ambiguity, intensity, synonymy and structure. Results showed that their model outperforms bag-of-words approach across-domains.

emotIDM [ 5 ], a model for irony detection in Twitter, has been developed by formulation of the task as a classi cation problem. It was evaluated on a set of representative Twitter corpora that included samples of ironic and not ironic messages, which were di erent along various dimensions: size, balanced vs imbalance distribution, collection methodology and criteria. Results showed good performances in classi cation terms across all these dimensions. It performed better in cases of datasets with balanced distribution, where a self-tagging methodology has been applied.

Arabic tweets irony detection has been addressed in [ 7 ], by designing a binary classi er based system for irony detection in Arabic tweets. The classi er uses four groups of features, among which surface, sentiment, shifter, and contextual features. Tweets with irony hashtags in Arabic language have been collected. Results showed that state-of-the art features can be successfully applied to Arabic with accuracy of 72.76%.

In this paper, we developed a ML-based model for irony detection in Arabic tweets. Our model uses Term Frequency/Inverse Document Frequency (TF/IDF) with ranges of n-grams for extracting features. We registered for Irony Detection for Arabic Tweets (IDAT) shared task and have submitted three runs using di erent classi cation algorithms namely, Support Vector Machine (SVM), Linear classi er that uses Stochastic Gradient Descent (SGD) as an optimizer and Ensemble classi er.

3 Model Description

Given a set of tweets T = fT1; T2; :::; Tng and a set C = fI; N g of classes representing Ironic and Non-ironic tweets respectively, the task of detecting ironic tweets can be formalised as a simple binary classi cation problem that assigns one of two prede ned classes of C to an unlabelled tweet Tk.

The general structure of our model is shown in Fig. 1. The rst step in our model is preprocessing which aims at cleaning the data and removing vain parts from it. The second step is feature extraction which is necessary for both training and testing purposes. The third step is training the classi cation algorithm. The following subsections give details of each step.

3.1 Preprocessing

Preprocessing is a key step in building models for Arabic language. In this step, each tweet Tk has been tokenized into a set of words or tokens to get n-gram bag of words. The following processes have been implemented to each tweet: Punctuation Elimination We removed punctuation marks such as f'+', ' ', '#', '$'.. g, which are increasing the dimension of feature space with redundant features. Example of redundancy, the following tokens f éJK. QªË@_ hAJ.“# , éJK. QªË@_ hAJ.“, éJK. QªË@ hAJ.“ g pronounced as "Sabah Al Arabiya" and means "morning of Al Arabiya"(Al Arabiya4 is a news agency). Existence of " " and "#" will add redundant features f éJK. QªË@_ hAJ.“# , éJK. QªË@_ hAJ.“g which a ects the performance of irony detection.

Tweet Cleaning Twitter users usually do not follow the standard rules of the language especially Arabic language. A common manner of users is to repeat a speci c letter in a word. Cleaning the tokens from this redundant letters helps in feature space reduction. In our experiments, the letter is assumed to be redundant if it is repeated more than two times. For example the words " éêêêêêêë" ("hahahah" i.e. giggles) and "Ég. @@@@@@A«" (i.e. "urgent") containing redundant letter and will be reduced to " éë" and "Ég. @A«" respectively. 4 https://www.alarabiya.net/

3.2 Features Extraction

TF/IDF with range of n-grams has been used to represent tweets as vectors. If <w1; w2; : : : ; wk> are the tokenized words in a tweet Tj , the vector associated to the tweet Tj will be represented as <vj1; vj2; : : : ; vjk> where vji is the weight of the token wi in tweet Tj which is calculated as:vji = tfji log

N + 1 dfi + 1 where tfji is the total number of occurrences of token wi in the tweet Tj , dfi is the number of tweets in which the token wi occurs and N is the total number of tweets.

We used range of 2-grams model, i.e. unigram and bigram. For example sentence "øPñ»PAƒ I¢®ƒ@ Ég. @" (Yes I beat Sarkozy) has following set of features

3.3 Building Classi ers

Three classi ers have been trained for our model mainly SVM [ 17 ], Linear classi er [ 17 ] and Ensemble classi er. SVM is a binary classi er which has been used for di erent NLP tasks e ectively [ 12, 11 ]. Linear classi er uses linear discriminant function.

Ensemble approach uses a set of classi ers as base classi ers and combines the output of these base classi ers to get the nal output [ 10 ]. Ensemble approach has been implemented in di erent NLP tasks such as Native Language Identi cation [ 11 ] and Named Entity Recognition [ 10, 13 ]. Random Forests (RF) classi er is a supervised algorithm which involves multiple decision trees and each tree is built using independently sampled random vector [ 3 ]. Multinomial Bayes classi er is an instance of Naive Bayes classi er that takes in account word frequency in documents [ 9 ].

SVM has been used to for the rst submission. Linear classi er uses SGD as an optimization algorithm; has been implemented for second submission. For third submission, an ensemble-based model that uses four models as base classiers, which are: RF, Multinomial Bayes, linear, and SVM classi ers. The structure of base classi ers is same as the structure shown in Fig. 1.

4 Dataset

The data provided by organizers was taken from Twitter regarding political issues and Middle East events from 2011 to 2018 and is divided into training set and testing set [ 6 ]. The training set contains 4023 labeled tweets and testing set contains 1005 unlabelled tweets.

5 Results

F-score has been used to evaluate the performance of all submissions. F-score is a harmonic mean of Precision (P) and Recall (R) [ 10 ] and calculated as follow: F score = 2 P R

P + R

We have used 5-fold cross-validation technique. The cross validation accuracy of all training classi ers for all submissions is given in Table 1. It is clear that SVM gives the best accuracy with higher standard deviation while development, while linear classi er gives the worst accuracy with minimum standard deviation. Among 26 submissions received by shared task organizers, our submissions achieve 5th, 11th and 13th ranks as shown in Table 2. It is clear that ensemble-based classi er gives better F1 score of our submissions. 13 Linear Based Classi er

6 Conclusion and Future Work

In this work we used a simple TF/IDF with range of n-grams model to extract feature for training the classi ers. The classi ers used are SVM, linear and Ensemble-based which are used to create the submitted runs. This work can be extended by using word embeddings as features. In addition, Arti cial Neural Network based classi ers can be used as a classi cation algorithm.

References

1. Alayba , A.M. , Palade , V. , England , M. , Iqbal , R.: Improving sentiment analysis in arabic using word representation . 2018 IEEE 2nd International Workshop on Arabic and Derived Script Analysis and Recognition (ASAR) (Mar 2018 ). https://doi.org/10.1109/asar. 2018 .8480191

2. Barbieri , F. , Saggion , H.: Modelling irony in twitter . In: Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics . pp. 56 { 64 . Association for Computational Linguistics, Gothenburg, Sweden (Apr 2014 ), https://www.aclweb.org/anthology/E14-3007

3. Breiman , L. : Random forests . Machine Learning 45(1) , 5 {32 (Oct 2001 ), https://doi.org/10.1023/A:1010933404324

4. CLIFT , R.: Irony in conversation . Language in Society 28 ( 4 ), 523 { 553 ( 1999 ). https://doi.org/10.1017/S0047404599004029

5. Far as, D.I.H. , Patti , V. , Rosso , P. : Irony detection in twitter: The role of affective content . ACM Trans. Internet Technol . 16 ( 3 ), 19 :1{ 19 :24 (Jul 2016 ), http://doi.acm. org/10.1145/2930663

6. Ghanem , B. , Karoui , J. , Benamara , F. , Moriceau , V. , Rosso , P. : Idat@ re2019: Overview of the track on irony detection in arabic tweets . In: Mehta, P. , Rosso , P. , Majumder , P. , Mitra , M. (eds.) Working notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings , CEUR-WS.org ( 2019 ), Kolkata, India, December 12 -15

7. Karoui , J. , Zitoune , F.B. , Moriceau , V. : Soukhria: Towards an irony detection system for arabic in social media . Procedia Computer Science 117 , 161 { 168 ( 2017 ), http://www.sciencedirect.com/science/article/pii/S1877050917321622

8. Lee , C.J. , Katz , A.N.: The di erential role of ridicule in sarcasm and irony . Metaphor and Symbol 13 ( 1 ), 1 { 15 ( 1998 )

9. McCallum , A. , Nigam , K. : A comparison of event models for naive bayes text classi cation . In: AAAI-98 workshop on learning for text categorization . pp. 41 { 48 . The COLING 2012 Organizing Committee ( 1998 )

10. Nayel , H.A. , Shashirekha , H.L. : Improving NER for clinical texts by ensemble approach using segment representations . In: Bandyopadhyay, S . (ed.) Proceedings of the 14th International Conference on Natural Language Processing, ICON 2017 , Kolkata, India, December 18-21 , 2017 . pp. 197 { 204 . NLP Association of India ( 2017 ), https://aclweb.org/anthology/papers/W/W17/W17-7525/

11. Nayel , H.A. , Shashirekha , H.L. : Mangalore-university@inli- re-2017: Indian native language identi cation using support vector machines and ensemble approach . In: Majumder, P. , Mitra , M. , Mehta , P. , Sankhavara , J . (eds.) Working notes of FIRE 2017 - Forum for Information Retrieval Evaluation, Bangalore , India, December 8- 10 , 2017 . CEUR Workshop Proceedings , vol. 2036 , pp. 106 { 109 . CEUR-WS.org ( 2017 ), http://ceur-ws. org/ Vol-2036/T4-2.pdf

12. Nayel , H.A. , Shashirekha , H.L. : Mangalore university inli@ re2018: Arti cial neural network and ensemble based models for INLI . In: Mehta, P. , Rosso , P. , Majumder , P. , Mitra , M. (eds.) Working Notes of FIRE 2018 - Forum for Information Retrieval Evaluation, Gandhinagar , India, December 6- 9 , 2018 . CEUR Workshop Proceedings , vol. 2266 , pp. 110 { 118 . CEUR-WS.org ( 2018 ), http://ceurws.org/Vol- 2266 / T2 -10.pdf

13. Nayel , H.A. , Shashirekha , H.L. , Shindo , H. , Matsumoto , Y. : Improving multiword entity recognition for biomedical texts . CoRR abs/ 1908 .05691 ( 2019 ), http://arxiv.org/abs/ 1908 .05691

14. Reyes , A. , Rosso , P. : Mining subjective knowledge from customer reviews: A speci c case of irony detection . In: Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011) . pp. 118 { 124 . Association for Computational Linguistics, Portland, Oregon (Jun 2011 ), https://www.aclweb.org/anthology/W11-1715

15. Reyes , A. , Rosso , P. : On the di culty of automatically detecting irony: Beyond a simple case of negation . Knowl. Inf. Syst . 40 ( 3 ), 595 {614 (Sep 2014 ), http://dx.doi.org/10.1007/s10115-013-0652-8

16. Reyes , A. , Rosso , P. , Veale , T. : A multidimensional approach for detecting irony in twitter . Lang. Resour. Eval . 47 ( 1 ), 239 {268 (Mar 2013 ), http://dx.doi.org/10.1007/s10579-012-9196-x

17. Theodoridis , S. , Koutroumbas , K. : Chapter 3 - linear classi ers . In: Pattern Recognition (Fourth Edition) , pp. 91 { 150 . Academic Press, Boston, fourth edition edn. ( 2009 ), http://www.sciencedirect.com/science/article/pii/B9781597492720500050

18. Utsumi , A. : A uni ed theory of irony and its computational formalization . In: COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics ( 1996 ), https://www.aclweb.org/anthology/C96-2162

19. Vanin , A.A. , Freitas , L.A. , Vieira , R. , Bochernitsan , M.: Some clues on irony detection in tweets . In: Proceedings of the 22Nd International Conference on World Wide Web . pp. 635 { 636 . WWW '13 Companion, ACM , New York, NY, USA ( 2013 ), http://doi.acm. org/10 .1145/2487788.2488012

20. Zhang , S. , Zhang , X. , Chan , J. , Rosso , P. : Irony detection via sentiment-based transfer learning . Information Processing and Management 56 ( 5 ), 1633 { 1644 ( 2019 ), http://www.sciencedirect.com/science/article/pii/S0306457318307428