IDAT@FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets

Abstract

This overview paper describes the rst shared task on irony detection for the Arabic language. The task consists of a binary classi cation of tweets as ironic or not using a dataset composed of 5; 030 Arabic tweets about di erent political issues and events related to the Middle East and the Maghreb. Tweets in our dataset are written in Modern Standard Arabic but also in di erent Arabic language varieties including Egypt, Gulf, Levantine and Maghrebi dialects. Eighteen teams registered to the task among which ten submitted their runs. The methods of participants ranged from feature-based to neural networks using either classical machine learning techniques or ensemble methods. The best performing system achieved F-score value of 0:844, showing that classical feature-based models outperform the neural ones.

Keywords

• Irony detection
• Arabic language
• Social media

-

Irony is a complex linguistic phenomenon widely studied in philosophy and linguistics. In the standard pragmatic model [ 11 ], irony is viewed as an apparent violation of the maxim of quality, stating that the speaker does not say what he believes to be false. In this model, when one ironically utters P , one conversationally implicates its opposite, that is N ot(P ). For example, if one says to his colleague "Congratulation for your great presentation" after a disappointing talk. This vision has been criticized by several authors who pointed out that logical opposition between what is said and what is intended captures only one type of irony. To overcome this de ciency, di erent theories have been proposed to deal with the multi-dimensional nature of opposition. Among them, we cite [ 31, 6, 13, 2, 33 ] that respectively describe irony in terms of echoic mention, allusional pretense, predicate and propositional negations, relevant inappropriateness, and implicit display. Irony is used here as an umbrella term that covers a variety of other gurative devices such as satire, parody, and sarcasm [ 6, 10 ].

Irony detection has gained relevance recently, due to its importance in various NLP applications such as sentiment analysis, hate speech detection, author proling, fake news detection, and crisis management (e.g., terrorist attacks, public disorder). For example, recent studies on irony show that the performances of sentiment analysis systems drastically decrease when applied to ironic texts [ 3, 8, 15, 34 ]. This is mainly due to the complexity of ironic contents that make use of gures of speech to convey non-literal meaning.

Most state of the art approaches to irony detection consider social media data and tweets in particular, as speci c hashtags (#irony, #sarcasm) are often employed by users to help readers understand their ironic contents. These hashtags are used as gold labels to detect irony in a supervised learning setting. Most related work concern English [ 14 ] with some e orts in French [ 18 ], Portuguese [ 4 ], Italian [ 9 ], Dutch [ 22 ], Hindi [ 32 ] and Arabic [ 17 ]. Also, many shared tasks on irony have been proposed, such as SemEval 2018 task 3 for English [ 14 ], DEFT 2017 for French [ 3 ], IronITA@Evalita 2018 for Italian [ 5 ], and IroSvA@IberLEF2019 for Spanish variants [ 28 ] (from Spain, Cuba and Mexico). As far as we know, this is the rst shared task on irony for the Arabic language and will be a good opportunity to compare the performances of Arabic irony detection to those reported in recent shared tasks in other languages. 2

Processing Arabic Tweets: Main Challenges

Computational processing of the Arabic language has received a great attention in the literature for over a twenty years4. Several resources and tools have been built to deal with Arabic nonconcatenative morphology and Arabic syntax [ 24 ]. There is also a wide range of Arabic NLP (ANLP) applications including question answering [ 26 ], automatic translation [ 30 ] and sentiment analysis [ 21 ]. However, the eld of ANLP is still very vacant at the layer of pragmatics. As far as we know, the sole e ort towards Arabic irony detection was done by Karoui et al. [ 17 ] who proposed a supervised approach to detecting ironic tweets. The performance of several groups of features (like surface, sentiment, shifter and contextual features) have been assessed achieving an accuracy of 72.36% on a dataset composed of 3; 466 tweets among which 50% were ironic.

Detecting irony in Arabic poses a signi cant challenge, as the Arabic language is mainly characterized by the lack of diacritics (dedicated letters to represent short vowels), complex agglutination, pro-drop structure, and free order word structure. For instance, [ 7 ] estimated that the average number of ambiguities for a token in Arabic can reach 19:2, compared to 2:3 in most other languages. Also, short vowels are not often explicitly marked in writing. Indeed, they are neither written in the Arabic handwriting of everyday use nor in general publications. 4 For a detailed description of Modern Standard Arabic and an overview of Arabic NLP, see [ 12 ]. Non diacritized texts are highly ambiguous. For example, the word ÕΫ can be (tonobil which means car ). diacritized in 9 di erent forms [ 23 ]: ÕΫ (science), ÕΫ ( ag ), ÕΫ (He was taught ), etc.

In addition to the speci cities of Modern Standard Arabic (MSA) discussed above, dialects pose a number of challenges including a large variations of unstandardized dialectal Arabic, and linguistic code switching between MSA and several dialects, and between Arabic and other languages like English and French. For example, the English word Table can be translated as éËðA£ (tawela) in Egyptian dialect, éÊK. A£ (tabla) in Algerian (Maghrebi) dialect, éJÊJ. £ (tabliah) in Levantine dialect, and éƒAÓ (masa) in some of the Gulf dialect speaking countries. Finally, there are also problems with the extensive use of transliterated words, such as the French word Automobile (Automotive) that becomes ÉJK. ñKñ£ 3 3.1

Data and Annotation

Data Collection

The collected dataset is composed of tweets posted on Twitter during the years 2011 to 2018 about di erent political issues and events related to the Middle East and the Maghreb. A set of prede ned keywords is used to collect tweets, which targeted speci c political gures (e.g., øPCJë (Hillary ), I.Ó@QK (Trump), úae„J‚Ë@ (Al-sissi ), ¼PAJ. Ó (Moubarak ), úae…QÓ (Morsi),úΫ áK. (BenAli) , PA‚ . (Bachar), etc.) which were the subject of the Arab spring and the presidential elections of Egypt and US. From these retrieved tweets, we selected those containing or not the Arabic ironic hashtags éKQm …#, èQj‚Ó#, ÕºîE#, Z@QîDƒ@#5. Before starting with any selection process, we discarded tweets that are duplicated or tweets that depend on external links, images or videos to understand their context.

The collection process resulted in a set of 22; 318 tweets (6; 809 ironic tweets and 15; 509 are not). These tweets are written using standard (formal) and dialectal Arabic, as shown in the examples below. Dialectal tweets contain di erent Arabic language varieties: Egypt (cf.(1)), Gulf (cf.(2)), and Levantine dialects (cf.(3)). (1) (#I will vote El Sisi why so that he stands behind Trump's o ce serving co ee and shining boots) 5 All of these words are synonyms meaning "Irony". (#we failed We see Venus as a star, and yesterday they saw Saturn as a moon .. You look like Kadha #irony) (I remember the old days when they said that Bachar is perfect but the people around him are corrupted...hahahahaha #irony) 3.2

Annotation Procedure and Agreement Study

We took a sample of 6; 000 tweets to investigate the validity of using the original tweets labels. This sample consists of 3; 000 tweets as ironic and 3; 000 as not. It has been manually annotated by two Arabic native speakers following a threesteps procedure where an intermediate analysis of agreement and disagreement between the annotators was carried out. Annotators were rst trained on 100 tweets, then were asked to annotate separately the 6; 000 tweets (this step allows to compute inter-annotator agreements, cf. below). The nal step was adjudication where the main case of disagreements was discussed and solved. Tweets that are either duplicates, do not contain enough context (not a clear sentence or just a link) or where annotators failed to agree have been discarded.

During the annotation process, we found 124 Farsi language tweets in the non-ironic tweets part. These tweets were retrieved automatically because both Arabic and Farsi languages use the same character encoding. We deleted these tweets since each language uses completely di erent meaning and words.

We measured the inter-annotator agreement using Cohen's Kappa and obtained a score of 76%, which referred to a strong agreement. This score is inline with agreements reported in annotating irony in tweets from other languages such as English (e.g., Kappa = 0:72 in the SemEval-2018 task 3) [ 14 ]), French (Kappa = 0:69 in the Deft 2017 shared task [ 3 ]), and Spanish (Kappa = 0:67 in IroSvA@IberLEF [ 28 ]) . The disagreement between the annotators is due to two main factors: (1) the misinterpretation or comprehension of some dialectal words; and (2) the lack of context knowledge to understand the ironic sense of the tweet. The examples (4) and (5) below respectively illustrate each of these two cases above. (4) (They will make us crazy!, sometimes Hosni Moubarak is dead and sometimes he is in coma) In this example, the irony is triggered by the word AKñJ K (made us crazy), written in the Gulf dialect which makes one of the two annotator having limited knowledge of this dialect miss the ironic meaning. éêêë HPA£ €AJË@ ½KYîE (Graduation ceremony of pilots in Sudan on this blessed day, Moubarak God guides you .. the people ew haha) The author of this tweet was o ending a graduation ceremony of pilots in Sudan country, while in that event an helicopter landed in the middle of the ceremony and the air generated by it made the chairs and tents ying away. The lack of context knowledge by one of the annotators made her/him annotate it as not ironic.

We also measured the agreement score between the annotators' labels and the original labels and obtained a kappa score of 0:60, which is moderate. The example (6) shows an ironic tweet (that is containing an ironic hashtag) where both annotators considered it as non ironic. (6) (Kadha : the overthrow of my system is laughable - video #irony) After the adjudication phase, we got a total of 5; 030 tweets among which 2; 614 were ironic tweets and 2; 416 not ironic. We got a total of 5,030 tweets instead of 6,000 after removing the 124 Farsi non ironic tweets and 846 tweets (22 ironic and 824 not ironic) containing a single word and/or several hashtags that make the tweet di cult to understand. Table 1 summarizes some statistics of our dataset. The distribution of tweets in the nal IDAT dataset is given in Table 2. The class distribution (ironic vs. non ironic) is quite similar, with a proportion of ironic tweets of about 52% in both train and test.

Task Description and Evaluation Measures

The task consists of classifying a tweet as ironic or not ironic. The IDAT training set has been released on May 31th and participants had one month and a half to train their systems. The test was then released on July 15th and each participant was allowed to submit a maximum of 3 runs within 10 days.

Participating systems were evaluated using standard evaluation metrics, namely accuracy and F-score as follows. O cial rankings is given according to F-score. (7) (8) (9) (10) 5

Accuracy = P recision =

T rue P ositives + T rue N egatives

T otal number of instances

T rue P ositives

T rue P ositives + T rue N egatives Recall =

T rue P ositives

T rue P ositives + F alse N egatives F score = 2

P recision Recall

P recision + Recall

Methods of Participants and Results

Eighteen teams have registered to the shared task among which ten submitted their runs. Participants were from 7 di erent countries: Algeria, Canada, Egypt, India, Jordan, Pakistan and UK. All team members were from public entities (Universities, Research Centers).

Participants used either traditional machine learning approaches (SVM, Multimodel Naive Bayes, Logistic Regression, Ensemble models) and/or deep learning methods (CNN, RNN, LSTM, Gated Recurrent Unit, Transformers). The tweet contents are represented by traditional bag of words (YOLO [ 20 ], SSNNLP [ 19 ]), n-grams (BENHA [ 27 ]) eventually weighted with TF-IDF (BENHA, YOLO, PITS [ 16 ]), emotion features (Kinmokusu [ 25 ], PITS, YOLO ) and word embeddings (Kinmokusu, Ali Allaith [ 1 ], RGCL [ 29 ], Amrita CEN, Tha3aroon). Embeddings were obtained using di erent models such as Word2Vec, FastText and BERT. Prior to learning, some participants employed well known preprocessing steps such as removing punctuations, usernames, URLs, multiple spaces and letter duplicates (BENHA, RGCL, Amrita CEN, PITS, Kinmokusu, Ali Allaith) while others used Arabic speci c cleaning to account for incorrect word spellings and reduce out of vocabulary terms (RGCL, YOLO, Tha3aroon). This includes byte pair encoding, replacing some Arabic letters (e.g., hamza with Z, ø with ø) and removing diacritics and Arabizi characters (the Arabic chat alphabet).

Table 3 presents participants' results for each submitted run. The results are ranked according to the F-score. For each system, best run is given in bold font. We also compare the results with those of two baselines: SVM with unigrams term frequency (BOW) and a random baseline. 1. YOLO using an ensemble model (based on 3 classi ers: Gradient Boosting, Random Forest and Multilayer Perceptron) relying on surface features (bag of words, TF-IDF, topic modeling). This classical ensemble outperforms both word-level Bi-LSTM ensemble with the same features set (run 2) and an hybrid ensemble that combines the rst two runs (run 3); 2. Chiyu Zhang UBC using BERT in a multi-task learning con guration, BERT being pre-trained on a dialectal Twitter dataset. Several gold data were used to train the model: sentiment analysis, gender detection, age detection, dialect identi cation, and emotion detection. This is the sole model that views dialects as constituting di erent domains and therefore proposed an in-domain pre-training model with dialectal data rather than exclusively on MSA. 3. BENHA using an ensemble model (based on 4 classi ers: Random Forest, SVM, linear and multinomial Bayes) relying on TF-IDF and n-grams.

Neural networks have also been used by Ali Allaith with Arabic FastText embeddings, and RGCL where six di erent architectures were evaluated: pooled Gated Recurrent Unit (GRU) (run 2), Long Short-Term Memory (LSTM), GRU with Attention, 2D Convolution with Pooling (run 1), GRU with Capsule (run 3) and LSTM with Capsule and Attention. Among them, 2D Convolution with Pooling was the best with an F-score of 0:818.

Similarly to Chiyu Zhang UBC, SSN NLP used a deep learning approach using transformers architecture which achieved better compared to GRU with Scaled Luon attention (run 2) and a Multi-Layer Perceptron using a 300 dimensions vector as given by the AraVec pre-trained word embeddings (run 3).

PITS submitted one run consisting of a voting system with three classi ers (Multimodel Naive Bayes, Support Vector Machine, and Logistic Regression) employing a combination of frequency-based and emotion-based features. The latter were obtained by the Deepmoji tool after translating tweets from Arabic to English via the Google translation API.

Finally, Tha3aroon and Amitra CEN used FastText word embeddings for representing tweets while Kinmokusu experimented with CNN and a combination of subword embeddings (obtained with Word2vec CBOW) with surface (word count, presence of hashtags, etc.) and sentiment features as given by external lexicons.

We also report, for each participant's best run, results in terms of accuracy, precision and recall (Table 4). Best recall (and accuracy) was obtained by YOLO while best precision by Chiyu Zhang UBC 's transformer model.

Overall, IDAT results (best F-score= 0:844) are higher compared to the one reported in other irony detection shared tasks in other languages. For instance, macro F-score= 0:783 at the French DEFT2017 [ 3 ], F-score= 0:705 at Task 3@SemEval2018 [ 14 ], and macro F-score= 0:716 at IroSvA@IberLEF-2019 for Spanish [ 28 ].

Conclusion

This paper overviews the rst shared task on irony detection in Arabic social media that aims at classifying a tweet as ironic or not. Eighteen teams participated in the task and a total of ten teams submitted their runs. Systems have been trained on a nearly balanced dataset composed of ironic and non ironic tweets about political issues that raised between 2011 and 2018 in the Middle East and Maghreb. The dataset has been manually annotated and inter-annotator agreement was good (Kappa = 0:76). The methods proposed by participants ranged from traditional features-based approaches relying on bag of words features to neural methods using pre-trained word embeddings. Several neural architectures were tested such as CNN, LSTM and Transformers. Ensemble methods have also been used. The best system achieved an F-score of 0:844 showing that classical features-based models outperform deep learning methods when applied to the IDAT dataset.

Acknowledgments

This publication was made possible by NPRP grant 9-175-1-033 from the Qatar National Research Fund (a member of Qatar Foundation). The ndings achieved herein are solely the responsibility of the last author. The work of Paolo Rosso was also partially funded by Generalitat Valenciana under grant PROMETEO /2019/121.

References

1. Allaith , A. , Shahbaz , M. , Alkoli , M. : Neural Network Approach for Irony Detection from Arabic Text on Social Media . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

2. Attardo , S. : Irony as Relevant Inappropriateness . Journal of Pragmatics 32 ( 6 ), 793 { 826 ( 2000 )

3. Benamara , F. , Grouin , C. , Karoui , J. , Moriceau , V. , Robba , I. : Analyse d'opinion et langage guratif dans des tweets presentation et resultats du De Fouille de Textes DEFT2017. In: Actes de DEFT@ TALN2017 ( 2017 )

4. Carvalho , P. , Sarmento , L. , Silva , M.J. , Oliveira , E.D.: Clues for Detecting Irony in User-Generated Contents: Oh. ..!! It's "so easy";-) . In: Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion . pp. 53 { 56 . ACM ( 2009 )

5. Cignarella , A.T. , Frenda , S. , Basile , V. , Bosco , C. , Patti , V. , Rosso , P. : Overview of the EVALITA 2018 task on irony detection in italian tweets (ironita) . In: Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018 ) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ), Turin, Italy, December 12-13 , 2018 . ( 2018 )

6. Clark , H.H. , Gerrig , R.J.: On the Pretense Theory of Irony . Journal of Experimental Psychology: General 113 ( 1 ), 121 { 126 ( 1984 )

7. Farghaly , A. , Senellart , J.: Intuitive Coding of the Arabic Lexicon . In: Proceedings of the Workshop Machine Translation for Semitic Languages associated to the Association for Machine Translation in the Americas . AMTA ( 2003 )

8. Ghosh , A. , Veale , D.T.: Fracking Sarcasm using Neural Network . In: Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . pp. 161 { 169 ( 2016 )

9. Gianti , A. , Bosco , C. , Patti , V. , Bolioli , A. , Caro , L.D.: Annotating Irony in a Novel Italian Corpus for Sentiment Analysis . In: Proceedings of the 4th Workshop on Corpora for Research on Emotion Sentiment and Social Signals , Istanbul, Turkey. pp. 1 { 7 ( 2012 )

10. Gibbs , R.W.: Irony in Talk Among Friends . Metaphor and symbol 15(1-2) , 5 { 27 ( 2000 )

11. Grice , H.P. : Logic and Conversation . In: Cole, P. , Morgan, J.L . (eds.) Speech Acts. Syntax and Semantics , Volume 3 , pp. 41 { 58 . Academic Press, New York ( 1975 )

12. Habash , N.: Introduction to Arabic Natural Language Processing . Synthesis Lectures on Human Language Technologies , Morgan & Claypool Publishers ( 2010 )

13. Haverkate , H.: A speech act analysis of irony . Journal of Pragmatics 14 ( 1 ), 77 { 109 ( 1990 )

14. Hee , C.V. , Lefever , E. , Hoste , V.: SemEval-2018 Task 3: Irony Detection in English Tweets . In: Proceedings of The 12th International Workshop on Semantic Evaluation , SemEval@NAACL-HLT, New Orleans, Louisiana, June 5-6, 2018 . pp. 39 { 50 ( 2018 )

15. Hernandez Far as, D.I. , Patti , V. , Rosso , P. : Irony Detection in Twitter: The Role of A ective Content . ACM Transactions on Internet Technology (TOIT) 16(3) , 1 { 24 ( 2016 )

16. Kanwar , N. , Kumar , R. , Mundotiya, Agarwal, M. , Singh , C. : Emotion based voted classi er for Arabic irony tweet identi cation . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

17. Karoui , J. , Benamara , F. , Moriceau , V.: SOUKHRIA: Towards an Irony Detection System for Arabic in Social Media . In: Third International Conference On Arabic Computational Linguistics, ACLING 2017, November 5-6 , 2017 , Dubai, United Arab Emirates. pp. 161 { 168 ( 2017 )

18. Karoui , J. , Benamara , F. , Moriceau , V. , Aussenac-Gilles , N. , Belguith , L.H. : Towards a Contextual Pragmatic Model to Detect Irony in Tweets . In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing : Volume 2 short papers . pp. 644 { 650 . ACL-IJCNLP' 15 ( 2015 )

19. Kayalvizhi , S. , Thenmozhi , D. , aKumar, B.S. , Aravindan , C. : SSN NLP@IDATFIRE-2019 : Irony Detection in Arabic Tweets using Deep Learning and Featuresbased Approaches . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

20. Khalifa , M. , Hussein , N.: Ensemble Learning for Irony Detection in Arabic Tweets . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

21. Kiritchenko , S. , Mohammad , S.M. , Salameh , M. : Semeval-2016 task 7: Determining sentiment intensity of english and arabic phrases . In: Proceedings of the International Workshop on Semantic Evaluation . SemEval ' 16 ( 2016 )

22. Liebrecht , C. , Kunneman , F., van den, B.A. : The Perfect Solution for Detecting Sarcasm in Tweets# Not . In: Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . pp. 29 { 37 ( 2013 )

23. Maamouri , M. , Bies , A. , Kulick , S. : Diacritization: A Challenge to Arabic Treebank Annotation and Parsing . In: Proceeding of the British Computer Society Arabic NLP/MT Conference ( 2006 )

24. Marton , Y. , Habash , N. , Rambow , O. : Dependency Parsing of Modern Standard Arabic with Lexical and In ectional Features . Computational Linguistics 39 ( 1 ), 161 { 194 ( 2013 )

25. Moudjari1 , L. , Akli-Astouati , K. : An embedding-based approach for irony detection in Arabic tweets . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

26. Mozannar , H. , Hajal , K.E. , Maamary , E. , Hajj , H.M.: Neural arabic question answering . CoRR abs/ 1906 .05394 ( 2019 ), http://arxiv.org/abs/ 1906 .05394

27. Nayel , H.A. , Medhat , W. , Rashad , M.: BENHA@IDAT: Improving Irony Detection in Arabic Tweets using Ensemble Approach . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

28. Ortega-Bueno , R. , Rangel , F. , Hernandez Far as, D., Rosso , P. , Montes-y Gomez , M. , Medina Pagola , J.E. : Overview of the Task on Irony Detection in Spanish Variants . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019 ), co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2019 ). CEUR-WS. org ( 2019 )

29. Ranasinghe , T. , Saadany , H. , Plum , A. , Mandhari , S. , Mohamed , E. , Orasan , C. , Mitkov , R.: RGCL at IDAT: Deep Learning models for Irony Detection in Arabic Language . In: Proceedings of the IDAT@FIRE2019 . In Mehta P., Rosso P. , Majumder P. , Mitra M . (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019) . CEUR Workshop Proceedings . ( 2019 )

30. Sadat , F. , Mohamed , E.: Improved Arabic-French machine translation through preprocessing schemes and language analysis . In: Proceedings of the Canadian Conference on Arti cial Intelligence . pp. 308 { 314 . AI ( 2013 )

31. Sperber , D. , Wilson, D.: Irony and the use-mention distinction . Radical pragmatics 49 , 295 { 318 ( 1981 )

32. Swami , S. , Khandelwal , A. , Singh , V. , Akhtar , S.S. , Shrivastava , M.: A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection . 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing) ( 2018 )

33. Utsumi , A. : Stylistic and Contextual E ects in Irony Processing . In: Proceedings of the 26th Annual Meeting of the Cognitive Science Society . pp. 1369 { 1374 ( 2004 )

34. Zhang , S. , Zhang , X. , Chan , J. , Rosso , P. : Irony detection via sentiment-based transfer learning . Information Processing Management 56 ( 5 ), 1633 { 1644 ( 2019 )