Computational Models for Irony Detection in Three Spanish Variants

Abstract

The lack of understanding of gurative language online, such as ironic messages, is a common cause of error for systems that analyze automatically the users' opinions online detecting sentiment, emotions or stance. In order to deal with this problem of automatic processing of natural language, IroSvA shared task at IberLef 2019 asks participants to detect, for the rst time, irony in short texts written in Spanish language, considering the three linguistic variants from Spain, Mexico and Cuba. Another novelty of this task is the presence of labels specifying the context of the utterance, such as current political or social issues discussed online. In the context of this shared task, we approached irony detection in Spanish short texts trying to exploit the provided topic information. In addition, we investigated the usefulness of stylistic, lexical and a ective features during the development of the irony detection models for the three Spanish variants. Experimental results and further analyses allow to shed some light on the analogies and di erences in the expression of irony in the three variants, and suggest new research directions, in a perspective of comparison with other languages.

Keywords

• Irony Detection
• Spanish
• Mexican Spanish
• Cuban Span- ish
• Linguistic Analysis
• A ective features

-

Irony is a special gurative device frequently used in speech and everyday life to communicate something that is di erent or contrary to what is literally said [ 29 ]. Irony has di erent functions in communications, which have been studied so far in the context of di erent disciplines such as linguistics, psychology and philosophy. Some of these studies highlight that one of the principal trait of the use of irony consists in emphasizing unexpected occurrences especially amusing the readers [ 1,44 ].

Other studies focus on a very common type of verbal irony, sarcastic irony, which is typically featured by a sharper tone of critique and by the speaker's intent to convey scorn or insult [ 14 ]. The lack of understanding of irony could deceive the reader into thinking that the transmitted message is true. Especially in texts produced by users online, the lack of context and the shortness of the message often make the correct interpretation more di cult for humans and especially for the machine.

In the Big Data era, the automatic understanding of contents online is one of the current purposes of the majority of companies or political organizations that want to analyze the opinions of the users about products, subjects and individuals. The presence of irony, as well as sarcasm, a ects the interpretation of the real opinions of the users, and impedes the correct functioning of systems of sentiment analysis, stance or emotion detection.

For this reason, especially in the campaigns of evaluations of automatic systems of Natural Language Processing (NLP), various shared tasks on irony detection have been proposed. The majority of these competitions provide corpora in English [ 18,45 ], but recently the analysis of irony is extended to other languages such as Italian [ 5,3,12 ] and French [ 6 ]. In the framework of IberLef 2019, for the rst time IroSvA shared task organizers [ 32 ] ask participants to classify ironic and non-ironic texts in three variants of Spanish: Castilian, Mexican and Cuban. The corpus provided for each linguistic variant is a collection of short texts annotated as ironic and non-ironic. These corpora contain also labels of speci c topics referred to current political or social issues discussed online in the chosen geographical areas of Spain, Mexico and Cuba.

Considering the importance of the context in the process of recognition of irony [ 46 ], for each variant we analyzed the impact of topic information and semantic context. Moreover, inspired by previous work on computational models for irony detection [ 21,43,33 ] we explored the role of features related to the a ective information present in the tweets and the psychological response stimulated by the message. On the basis of the analyses in [ 16 ], we took into account also the presence of abusive language. Finally, considering the previous studies about irony detection in other languages, such as English in [ 8 ], we examined also the role of lexical and stylistic features. Comparing our results with the four challenging baselines provided by the organizers, only the system proposed for the Cuban variant overcomes all the baselines. The other two systems overcome only the baseline calculated on majority voting.

A preliminary analysis of errors show some important analogies among the three variants which are in line with the observations emerged in other languages [ 26,11 ]. Although from the rst experiments in this work some slight di erences emerged among these variants, these analogies suggest a new challenge in a multilingual direction.

The paper is organized as follows. The next section summarizes the related work. Section 3 describes the IroSvA dataset and the used approaches focusing on the feature engineering and performed experiments. Section 4 reports the obtained results in the competition. Finally, Section 5 and 6 discuss the results and draw some conclusions, proposing a plan for future analyses. 2

Related Work

In recent years, the importance to recognize the gurative language to understand better the opinion of users have encouraged various researchers to explore texts such as commentaries [ 47 ], reviews [ 38 ] and tweets [ 39 ]. Most of the studies are focused on English language [ 23 ], but recently the need to develop linguistic and computational resources also for other languages incited the NLP community to focus on Italian [ 5,3,12 ], French [ 6 ], Dutch language [ 30 ] and Spanish.

The literature about gurative language detection in the Spanish language is actually limited. The authors of [ 10 ] organized the HAHA shared task in the context of IberEval 2018 about identi cation of humor in Spanish tweets. About satire detection, the authors of [ 13 ] proposed a psychological based approach exploiting news satirical sources on Twitter for Mexican and Castilian variants of Spanish, while the authors of [ 4 ] employed linguistic and semantic features (such as ambiguity and synonyms), sentiment analysis and slang words in a similar collection of tweets from Spain. The studies about gures of speech of irony and sarcasm are really few for the Spanish language. To the best of our knowledge, the study proposed in [ 22 ] is the rst to explore the irony in Spanish tweets considering the sarcasm as a subclass of irony. In particular, they explored word and character level of the texts employing n-grams of words and characters and word embedding, using Support Vector Machines (SVM) and Random Forests as classi ers. Recently, the authors of [ 25 ] explored deeply the function of sarcasm in Spanish dialogues online creating a corpus annotated taking into account the presence of sarcasm and the tone of nastiness.

Since the literature about the identi cation of ironic texts in Spanish is poor, the studies that inspired this work examine the characteristics of irony based on corpora developed on other languages. The larger part of works on this eld exploits various combinations of features and computational techniques. Especially with classic machine learning techniques, some researchers examined the impact of stylistic features [ 8 ], pragmatic symbols (such as hashtags, mentions and emojis in tweets) [ 28,19 ], syntactic patterns [ 20,41 ], sentiment and emotional lexica [ 21 ], semantic context and users information [ 2,24 ]. Recently, technique of deep learning have been exploited also for irony detection [ 50,49 ].

On the basis of these previous studies, we carried out a deep analysis of the three corpora examining the impact of emotions, psychological reactions of readers, topics, semantic contexts, lexical, abusive speech and stylistic features. In the next sections, we describe our approach and the performed analyses that show some analogies and di erences among the ironic characteristic of the texts in the three variants of Spanish. 3

Datasets and Approach Description

The IroSvA shared task is separated on three subtasks: { Subtask A: Irony detection in Spanish tweets from Spain { Subtask B: Irony detection in Spanish tweets from Mexico { Subtask C: Irony detection in Spanish news comments from Cuba For each subtask the organizers provided a collection of short texts about speci c social and politic issues. For the Subtask A and B the datasets contain tweets about the administration of Mexico city, the theory of the at Earth, the exhumation of the dictator Francisco Franco or the actions of some politicians. For the Subtask C, the organizers provided a collection of news comments especially about Internet services and economical problems in Cuba extracted from the news papers journals Cubadebate (http://www.cubadebate.cu/), Granma (http://www.granma.cu/) and OnCubaNews (https://oncubanews.com/).

Each short text (tweet and news comment) is annotated as ironic and nonironic and contains the label of the referred topic. Below, Table 1 describes the composition of the released datasets.

For the classi cation of ironic and non-ironic texts, we employed for each variant a classical supervised machine learning approach exploiting a combination of stylistic, semantic, a ective and lexical features, named SCoMoDI (Spanish Computational Models to Detect Irony). In particular, we used a simple SVM classi er with radial basis function kernel using the following parameters: C = 5 and = 0:01. The kernel and the parameters of the SVM classi er have been set on the basis of various experiments. Considering the imbalanced collection of data, we used the function to balance the weights of the classes provided by Scikit-learn library [ 34 ] for Python. In this section, we describe the features used to build the models of irony detection in the three variants of Spanish. Lexical Features As lexical features we used unigrams of words weighted with TF-IDF (Term Frequency-Inverse Document Frequency) measure. To extract the unigrams, we pre-processed the texts deleting all symbols and numerical characters and selecting words using a tokenizer able to take into account the compound nouns. Finally, in order to weight the words without considering their in ectional morphology, we used the SnowballStemmer for the Spanish language provided by NLTK (Natural Language Toolkit) [ 7 ].

Stylistic Features Taking into account the corpora-based analyses carried out in [ 27 ] for English, French and Italian, we examined the impact of features such as hyperbole expressed by exclamation marks (!, <), ellipsis expressed by dots (:::), questions denoted by question marks (?, >) and quotes expressed by inverted commas (\", `'). Considering the fact that some ironic texts could be characterized by a sarcastic tone against someone, we took into account also the typical symbol of mention in Twitter (@). In the features vector, these features are represented by a simple count of the number of times each item appears in the text.

Semantic Similarity Features In this group we gather semantic contexts and topic information. The semantic contexts of each text are computed calculating the cosine of similarity between the vocabulary of the text and the vocabularies extracted from each group of ironic texts labeled with the same topic. The cosine of similarity is calculated on the basis of pre-trained word embedding of the Spanish Billion Words Corpus (available at http://crscardellino.github. io/SBWCE/) provided by the authors of [ 9 ]. To lead the classi er to capture similarities between texts belonging to the same topic, we extracted the topic distribution of the text considering the number of topics of each subtask (see Table 1). To this purpose, we created the Latent Dirichlet Allocation (LDA) models on the provided training sets using Gensim library [ 37 ] for Python, taking into account also bigrams and trigrams of words. The idea is to gather the texts that talk about the same topic in a similar manner in the same class. A ective Features We considered a ective features exploiting di erent types of lexical resources for capturing di erent facets of a ect.

Emotional Categories To identify the emotions involved in each text, we counted the number of words that belong to emotional lexica, such as the multilingual EmoLex provided by the authors of [ 31 ] and the Spanish Emotion Lexicon (SEL) provided by the authors of [ 42 ] and [ 36 ]. We considered for each variant only the emotions that are relevant for the classi cation. It is surprising that, for all the variants, the most signi cant emotions are negative, such as anger, fear, disgust and sadness.

Dimensional Models of Emotions In order to understand the mental responses to stimuli of ironic texts, we investigated the impact of psychological dimensions such as imagery, activation and pleasantness. Inspired by [ 40,21 ], we use an automatic translated Spanish version of the Dictionary of A ect in Language (DAL) [ 48 ]. From our analysis the dimensions that turned out useful for the classi cation in all the three variants are pleasantness and imagery. Abusive Language Features Inspired by the work of [ 25 ] and considered the relevance of negative emotions, we analyzed also the impact of abusive language counting the words included in the Spanish lexica of derogatory expressions and profanities created by the authors of [ 17 ]. These lists of words prove to be signi cant for the classi cation especially in Mexican and Castilian tweets.

During the experimental phase, we used 5-fold cross validation on the training sets tuning the system on the metric used for the competition: the average of F1-scores of the classes. To study the impact of the described features, we carried out also the ablation feature test and, on the basis of these analyses, we created the models for each variant. The highest F1-scores values obtained with the relevant features are reported in Table 2. 4

Evaluation and Results

The organizers of IroSvA shared task provided four baselines calculated considering di erent representation of the texts. They used: n-grams of words (Word nGrams), word embeddings (W2V) and low dimensionality statistical embedding (LDSE) [ 35 ]. They used also the majority voting (Majority) technique as additional baseline.

As evaluation measure for the ranking the average of F1-score (avg) of all three variants is used. In Table 3 we report the results obtained in the competition compared with the provided baselines.

As we can see, only the model built for the Cuban variant (Subtask C) overcomes slightly all the provided baselines, while the other models overcome only the Majority baseline. This di erence could be due to the textual genre of news comments which do not contain Twitter mentions (@USER), hashtags or emojis. Another in uential factor could be the use of a set of features able to capture characteristics such as unigrams that in general help the text classi cation. 5

Error Analysis and Discussion

Although the di erent genres of texts, analyzing the misclassi ed texts, we noticed that in all the three variants of Spanish the irony is expressed similarly. Actually, we individuated various gures of speech involved in the expression of irony. With the proposed models we aimed at capturing some of them by exploiting textual marks, but the error analysis highlights that it was not su cient.

In particular, we found that hyperbole (examples 1) and ellipsis (examples 2) are expressed more at a semantic level. See, for instance, the following examples from the IroSvA test set where irony was not recognized: (1) Felicidades director muy buena tarifa as se hace.

Congratulations director it is a very good rate, this is how it should be done. (2) Cuando yo sea grande quiero ser como los inventores del paquete. When I grow up, I want to be like the inventors of this o er.

As de ned in [ 29 ], hyperbole is expressed by \exaggerated or extravagant terms used for emphasis and not intended to be understood literally". In fact, the example 1 (from Subtask C) is a clear example of hyperbole that aims to exaggerate positively the actions of someone who is doing bad his work. The importance of hyperbole have been already underlined by [ 15 ] in irony and sarcasm detection.

We found this same phenomenon in the misclassi ed tweets of Subtask A (example 3) and Subtask B (example 4): (4) El pueblo sabio y bueno salio a expresar su voz @lopezobrador The wise and good people came out to express their voice.

In the example 2 (from Subtask C), the irony is expressed by ellipsis. In [ 29 ] ellipsis is de ned as \omission of a word easily supplied". In this news comment, the author wanted to subtract, on purpose, some words containing information that could complete the meaning of the sentences. This subtraction is possible because of the presence of context that give us some intuition about the real meaning of the message. This same phenomenon is found especially in Subtask B (example 5): (5) Sera que le da clases particulares el @brozoxmiswebs It is possible that he teaches him private lessons.

Unfortunately, the simple syntactic features that we used especially for Subtask C are not su cient to capture these more complex puns based on semantic incongruity.

Another common linguistic phenomenon found during the error analysis in Mexican and Cuban variants is the use of apostrophe to stimulate the ironic interpretation of the message. In [ 29 ] the apostrophe is de ned as action of \breaking o a discourse to address some person or personi ed thing either present or absent", as we can see in the following examples extracted from the misclassi ed texts in Subtask B (example 6) and C (example 7): (6) Con todo respeto sen~or presidente, le solicito atentamente que haga una auditor a al @ColegioNal mx cuyos miembros se rayan y donde la ma a de Octavio Paz se ha instalado.

With all due respect, Mr. President, I kindly ask you to do an audit in the @ColegioNal mx whose members bene t and where the ma a of Octavio Paz has settled. (7) Y ahora es que usted se entera que la honestidad paso de moda? And only now you realize that honesty went out of fashion?

Moreover, the rhetorical questions seem to be one of the most used device to express irony in all the variants of Spanish. In [ 29 ] the rhetorical question is de ned as question \which implies an answer but does not give or lead us to expect one". We noticed that although for Subtask B we took into account the presence of question marks, this expedient is not enough to classify correctly irony. Observe the following texts: (8) Cuando pediran perdon Alemania e Italia a los Valencianos , por mandarnos a la Oltra y Grezzi ? https://t.co/MdFup1pbvu When Germany and Italy will apologize to Valencians, for sending us Oltra and Grezzi? https://t.co/MdFup1pbvu (9) Disculpa, sabes si para trabajar en el @Conacyt MX >Debo llevar mi curriculum impreso o depilado? Excuse me, do you know if to work at the @Conacyt MX Should I bring my curriculum printed or shaved? (10) Otra interrogante, por que nadie fuera de Cuba ha denunciado que los cubanos violamos abiertamente los derechos de los productores de esas programaciones?? O sera que el paquete ha venido a ser el primer \embajador" en el restablecimiento de las relaciones?? Another question, why anyone outside of Cuba have not declared that Cubans openly violated the rights of the producers of these programs? Or has the package become the rst \ambassador" in the restoration of relations ??

In the examples 8 (from Subtask A), 9 (from Subtask B) and 10 (from Subtask C) we can see that rhetorical questions involve also other gures of speech such as apostrophe (example 8 and 9) and metaphor (example 9 and 10) which ll the messages with various allusions, making its interpretation more di cult.

These observations suggest that there are some analogies on how Spanish speakers prefer to express irony. Moreover, some of these features have been already explored also in English, French and Italian ironic tweets [ 27 ]. Therefore, it seems that such kind of puns are in general characterizing the expressions of irony independently from the language and genre of the text. 6

Conclusions

In this paper we describe our participation at IroSvA shared task presenting an initial study on irony detection in the Spanish language. In fact, on the basis of the previous works in other languages, we analyzed the impact of various features on the classi cation of Spanish ironic and non-ironic texts. Moreover, the carried out analyses highlight some analogies and slight di erences on the way users express irony in the three proposed variants. We observed that Spanish irony seems to be characterized by speci c puns and especially by negative emotions. However, only Mexican and Castilian ironic tweets seem contain abusive language.

As future work, we planned to examine deeply the role that the topic and the context play in Spanish irony detection, proposing a comparison with a topic-independent approach. Moreover, considering our intuition about the use of similar gures of speech to express irony in di erent languages, we would like to compare multilingual data exploring their real impact in irony detection.

References

1. Attardo , S. : Irony as relevant inappropriateness . Journal of pragmatics 32(6) , 793 { 826 ( 2000 )

2. Bamman , D. , Smith , N.A. : Contextualized sarcasm detection on twitter . In: Ninth International AAAI Conference on Web and Social Media ( 2015 )

3. Barbieri , F. , Basile , V. , Croce , D. , Nissim , M. , Novielli , N. , Patti , V. : Overview of the evalita 2016 sentiment polarity classi cation task . In: Proceedings of EVALITA 2016 ( 2016 )

4. Barbieri , F. , Ronzano , F. , Saggion , H.: Is this tweet satirical? a computational approach for satire detection in spanish . Procesamiento del Lenguaje Natural 55 , 135 { 142 ( 2015 )

5. Basile , V. , Bolioli , A. , Nissim , M. , Patti , V. , Rosso , P. : Overview of the evalita 2014 sentiment polarity classi cation task . In: Proceedings of EVALITA 2014 ( 2014 )

6. Benamara , F. , Grouin , C. , Karoui , J. , Moriceau , V. , Robba , I. : Analyse d'opinion et langage guratif dans des tweets: presentation et resultats du de fouille de textes deft2017 . In: Atelier TALN 2017: De Fouille de Textes (DEFT 2017 ) ( 2017 )

7. Bird , S. , Klein , E. , Loper , E.: Natural Language Processing with Python. O'Reilly Media ( 2009 )

8. Buschmeier , K. , Cimiano , P. , Klinger , R.: An impact analysis of features in a classi cation approach to irony detection in product reviews . In: Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . pp. 42 { 49 ( 2014 )

9. Cardellino , C. : Spanish Billion Words Corpus and Embeddings (March 2016 )

10. Castro , S. , Chiruzzo , L. , Rosa , A. : Overview of the haha task: Humor analysis based on human annotation at ibereval 2018 . In: Proceedings of IberEval 2018 . vol. 2150 . CEUR-WS ( 2018 )

11. Cignarella , A.T. , Bosco , C. , Patti , V. , Lai , M. : Application and analysis of a multilayered scheme for irony on the italian twitter corpus twittiro . In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018) ( 2018 )

12. Cignarella , A.T. , Frenda , S. , Basile , V. , Bosco , C. , Patti , V. , Rosso , P. : Overview of the evalita 2018 task on irony detection in italian tweets (ironita) . In: Proceedings of EVALITA 2018 . vol. 2263 , pp. 1 { 6 . CEUR-WS ( 2018 )

13. Del Pilar Salas-Zarate, M. , Paredes-Valverde , M.A. , Rodriguez-Garcia , M.A. , Valencia-Garc a , R., Alor-Hernandez , G. : Automatic detection of satire in twitter: A psycholinguistic-based approach . Knowledge-Based Systems 128 , 20 { 33 ( 2017 )

14. Du Marsais , C.C. : Traite des tropes: Pour servir d'introduction a la rhetorique et a la logique . Chez la veuve Gaspard Fritsch (1757)

15. Filatova , E.: Irony and sarcasm: Corpus generation and analysis using crowdsourcing . In: Lrec . pp. 392 { 398 . Citeseer ( 2012 )

16. Frenda , S.: The role of sarcasm in hate speech: A multilingual perspective . In: Proceedings of Doctoral Symposium at SEPLN 2018 . CEUR-WS ( 2018 )

17. Frenda , S. , Somnath , B. : Deep analysis in aggressive mexican tweets . In: Proceedings of IberEval 2018 . vol. 2150 , pp. 108 { 113 . CEUR-WS ( 2018 )

18. Ghosh , A. , Li , G. , Veale , T. , Rosso , P. , Shutova , E. , Barnden , J. , Reyes , A. : Semeval-2015 task 11: Sentiment analysis of gurative language in twitter . In: Proceedings of SemEval 2015 . pp. 470 { 478 ( 2015 )

19. Gonzalez-Ibanez , R. , Muresan , S. , Wacholder , N.: Identifying sarcasm in twitter: a closer look . In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers-Volume 2 . pp. 581 { 586 . Association for Computational Linguistics ( 2011 )

20. Hao , Y. , Veale , T. : An ironic st in a velvet glove: Creative mis-representation in the construction of ironic similes . Minds and Machines 20 ( 4 ), 635 { 650 ( 2010 )

21. Hernandez Far as, D.I. , Patti , V. , Rosso , P. : Irony detection in twitter: The role of a ective content . ACM Transactions on Internet Technology (TOIT) 16(3) , 19 ( 2016 )

22. Jasso , G. , Meza , I. : Character and word baselines for irony detection in spanish short texts . Procesamiento del Lenguaje Natural 56 , 41 { 48 ( 2016 )

23. Joshi , A. , Bhattacharyya , P. , Carman , M.J.: Automatic sarcasm detection: A survey . ACM Computing Surveys (CSUR) 50(5) ( 2017 )

24. Joshi , A. , Sharma , V. , Bhattacharyya , P. : Harnessing context incongruity for sarcasm detection . In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2 : Short Papers ) . vol. 2 , pp. 757 { 762 ( 2015 )

25. Justo , R. , Alcaide , J.M. , Torres , M.I. , Walker , M. : Detection of sarcasm and nastiness: New resources for spanish language . Cognitive Computation 10 ( 6 ), 1135 { 1151 ( 2018 )

26. Karoui , J. , Benamara , F. , Moriceau , V. , Aussenac-Gilles , N. , Hadrich Belguith , L. : Towards a contextual pragmatic model to detect irony in tweets . In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguisticsand the 7th International Joint Conference on Natural Language Processing . pp. 644 { 650 . Association for Computational Linguistics (ACL) ( 2015 )

27. Karoui , J. , Benamara , F. , Moriceau , V. , Patti , V. , Bosco , C. , Aussenac-Gilles , N.: Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study . In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics . vol. 1 , pp. 262 { 272 ( 2017 )

28. Kunneman , F. , Liebrecht , C. , Van Mulken , M. , Van den Bosch, A.: Signaling sarcasm: From hyperbole to hashtag . Information Processing & Management 51 ( 4 ), 500 { 509 ( 2015 )

29. Lanham , R.A. : A Hypertext Handlist of Rhetorical Terms: For Macintosh Computers . University of California Press, Berkeley, CA, USA ( 1996 )

30. Liebrecht , C. , Kunneman , F., van Den Bosch , A.: The perfect solution for detecting sarcasm in tweets# not . In: Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . pp. 29 { 37 . ACL ( 2013 )

31. Mohammad , S.M. , Turney , P.D.: Crowdsourcing a word{emotion association lexicon . Computational Intelligence 29 ( 3 ), 436 { 465 ( 2013 )

32. Ortega-Bueno , R. , Rangel , F. , Hernandez Far as, D.I. , Rosso , P. , Montes- y-Gomez, M. , Medina , J.E. : Overview of the task on irony detection in spanish variants . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019 ), colocated with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2019 ). CEUR-WS.org ( 2019 )

33. Pamungkas , E.W. , Patti , V.: #NonDicevoSulSerio at SemEval -2018 task 3: Exploiting emojis and a ective content for irony detection in English tweets . In: Proceedings of The 12th International Workshop on Semantic Evaluation . pp. 649 { 654 . Association for Computational Linguistics ( 2018 )

34. Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M. , Prettenhofer , P. , Weiss , R. , Dubourg , V. , Vanderplas , J. , Passos , A. , Cournapeau , D. , Brucher , M. , Perrot , M. , Duchesnay , E.: Scikit-learn: Machine learning in Python . Journal of Machine Learning Research 12 , 2825 { 2830 ( 2011 )

35. Rangel , F. , Rosso , P. , Franco-Salvador , M.: A low dimensionality representation for language variety identi cation . In: 17th International Conference on Intelligent Text Processing and Computational Linguistics , CICLing'16 . Springer-Verlag, LNCS(9624) , pp. 156 - 169 ( 2018 )

36. Rangel , I.D. , Guerra , S.S. , Sidorov , G.: Creacion y evaluacion de un diccionario marcado con emociones y ponderado para el espan~ol . Onomazein: Revista de lingu stica, lolog a y traduccion de la Ponti cia Universidad Catolica de Chile 29 , 31 { 46 ( 2014 )

37. Rehurek , R. , Sojka , P. : Software Framework for Topic Modelling with Large Corpora . In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks . pp. 45 { 50 . ELRA, Valletta, Malta (May 2010 )

38. Reyes , A. , Rosso , P. : Making objective decisions from subjective data: Detecting irony in customer reviews . Decision support systems 53(4) , 754 { 760 ( 2012 )

39. Reyes , A. , Rosso , P. , Buscaldi , D. : From humor recognition to irony detection: The gurative language of social media . Data & Knowledge Engineering 74 , 1 { 12 ( 2012 )

40. Reyes , A. , Rosso , P. , Veale , T. : A multidimensional approach for detecting irony in twitter . Language resources and evaluation 47(1) , 239 { 268 ( 2013 )

41. Rilo , E. , Qadir , A. , Surve , P. , De Silva , L. , Gilbert , N. , Huang , R. : Sarcasm as contrast between a positive sentiment and negative situation . In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing . pp. 704 { 714 ( 2013 )

42. Sidorov , G. , Miranda-Jimenez , S. , Viveros-Jimenez , F. , Gelbukh , A. , CastroSanchez , N., Velasquez , F. , D az- Rangel , I. , Suarez-Guerra , S. , Trevino , A. , Gordon , J.: Empirical study of opinion mining in Spanish tweets . MICAI 2012. LNCS 7629 , 1 { 14 ( 2012 )

43. Sulis , E. , Hernandez Far as, D.I. , Rosso , P. , Patti , V. , Ru o, G.: Figurative messages and a ect in Twitter: Di erences between #irony, #sarcasm and #not . Knowledge-Based Systems 108 , 132 { 143 ( 2016 )

44. Utsumi , A. : A uni ed theory of irony and its computational formalization . In: Proceedings of the 16th conference on Computational linguistics . vol. 2 , pp. 962 { 967 . Association for Computational Linguistics ( 1996 )

45. Van Hee , C. , Lefever , E. , Hoste , V. : Semeval-2018 task 3: Irony detection in english tweets . In: Proceedings of The 12th International Workshop on Semantic Evaluation . pp. 39 { 50 ( 2018 )

46. Wallace , B.C. : Computational irony: A survey and new perspectives . Arti cial Intelligence Review 43 ( 4 ), 467 { 483 ( 2015 )

47. Wallace , B.C. , Charniak , E. , et al.: Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment . In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1 : Long Papers ) . vol. 1 , pp. 1035 { 1044 ( 2015 )

48. Whissell , C.M.: The dictionary of a ect in language . In: The measurement of emotions , pp. 113 { 131 . Elsevier ( 1989 )

49. Zhang , M. , Zhang , Y. , Fu , G.: Tweet sarcasm detection using deep neural network . In: Proceedings of COLING 2016, The 26th International Conference on Computational Linguistics: Technical Papers . pp. 2449 { 2460 ( 2016 )

50. Zhang , S. , Zhang , X. , Chan , J. , Rosso , P. : Irony detection via sentiment-based transfer learning . Information Processing & Management 56 ( 5 ), 1633 { 1644 ( 2019 )