Irony in text allows expressing implicit negative opinions, using a figurative and humorous language. IroSvA is a challenging task proposed this year that allows the evaluation of Irony Detection algorithms in the analysis of short texts for three Spanish language variants (Cuban, Spain and Mexican). Our proposal focuses on the study of three different representations of textual information and similarity measures using a weighted combination of these representations. We use an adaptation of the impostors method for classifying texts in Ironic or Non ironic. We consider the non-ironic texts of the training dataset as the list of impostors. The results achieved are encouraging and the best were obtained for the cuban variant.
Irony is a fundamental rhetorical device. It is a uniquely human mode of communication, curious in that the speaker says something other than what he or she intends [16]. Irony is an active part in the speech of users on the web, when it comes to expressing their opinions (blogs, forums, social networks, specialized sites). Hence the importance of its computational detection for the analysis of data by companies with access to data generated by these or entities with an interest in sentiment analysis, among others. The detection of irony is defined as "a set of characteristics and techniques that allow you to decide whether a text is ironic or not" [12]. Another definition is "Irony detection is an interesting machine learning problem, because, in contrast to most text classifications tasks, it requires a semantics that cannot be inferred directly from word counts over documents alone" [16]. As such, modeling irony has a large potential for applications in various research areas, including text mining, author profiling, detecting online harassment and, perhaps one of the most investigated applications at present, automatic sentiment analysis. [4]. Therefore, detecting irony involves the most challenging areas of natural language processing. Research in user-generated content is a very challenging task. Despite the social media texts convey an invaluable source of information, they are difficult to process because they are noisy, informal, with little context, and plenty of grammatical mistakes [3]. In the task of detecting irony, although in its beginnings, the English language [14,1,3] was the most studied. There are already works in other languages such as Dutch [8], Catalan [9], Arabic [7] and still few are considered in Spanish [12,6]. SemEval 2018 Shared Task [5] on "irony detection in tweets" and Ironita 2018 [2] were the first tasks in irony detection. In SemEval 2018 "The systems that were submitted represent a variety of neural-network-based approaches (i.e. CNNs, RNNs and (bi-)LSTMs) exploiting word and character embedding as well as handcrafted features. Other popular classification algorithms include Support Vector Machines, Maximum Entropy, Random Forest, and Naive Bayes. While most approaches were based on one algorithm, some participants experimented with ensemble learners (e.g. SVM + LR, CNN + bi-LSTM, stacked LSTMs), implemented a voting system or built a cascaded architecture (for Task B) that first distinguished ironic from nonironic tweets and subsequently differentiated between the fine-grained irony categories." [5]. IroSvA (Irony Detection in Spanish Variants) is the first shared task fully dedicated to identify the presence of irony in short messages (tweets and news comments) written in Spanish. [10]. The task will be structured into three subtasks, each one of these, for predicting whether messages are ironic or not, in one of the three Spanish variants. The three subtasks aim to the same goal: participants should determine whether a message is ironic or not according to specified context. Recent advances in detection of irony have shown that the supervised classification methodology with a great extent of feature engineering produces satisfactory indicators for irony or sarcasm. This methodology has been tested in short texts, such as product reviews, news commentaries and tweets [6]. Supervised Classification focused in determine the class of an object based on a set of known objects grouped by class.
Our approach is based on the representation of the text using three different vectors of features. For classifying a new text in the Ironic or Non-ironic class, we used the proposed General Impostor Method (GIM) of [15], but with a simplified variation of the Impostor Method (IM) presented by [15]. The similarity between texts is defined by a weighted similarity considering the three vector representation. In the next sections we explain the representation, the weighted similarity and the GIM. The IM uses a set of Non-ironic documents related with the Ironic sample and analyze the similarity of an unknown document with the set and the Ironic sample. The unknown document would be Ironic if it is more similar to the Ironic one than a randomly subset of the Non-ironic documents.
A text (document) is modeled as an object with three vector representations, but using some features in common. The idea behind this type of representation is to adjust what representation could impact for different genres of text. One representation, and denoted by R 1 , is based on the classical Bag of Word (BoW) vector and we build this with the tokens extracted from a Natural Language tokenizer or with lemmas extracted using a Natural Language lemmatizer. The second representation, R 2 , is based only on the frequency of Punctuation Sign, considering all punctuation extracted by a Part of Speech Tagger. For the third representation, R 3 , are computed different stylistic features and the frequency of them. Some of the stylistic features are: entire capitalized words (QUE BUENO), character flooding (oooooohhhhhh), repetition of closed exclamation sign (!!!!!!!), etc.
The similarity measures to compare two documents needs to consider all the representations proposed, but also to be flexible, so that we can use only some of them. For that reason we use the following measure.
For the β * similarity function we implemented different similarities proposed in the literature, for example, cosene, jaccard, dice, tanimoto or distance like euclidean or minmax. In the evaluation phase we tested with all of them and used the one that allows us to obtain the best result for each representation. The parameters α, γ, δ, let us give importance to the representation, and if one of the parameters is 0, then, not considering that representation. α + γ + δ = 1.
We used the proposed General Impostor Method (GIM), but not considering in the IM the step of randomly choose a subset of features from the representation of text. The set of impostors S corresponds to the set of non-ironic documents provided in the training dataset. D 1 is the document to be classified and D 2 is an ironic document. ∆ * is a parameter that needs to be adjusted using the training dataset and allows us to determine that D 1 is ironic if it has more similarity with ironic D 2 than a percentage of non-ironic impostors.
Input: (D 1 , D 2 ) A pair of documents. S: a set of impostors documents Output: (ironic) or (non-ironic) 1. Score = 0 2. Repeat k times a. Randomly choose n impostors from S : I 1 , ..., I n b.
The set of Non-ironic texts are considered the impostors texts and for the classification of the evaluation dataset we used all the Non-ironic texts provided in the training dataset. All the parameters were optimized and the best value used for the evaluation phase and resumed in the Evaluation section. Next we present the pseudo-code of IM and GIM.
Input: (D) A document to be classified. Y : (D 1 , ..., D n ) ironic documents Output: (ironic) or (non-ironic) 1. For each pair of documents (D, D i ) a. Run original IM to obtain a similarity binary score S(D, D i ) 2. Score = Average over similarity scores [S(D, D 1 ), ..., S(D, D n )] 3. return (ironic) if Score > θ * ; else (non-ironic)
The evaluation was executed for the three variants of Spanish language presented by the task, and for each of them we needed to optimize the parameters of the GIM and the weighted similarity measure. For that purpose we run a 10 cross fold validation over the training dataset provided. The range of the parameters evaluated was varied and the best parameter chosen is illustrated in the Table 1(parameters of UO-run2). The structure and data distribution for the training and test datasets were presented by [10], and also the metrics used for evaluation. The similarity between documents was calculated considering the euclidean distance function for each of the three representation, because it gets the best results in the cross validation evaluation over the rest of the comparison functions implemented.
In the Table 2, we present in row 1 and row 2 the results achieved by the two runs sent to the task, and all the baselines exposed by the organizers, and the highest value obtained by [13]. The main difference of UO-run1 and UO-run2 is that in run1 the BoW representation takes as features the token extracted by the Freeling [11] NLP tokenizer and in run2 the BoW representation considers as features the lemma extracted by the FreeLing lematizer. Also the parameters ∆ and θ were 0.5 for UO-run1. Our best result was achieved for UO-run2, and for the cuban variant, also similar to those of two of the baselines. It is important to notice that with lemma representation the results were always better than the representation based only on lexical tokens, because in the first one we reduced the lexical variety of words referring to the same lemma.
In general, based on the results achieved in the cross validation from the training dataset of the Spain and Mexican dataset (spanish tweets), the R 3 representation gets the worse results and this is based on the stylistic variety between ironic tweets, and also the similarity in stylistic features between ironic and non-ironic tweets. As future directions, we will introduce representations based on Word nGrams and feature selection methods based on the importance of features by class.