This paper describes our approach to addressing the Profiling Irony and Stereotype Spreaders on Twitter (IROSTEREO) 2022 shared task. For classifying the users, we consider the number of ironic tweets they posted. The ironic content in a given tweet is determined according to an irony detection model which mainly relies on affective information. Different traditional classifiers were evaluated being the Random Forest the one with the highest performance. According to the official results, our system obtained a 0.933 accuracy rate. Additionally, a sub-task on stance detection focused in ironic users was also organized. For participating in this sub-task, we evaluated the same set of features than in the main task, obtaining the first place in the official ranking.
Nowadays, social media platforms have become one of the main communication channels. People share ideas, information, opinions, and judgments about a wide variety of topics from general events to personal experiences. Such content has been used for different research purposes ranging from sentiment analysis to author profiling. Considering the latter, different aspects like gender, age, personality, native language, political ideology, among others [1]. Recently, interest in profiling authors who use social media to achieve particular communication purposes has grown. Some shared tasks have been organized to profile users spreading fake news [2] in 2020, hate speech [3] in 2021, and this year irony and stereotypes [4]. IROSTEREO aims to determine whether or not an author spreads Irony and Stereotypes considering a set of tweets posted by him/her. Then, the authors must be classified as ironic or not depending on the number of ironic tweets he/she has. In addition, a subtask on Stance Detection is also proposed aiming to identify if an ironic author is in favour or against the target of a given tweet.
Most of the time, people only have an intuitive definition of what irony is. Thus, dealing with this kind of figurative language device from a computational linguistics perspective is an ongoing and challenging task. For some natural language processing areas like sentiment analysis and human-computer interaction, irony detection is a very related task that could help avoid misinterpreting ironic statements as literal.
Irony is a concept difficult to define which serves to express opinions in an indirect way. It has been considered a linguistic device where the speaker intends to communicate the opposite meaning of what is literally said [5]. Irony serves to express an evaluative judgment towards a particular target [6] and/or to reveal the speaker's position (approval or disapproval) on the result of something [7]. In addition, when Irony involves a negative evaluation towards a particular target it is considered as Sarcasm [8,9].
Data from different social media platforms such as Amazon reviews, Reddit, and mainly Twitter has been used for research purposes, being Twitter the most widely exploited [10]. Irony detection has been addressed as a text classification task by using different perspectives like textual-based features [11,12,13], information regarding the context surrounding the comments [14,15,16], and deep learning-based methods like word-embeddings, convolutional neural networks, and transformers [17,18,19,20,21]. Some works have also considered irony detection as a class imbalance problem [22,23,24] due to the inherent data skew on the presence of irony in social media. Research considering irony and profiling information is scarce. In [25], a Spanish dataset collected from Facebook labeled with emotions, irony, and the author's gender is described.
Ironic utterances are very related to the expression of feelings, emotions, and evaluations (often in an indirect way) towards a particular target. Research has been done on the role of affect in the presence of irony [6,26]. We are interested in assessing the role of affective-based information in profiling ironic users. For determining whether or not a given user can be profiled as being ironic, we decided to take into account the number of ironic tweets published by her/him. First, we classify each tweet per user as ironic or not by using emotIDM (for more details see Section 2.1). In this model, a wide set of resources covering different facets of affect from sentiment to finer-grained emotions is exploited for identifying the presence of irony. Then, depending on the number of ironic tweets automatically identified, a given user is labeled as "ironic" or not. In the following paragraphs, we describe in detail the proposed approach.
In order to determine whether or not a given tweet is ironic, we decided to take advantage of an irony detection model which relies mainly on affective information, i.e., emotIDM [27]. This model was evaluated over a set of corpora in the state-of-the-art achieving higher results than in the literature and validating the usefulness of affect-related information for detecting ironic content in tweets. emotIDM comprises a total of 78 features distributed in three different groups for representing a tweet: (i) Structural. Aspects like punctuation marks, length of words and length of chars, part-ofspeech labels, Twitter marks (i.e., hashtags, mentions, etc.), semantic similarity between the words composing a given tweet, among others are considered. (ii) Sentiment. Different facets of sentiment are considered from an overall value in terms of how many positive and negative words a given tweet contains, to a polarity degree in numerical terms depending on the words composing a tweet. A wide range of English lexical resources were exploited like: AFINN [28], Hu&Liu [29], and SentiWordNet [30], among others. (iii) Emotions. With the intention of considering as much information regarding emotions as possible, the main theories in the nature of emotions are comprised in emotIDM: a) Categorical model where the emotions are associated to labels such as "anger", "fear", "joy", "surprise", "disgust", etc. by means of lexical resources like EmoLex [31], EmoSenticNet [32], and LIWC [33]; and b) Dimensional model where emotions are associated to its position in a space of independent dimensions like "activation", "pleasantness", "imagery", etc., by using ANEW [34], Dictionary of Affect in Language [35], and SenticNet [36].
To classify a given user as ironic or non-ironic, we decided to consider an ironying degree, i.e., the number of ironic tweets posted by him/her. The tweets are labeled according to emotIDM. For doing so, we evaluate two different approaches. The first approach (denoted as majority) was to assign a user as ironic considering the number of ironic (henceforth numIronic) and non-ironic tweets (henceforth numNonIronic) he/she has, then the final decision was made according to:
if numIronic > numNonIronic: the user is labeled as userIronic otherwise:
the user is labeled as userNonIronic
In the second approach, we decided to calculate the ironying degree as a threshold (denoted a iD) which was determined according to the average of how many ironic tweets each user in the training set has. By taking advantage of the iD values and the numIronic from each user, the following criterion1 was used to classify the users:
the user is labeled as userIronic otherwise:
the user is labeled as userNonIronic
Participating teams in IROSTEREO were provided with training and test subsets of data. The former is composed by 420 users equally distributed in the two classes: ironic and non-ironic.
For each user a total of 200 tweets are available. Concerning to the test partition, the aim is to classify a total of 180 users by using the same amount of tweets than in the training data. The official evaluation metric is the Accuracy.
As mentioned before, in order to determine whether or not a given user is ironic, we rely on the labels assigned by emotIDM for the tweets posted by her. Then, assessing its performance for doing this task is very important. In this sense, binary classification experiments with a 5-fold cross-validation setting were carried out. The Scikit-learn implementation of traditional classifiers such as Support Vector Machine (SVM), Decision Tree (DT), k-Nearest Neighbors (kNN, with values of 3, 5, and 7 for 𝑘), and Random Forest (RF) with default parameters was used. For experimental purposes, the official training data was splitted into two subsets: train and val in order to determine the ironying degree.
First, we decided to assess the performance of emotIDM to categorize the tweets as a standard irony detection setting. We evaluated different configurations with respect to the group of features used, for participating in the shared task we only consider a total of 60 features concerning to the Sentiment and Emotions described in Section 2.1. In these experiments, all tweets were merged as a single dataset without distinction of belonging to its corresponding author. We assume that all tweets belonging to authors labeled as ironic/non-ironic have the same class at tweet level. Table 1 shows the obtained results in terms of Accuracy for both partitions train and val. As it can be observed, the results are almost the same among classifiers concerning the data partition. The best classification rate was obtained by the RF in both cases. It is interesting to note that, unlike [27] the highest classification rate was not obtained with DT, however, RF was not considered in the evaluation setting described. Notwithstanding, in [37] the best classification performance was achieved by RF for identifying ironic tweets. Both approaches also exploited affective information for detecting irony on Twitter. In our approach, the criterion to classify users as ironic or not is the ironying degree. As mentioned before, one way to obtain such a value (denoted as iD) is to determine to how many ironic tweets each user has in the train partition according to the classification obtained from emotIDM. Table 2 shows the obtained values. As it can be observed, the iD value is very similar across the classifiers and it represents practically half of the available tweets per user. Finally, considering only those users in the val subset, we labeled all the tweets for each user and calculate her/his ironying degree obtained by either criteria: majority and iD to determine whether or not each user is ironic. Table 3 shows the obtained results of classifying the users in the val subset. An improvement in terms of Accuracy was observed in all classifiers when the iD is used as a decision criterion with respect to use the difference between ironic and non-ironic tweets.
Obtained results of labeling users as ironic or non-ironic according to the number of ironic tweets determined by emotIDM on the val subset during the development phase. For participating in the shared task we chose to exploit emotIDM together with a Random Forest to determine an iD threshold. Then, for each user in the official test set provided we labeled each tweet with emotIDM and calculate his/her ironying degree. According to the official results, we ranked in the 32 𝑡ℎ position up to 64 𝑡ℎ with an accuracy rate of 0.933. It is important to mention that, the difference in comparison with the best ranked approach is of 0.0611. Besides, the proposed method achieves higher results than three out of the four baselines considered by the organizers.
We have the intuition that most of the mislabeled users are due to the errors provoked by the emotIDM model. Concerning the official training data, we decided to analyze the obtained results in the val partition. First, we identified that for some users labeled as nonironic (being ironic in the golden label) the errors could be due to very small differences in terms of the number of ironic tweets determined by emotIDM. The same phenomenon occurs in the opposite way. With respect to the correctly classified users, in the case of the ironic ones, we observed that there are some cases where the number or ironic tweets identified represents more than the 60% of the available samples. Probably, a higher threshold could help to improve the performance in identifying ironic users. In the case of the nonironic, some instances having less than 10% of ironic tweets were identified. We hypothesise that, enriching the subset of emotIDM used for participating in the shared task with features capturing stylistic cues could help to capture ironic profiles.
In order to participate in the subtask dedicated to identify if an ironic author is INFAVOR or AGAINST the target of a given tweet, we decided to assess the performance of emotIDM for this challenging task. Affective-based information for dealing with this task has been already evaluated in the state-of-the-art [38]. The criterion to determine the stance of an author is similar to the majority one used in the profiling task, the difference is that we classify each tweet as INFAVOR or AGAINST instead of in terms of irony, and then we count how many tweets are for each class, the majority one is then assigned. Organizers provided with 140 users (for each of them a total of 200 tweets are available) distributed as 94 labeled as AGAINST and 46 as INFAVOR.
Attempting to compensate class imbalanced towards the AGAINST class, we decided to apply the random over-sampling (ROS) implementation in Scikit-learn with default parameters. The same set of classifiers mentioned before was used. Given the fact that, emotIDM have not been evaluated before for stance detection, we experimented with all the features at the same time (allFeatures), and by separating them by groups as mentioned in Section 2.1. Table 4 shows the obtained results over the training data considering two data settings Original and ROS, and with two groups of features showing the best performance. These experiments were performed in a five fold-cross validation setting, in each fold, ROS was applied only for the training partition while the test partition was left untouched. The official metric in this sub-task is the Macro F-score. As it can be observed, the kNN classifier reaches the highest results. Besides, the use of ROS has a positive impact in the performance with both configurations. For participating in the shared task, we submitted the four configurations showing the best performance during the development phase. Table 5 shows the official results obtained in the shared task. Considering the official ranking with the configuration composed by "Structural + 3NN " we ranked in the first position among the 14 submissions in the shared task. One more time, our best-ranking proposal achieves higher results than two of the three the baselines established by the organizers. Interestingly, the Structural subset of features was not used in the task concerning ironic users. The second best result in the shared task was achieved by using all the features in emotIDM, which could serve as a starting point to further investigate the usefulness of affective-based information for stance detection.
In this paper, we describe our participation in IROSTEREO 2022. We propose an approach to classify users as ironic or not depending on the amount of potentially ironic tweets he/she posted. To identify the presence of irony, we took advantage of a wide range of lexical resources comprising different facets of affective information. Our system reached a 0.933 accuracy rate according to the official results. We consider that our approach obtained competitive results despite its simplicity, which could validate the usefulness of considering the role of affect for detecting irony in social media. Our model reaches a higher performance than most of the baselines, one of them using deep learning approach. With respect to the stance detection subtask, we ranked at the first position according to the official results. In both subtasks, the baseline outperforming our proposal is the LDSE (Low-Dimensionality Statistical Embedding) which considers the probability of distribution of the occurrence of terms for text representation. Our approach is not directly using term-based information for text representation. As future work, it could be interesting to enhance the proposed approach with a deep learning-based classification schema. Moreover, further analysis of the correctly classified instances could be interesting, particularly in those where the stereotypes were spreading. Furthermore, an analysis of the performance of emotIDM for dealing with stance detection is also an interesting research direction.