Language is, without doubt, one of the most creative skills among all mankind's cleverness. It is an extraordinary powerful machinery based on an idea of ingenious simplicity which allows us to compose out of twenty-five to almost forty sounds (e.g. Romance languages and English) an infinite variety of expressions. Beyond straightforward phonemes mix (grapheme in case of writing), which are constrained by lexical, syntactic and semantic rules, such expressions allow us to disclose to others its whole inside. Language does not only provide a straightforward way to communicate with each other by direct speech/writing, but also it enables indirect communication through creative and figurative language devices.
Irony is one of the most pervasive figurative device used in everyday communication and in social media platforms. Irony 1 implies the use of words that mean the opposite of what is really intended [1]. Usually, people use ironic speech/writing to express negative "private states"(sentiment, opinions, attitudes, beliefs, etc.) where the positive surface meaning differs from the implied one. This linguistic shift in meaning produced by ironic language endows humans with a valuable resource to explore creativity in language and semantics. However, it simultaneously provides a tool that can indirectly and subtly mask language of hatred, offence and discrimination towards specific individuals or social groups. The work introduced by [2] addressed the issue of the hurtfulness of sarcasm in the content of social media. The authors found that ironic expressions of irony involve very negative emotions and sarcastic messages tend to be expressed with a more hurtful language, revealing the aggressive intention of the author towards the targeted victim.
In the context of interpersonal communication, it was introduced the problem of irony bias [3,4,5]. The authors investigated the role of verbal irony in the communication and maintenance of social stereotypes. They observed that irony is found more appropriated in situations in which stereotypes are violated than in situations in which social stereotypes are confirmed. A biased use of irony contributes to social stereotyping and may increase prejudice against minority groups. For that reason, it is crucial to detect, and if it is possible, contrast the diffusion of abusive, discriminatory, and stereotypic language also when it is disguised by figurative devices like irony and sarcasm.
Having previously focused on hate speech spreaders [6], at PAN'22 we have addressed the problem of profiling irony and stereotypes spreaders in social media, more specifically on Twitter. Special emphasis was given to those authors that employ irony to spread stereotypes, for instance, towards women, immigrants or the LGTB+ community. The goal is to classify authors as ironic or not, depending on their number of tweets with ironic content. This will allow for identifying possible stereotype spreaders on Twitter, as a first step towards preventing it. Our hypothesis is that users who do not spread irony and stereotypes may have a set of different characteristics compared to users who do. For example, they may use different linguistic patterns, writing style or affective information when they share posts compared to hate speech spreaders.
The remainder of this paper is organized as follows. Section 2 covers the state of the art on irony detection, author profiling, and stereotypes in language. Section 3 describes the corpus and the evaluation measures, and Section 4 presents the approaches submitted by the participants. Section 5 discusses the results achieved by the participants. Section 6 draws on analyses on the training dataset. Section 7 is devoted to the subtask of profiling stereotype stance at ironic author level. Finally, Section 8 presents the conclusions.
The purpose of this section is to provide the theoretical background. We outline the relevant works in computational irony detection, author profiling and stereotypes in language.
The problem of computationally irony detection has been investigated from different perspectives. Pioneer works had focused on the role of some surface linguistic features obtained from the text on its own such as n-grams, punctuation marks, part-of-speech tags, and simple syntactic patterns, among others [7,8,9,10,11]. Other works focused on some theoretical aspects of irony, such as unexpectedness, contradiction and opposition. Based on that, several features for capturing semantic ambiguity and polarity contrast have been studied [12,13,14,15]. Similarly, some research agrees on the affective component behind ironic messages. Several approaches have stressed affective information for improving irony detection [16,17,18,19]. Verbal irony is, without any doubt, a pragmatic phenomenon; hence, contextual and extra-linguistic information is crucial for its comprehension. In this direction, information concerning the context surrounding a given message has been used to determine whether a text has an ironic or sarcastic intention [20,21,22,23].
Recently, deep learning based methods have attracted the focus of the research in several NLP tasks, including irony detection. In this direction, in [24] a pre-trained Robustly Optimized BERT Pre-training Approach (RoBERTa) [25] model was used to represent the sentences. After that, these were contextualized using a Recurrent Convolutional Neural Network (RCNN) to address irony and sarcasm detection. The authors in [26] proposed to use a Transformer architecture to contextualize pre-trained word embeddings. Specifically, they contextualized Word2Vec word embeddings, trained with several millions of tweets both for English and Spanish. This strategy, opposite to pre-trained Bidirectional Encoder Representations from Transformers (BERT), allows the system to be trained from in-domain representations using the same robust backbone architecture as BERT. From another point of view, the model introduced in [27], proposed strategies to improve irony detection by transferring knowledge from sentiment resources. For that, the authors proposed three different attentive Long Short Term Memory (attentive-LSTM) approaches that differ in the way of including the sentiment resources, either injecting the sentiment directly to the attention mechanisms or merging the output of different networks specialized on sentiment analysis and irony detection. In [28] an attentive-LSTM model was proposed for irony and satire detection in Spanish. The model takes advantage of three representations learned from, sentence-embedding, the BERT-based model and linguistic features. These representations were used to inform the proposed attentive-LSTM model to improve irony detection. In a similar fashion, in [2] a transformed-based system was introduced. The authors investigated the impact of hurtful and affective features on irony and sarcasm detection in Italian tweets.
From a multilingual point of view, most of the research carried out on irony detection has been done in English. Notwithstanding, there have been some efforts to investigate such figurative language device in other languages such as: Chinese [29], Czech [11], Dutch [10], French [30], Italian [31], Portuguese [7,32], Spanish [33,34], and Arabic [35,36]. Even when the classification of a text as ironic or not has been widely studied from different perspectives, there are no references to computational works that attempt to profile authors who can be considered ironic and utilize this rhetorical figure to spread and perpetuate stereotypes on social media platforms.
Pioneer researchers on author profiling focused on the analysis of blogs and formal texts [37,38], based on Pennebaker's [39] theory. This theory connects the use of the language with the personality traits of the authors.
With the rise of social media, researchers proposed methodologies to profile the authors of posts where the language is more informal [40]. Since then, several approaches have been explored. For instance, [41] approached the age and gender identification problem with a second order representation which relates documents and user profiles. The authors of [42] proposed Low Dimensionality Statistical Embedding (LDSE), a statistical embedding which drastically reduces the dimensionality while seizing the whole vocabulary, and that has been commonly used as baseline in author profiling shared tasks.
Recently, the research community has focused on the usage of emotions and personality traits to address different problems, such as the Emograph graph-based approach enriched with topics and emotions [43]. Furthermore, due to the lack of large annotated corpora to train the new and powerful deep learning methods, the research community is focusing on using little or no training data to address the author profiling task [44].
During the past three editions, at PAN we focused on profiling users who spread harmful information, as well as profiling bots due to their key role in its propagation on Twitter. Concretely, in 2019 the goal was discriminating bots from humans [45], in 2020 identifying possible fake news spreaders [46], and in 2021 the focus was on profiling potential hate speech spreaders [6]. This year we aim at identifying potential spreaders of ironic contents, mainly when referring to special social categories or stereotypes.
Stereotypes are generally defined as a set of widespread beliefs that are associated with a group category [47]. This is the theoretical assumption of the Stereotype Content Model (SCM) [48] very popular in computational linguistic approaches to stereotype detection. SCM states that two dimensions persist in social cognition when people are making sense of individuals or groups: perceived warmth (trustworthiness, friendliness) and competence (capability, assertiveness). Supporting this approach, most of the attempts to study stereotypes in Computational Linguistics have focused on a particular target group such as gender [49,50], ethnic minorities [51], religion [52], immigrants [53,54,55] and age [56]. Most of them use a word embeddings representation and rely on the association of attributes to a social group. The common goal has been to identify which stereotypical beliefs are associated to each particular group, introducing bias in large language models, which are increasingly used in AI applications.
However, what most research overlooks is the fact that before of this description of group in terms of concrete traits, a previous homogenization of the group must be done. This homogenization of diversity is at the base of the over-generalization that allows the success of the stereotypical reasoning. As Lippman argued in his seminal work [57] about stereotypes, this cognitive process that disregards the variability of the real world occurs because "we do not first see and then define, we define first and then see". The original idea of stereotype, as Lippmann defines it, relies more on a cognitive process that disregards diversity into a group or into a particular event than in the content itself of a particular stereotype.
Following this idea, we start from the premise that a vision of the word in terms of social categories is previous to the use of a stereotype, that is to say, is previous to this cognitive process that assumes that a singular individual has some characteristics simply based on their perceived membership in the group. In this sense, in this task of author profiling, we have tried to operationalize the idea that some people share a vision of the world that intensively uses social categories to describe and explain reality, a prejudiced mentality that systematically privileges a worldview in terms of homogenous social groups and undervalues the internal diversity of any social group.
Few works have approached the study of stereotypes affecting more than a target group. The authors in [58] create StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. They contrast both stereotypical bias and language modelling ability of popular models like BERT, GPT2, RoBERTa, and XLNET, showing that these models exhibit strong stereotypical biases. Recently, Sap and colleagues [59] approach also the presence of several target groups in the Social Bias Frame, a new conceptual formalism that aims to model the pragmatic frames in which people project social bias and stereotypes onto others. To support this research, they developed the Social Bias Inference Corpus (SBIC) with 150,000 structured annotations of social media posts covering 34,000 implications about social groups. For example, in front of a sentence as "If cameras do really add ten pounds, do Africans really exist?", annotators from Amazon Mechanical Turk indicate whether or not: (i) the post is offensive, (ii) the intent is to offend, and (iii) it contains sexual content. Only if annotators indicate potential offensiveness they answer the group implication question: who is referred to/targeted by this post? Two possible answers were: (i) yes, this could be offensive to a group and (ii) no, this is just an insult to an individual or a non-identity-related group of people. If the post targets or references a demographic group, annotators select or write which group is referenced. For each selected group, they then write two to four stereotypes that are used in this post; for the given example, annotators write as stereotype: "Africans are all starving". Finally, workers are asked whether they think the speaker is part of one of the minority groups referenced by the post. From 16,739 instances in SBIC, 8,167 refer to a group of people in the field of "target minority". On the basis of this work we have constructed the IROSTEREO corpus as it is explained in Section 3.1.
The purpose of this section is to introduce the technical background. We outline the construction of the corpus, introduce the performance measures and baselines, and describe the software submissions.
In this section, we describe the methodology followed to build the corpus, introduce the taxonomy and the stereotype categories, explain the retrieval of the tweets and the annotation process, and finally give some statistics of the obtained corpus.
To build the IROSTEREO corpus we examine the "target minority" field of the SBIC by [59] which has 150,000 structured annotations of social media posts covering 34,000 implications about social groups. We identify 600 unique labels that could be considered a social category in SBIC. We define a social category following a long tradition of research in social psychology [60] [61] which considers that a social group exist when two or more persons define themselves as members of the group and when their existence is recognised by at least one other person. Sap et al. [59] classify the groups referenced in seven categories: (1) body ( 2) culture ( 3) disabled (4) gender ( 5) race (6) social and ( 7) victims.
In order to focus specifically on stereotypes as the expression of a prejudice against certain social categories that are often the object of an ironic and hurtful discourse, we create a more granular taxonomy to classify the 600 labels in 17 categories: (1) national majority groups, (2) illness/health groups, (3) age and role family groups, (4) victims, (5) political groups, (6) ethnic/racial minorities, (7) immigration/national minorities (8) professional and class groups, (9) sexual orientation groups, (10) women, (11) physical appearance groups, (12) religious groups, (13) style of life groups, ( 14) non-normative behaviour groups, (15) man/male groups, ( 16) minorities expressed in generic terms and (17) white people. As keywords to retrieve the tweets, we use the labels associated to groups only from categories 5 to 14 of the taxonomy.
The Twitter API was used to retrieve tweets with two conditions: (i) tweets that contain the hashtag irony or sarcasm and at least one of the labels included in categories 5 to 14 of the taxonomy and (ii) the same labels about social groups but without irony or sarcasm. Users with more cases in classes (i) and (ii) were identified and the tweets that accomplish these two conditions were downloaded. The annotators had to identify ironic tweets and tweets that use stereotypes among this set of users. To identify irony, the annotators were asked to mark the tweets where the user "express the opposite of what was saying as a disguised mockery". If a user had more than five ironic tweets, it was labelled as ironic.
To identify the use of stereotypes, annotators were asked to check if the social categories present in the tweets were used to refer to a social group by associating them with a homogenising image of the category. For example, they talk about gays or Muslims in general, as if they were all similar, and could be well described with that word. If a user had more than five tweets containing a stereotyped image of a group, the user was labelled as a user who utilises stereotypes. Positive examples of classes 1 (users that express irony without stereotypes), 2 (non-ironic users that use stereotypes) and 3 (users that express irony and use stereotypes) were selected and 200 tweets from their timeline were downloaded. To find the non-ironic and non-stereotype class (4) the lexicon used in the three previous classes was analysed in order to reduce topic bias. Moreover, tweets should not contain the labels of social categories associated to stereotypes.
The annotation process was carried in two steps. During the first one, data were annotated by two independent annotators. The inter-annotator agreement (IAA) between the first two annotators was 0.7093. During the second one, those instances where a disagreement exists, we asked for a third annotation to solve it. Moreover, the second annotation was done also to check that class 4 does not contain irony samples.
Table 1 presents the statistics of the corpus that consists of 600 authors for English language, completely balanced between the two classes (ironic and non-ironic), and with a 66/33 balance between users employing stereotypes or not for each class. For each author, we retrieved via the Twitter API their timeline and sampled 200 tweets. We have split the corpus into training and test sets, following a proportion of 70/30 for training and testing respectively.
Since the dataset is completely balanced for the two target classes, ironic vs. non-ironic, we have used the accuracy measure and ranked the performance of the systems by that metric.
As baselines to compare the performance of the participants with, we have selected:
• RF + char 2-grams character 𝑏𝑖𝑔𝑟𝑎𝑚𝑠 and Random Forest classifier.
• LR + word 1-grams Bag of Words (BOW) with Logistic Regression classifier.
• LSTM+Bert-encoding We represent each tweet in the profile utilising pretrained Bert-base model. Later, we fed an LSTM with these vectors as input.
• LDSE [42]. This method represents documents on the basis of the probability distribution of occurrence of their words in the different classes. The key concept of LDSE is a weight, representing the probability of a term to belong to one of the different categories: irony vs no-irony spreader. The distribution of weights for a given document should be closer to the weights of its corresponding category.
Similar to previous year2 , we asked for software submissions. Within software submissions, participants submitted executables of their author profiling software instead of just the output of their software on a given test set. For the software submissions, the TIRA experimentation platform was employed [62,63], which renders the handling of software submissions at scale as simple as handling run submissions. Using TIRA, participants deploy their software on virtual machines at our site, which allows us to keep them in a running state [64].
This year, 65 teams participated in the author profiling shared task and 33 of them submitted the notebook paper. We analyse their approaches from three perspectives: preprocessing, features used to represent the authors' texts and classification approaches.
In order to prevent bias towards some URLs, user mentions or hashtags, the corpus was provided with these elements already masked. In the same vein, some participants cleaned other Twitterspecific elements such as RT, VIA, and FAV 3 [74,73,65], or stemmed or lemmatised the terms [82]. Some users also removed infrequent terms or meaningless ones [65,79,70]. The authors in [83] carried out a 𝑋 2 test which combined with Pointwise Mutual Information (PMI) and TF-IDF was used to select the most contributing words to the representation, or the authors in [75] who added the labels (I/NI) to the end of each tweet. Finally, the authors in [82] used GloVe as a pre-trained embedding matrix to filter out features.
The participants have used a high variety of different features and their combinations, albeit we can group them into the following main groups: (i) 𝑛-grams; (ii) stylistics; (iii) personality and emotions; and iv) deep learning-based such as embeddings and transformers.
Regarding 𝑛-grams, a high variety of them have been used by several authors, in most of the cases in combination with other representations. Mainly, character 𝑛-grams [81,74], word 𝑛-grams [83,72,81,74] (sometimes weighted with TF-IDF [84]), and syntactic 𝑛-grams (e.g. POS) [74].
The authors of [74] have combined different types of 𝑛-grams with sentiment and emotions, as well as hateful content (aggressive, hateful and targeted), while the authors of [84] have combined stylistic features such as average vocabulary size, average number of tokens, average tweet length, average number of hashtags, mentions and URLs, average number of emojis, or the ratio between lowercased and uppercased words, the LiX score, TF-IDF unigrams, TF-IDF profanity, TF-IDF emojis, Part of Speech (POS) tags count, sentiment analysis and punctuation signs.
Different transformers have been also widely used to extract features. For instance, BERT [85,73,79,75,86,66,67,69,71], SBERT [87], BERTweet [70] or combining them with other feature extraction methods. For example, BERT and Twitter RoBERTa with LM HateXPlain [88] finetuned with the HatEval dataset [89], SBERT with emojis [87], psychometrics, emotions and irony with SBERT [90], BERT with TF-IDF 𝑛-grams [91,92], or SBERT with graph-based and one-hot embeddings [65].
The authors of [93] have combined different stylometric features such as lexicon-based, social media jargon or POS with static embeddings (FastText) and contextual embeddings obtained with BERT and RoBERTa. Similarly, the authors of [68] have combined different 𝑛-grams with stylistic features based on lexicons with sentence transformers.
Convolutional Neural Networks (CNNs) have been also used to extract features [78]. The authors of [82] combined a CNN with TF-IDF unigrams and a Bidirectional Gated Recurrent Unit (BiGRU) to represent the authors' texts. The authors of [80] have used a TextVectorizer to extract their features while word embeddings have been used by the authors of [76,94]. The authors of [95] have combined sequence probabilities and 𝑛-grams with GPT2 and DistilGPT2.
The authors of [86] combined a semantic representation obtained with a transformer with punctuation signs and auxiliary words representations. Similarly, the authors of [90] have obtained ironic-, contextual-and psychometric-related features with transformers that had been fine-tuned with datasets annotated with sentiment and emotions from the Kaggle competition 4 .
Finally, the authors of [96] approached the task by identifying irony at individual tweet level. To that end, they have combined three types of features: i) structural features such as punctuation marks, length of words, part-of-speech labels, Twitter marks, semantic similarity, etc.; ii) sentiment words by applying different lexical resources such as AFINN, Hu&Liu, and SentiWordNet; and iii) fine-grained emotions, by means of emotional lexicons such as EmoLex, EmoSenticNet, ANEW, Dictionary of Affect in Language, and SenticNet, among others.
Most participants have used traditional approaches, mainly Random Forest (RF) [72,96,83,84], Logistic Regression (LR) [91,74], Bayes (NB) [95], Multilayer Perceptron (MLP) [73], Gradient Booster Classifier (GBC) [90], or 𝑘-Nearest Neighbours (k-NN) [86].
Ensembles of classifiers have been also used by various authors. For example, Support Vector Machine (SVM) and RF with a hard voting classifier [81], SVM with a Grading Boosting Classifier [90], or SVM, RF and LR with soft-and hard-voting ensemble [79]. Some participants have combined traditional approaches with deep learning ones through stacking ensembles with Logistic Regression with a meta-learner and SVM, Naive Bayes and Decision Trees (DT), together with CNN [77], and a meta-learner with SVM, DT, Naive Bayes and CNN [80].
Deep learning has been widely used to approach this year task, mainly CNNs [85,76,78], Graph Convolutional Neural Networks (GCNN) [65], Linear Feed Forward Networks [87], as well as combinations such as Bidirectional Long Short Term Memory (BiLSTM) and CNN [94] or just fully-connected networks [82,93,89]. Some participants used AutoML (AutoKeras) [70] and AutoGluon [71,69] to automate the selection of the classifier.
With respect to transformer-based approaches, BERT and some of its variants have been the most used ones, usually combined with other approaches. For instance, BERT with Decision Rules [75], BERT with SVM, MLP, Gaussian Naive Bayes and RF [86], BERT with CNN, LSTM and attention layer [92], BERT and DistilBERT with RF and SVM [68], or just several BERT with a voting classifier [66,67].
In this section, we present the results of the shared task, as well the analysis of the most common errors made by the teams.
In Table 2, the overall performance (in terms of accuracy) of the participants is presented. The top-ranked participants approached the task as follows. The overall best result (99.44%) has been obtained by Yu et al. [85] with a BERT feature-based CNN model. The second best result has been achieved by Tahaei et al. [87] with a combination of SBERT and emojis. The two ex aequo third best performing teams, respectively, used a Multilayer Perceptron trained with features extracted from a pre-trained BERT model 5 , and a Random Forest fed with unigrams pre-selected with several techniques such as 𝐶ℎ𝑖 2 , PMI, and TF-IDF with the aim to maximise the probability difference of each feature for each class [83].
Overall accuracy of the submission to the task on profiling irony and stereotypes spreaders on Twitter.
1 wentaoyu [85] 0.9944 2 harshv [87] 0.9778 3 edapal 0.9722 3 ikae [83] 0.9722 5 JoseAGD [93] 0.9667 5 Enrub 0.9667 7 fsolgui 0.9611 7 claugomez [92] 0.9611 9 AngelAso 0.9556 9 alvaro [86] 0.9556 9 xhuang [95] 0.9556 9 toshevska 0.9556 9 tfnribeiro_g [84] 0.9556 14 josejaviercalvo 0.9500 14 taunk [72] 0.9500 14 As can be observed in Figure 1 and Table 3, the results do not follow a normal distribution (𝑝-value = 2.2e-16) when we consider all of them. There are several outliers on the bottom side of the distribution, as can be also seen in Figure 2. When getting rid of the outliers (from the left), the top performing systems do follow the normal distribution (𝑝-value | acc>0.85 = 0.1382), allowing the usage of the 𝑡-student test for the comparison of the significance of their differences. In these regards, the best performing team is not significantly better than the second and third ones (𝑧 𝑐 = 1.3451 and 𝑧 𝑐 = 1.6435). Indeed, statistically significances appear with respect to the fifth best performing team (𝑧 𝑐 = 2.1418).
We have aggregated all the participants' predictions for irony vs non-irony spreaders, except baselines, and plotted the confusion matrix in Figure 3. It can be seen that the error is higher in the case of false positives (from non-irony to irony spreaders): 15.45% vs. 9.18%. This higher number of false positives is something to be investigated further in future research since it may introduce a bias towards the ironic class.
With the aim to study how the authors of the different classes (irony vs non-irony) use the language, in this section, we analyse in detail: (i) the most commonly used topics per class; (ii) the usage of Twitter elements such as the number of words, hashtags, mentions and shared URLs; (iii) their writing style; (iv) the emotions they convey; (v) and their psychographics and
As described in Section 3.1, we collected the users by querying Twitter APIs with a list of keywords associated with irony and social categories, and the annotators labelled the proper usage of ironic language and social categories. However, the keyword-based data collection process might have introduced a bias in the corpus regarding the topics they cover.
The problem of topic bias has been analysed on hate speech corpora. Particularly, in the works [97,98] the authors computed statistical scores in order to determine the correlation between the words and hate speech microblogs. In this work, we perform a two-fold analysis in our corpus similar to the one of [98]: i Determining the set of unique words in each class and analysing how this vocabulary impacts the learning process; ii Determining the set of words that are highly polarized according to the indexes introduced in [98].
We used the Polarized Wiredness Index (PWI) which takes into account how polarized the words are in each class in the corpus (irony and non-irony). PWI compares the relative frequency of a word as it occurs in the subset of a labelled dataset identified by one value of the label against its complement. Let us consider an annotated corpus 𝐶 = {(𝑑 Where 𝑁 𝑙 () and 𝑁 𝑙 ^() represent the frequency of the term 𝑤 𝑗 in the class 𝑙 = 0 and in its complement 𝑙 = 1, respectively. 𝑇 𝑙 and 𝑇 𝑙 ^represent the total count of words in the class 𝑙 (irony) and in its complement (non-irony), respectively.
Regarding the first point, we identified 1,379 words which only appear in one class 6 . In the irony class, we found 334 unique terms, whereas, in the class non-irony, we identified 1,045 unique terms. With the aim of investigating how this vocabulary may impact the learning process, we trained an SVM and RF classifiers considering as features the words in this vocabulary. As a result, the RF model achieved an Acc=0.8763 and the SVM an Acc=0.8817.
Later on, we analyse if even when the words are in both classes, their representativeness in each one is biased. For that, the PWI index was computed for the words in the corpus. Table 4 illustrates the highest-ranking words (25 words) according to their PWI in both classes. :-(, :-),:P, ;), ;-), abound, alarmist, antizionist, appointee, assurance, aws, bama, bhakts, bound, brampton, carb, cathie, cda, conway, corey, darn, desert, djt, dowry, du30, duterte From the first column, it can be noticed that in the Non-Irony class there are high-PWI words related to ethnic (e.g. aboriginals, ados, africans, americas, bipoc, charlottesville, etc.). Looking at the high-PWI words of the second column, the most characteristic words in the Irony class are related to politics (e.g., conway, djt, du30, duterte) and religion (e.g., antizionist, bhakts, brampton, dowry and cathie), but also to some stylistic elements like emojis. In order to analyse how these polarized words may have impacted the learning process, we select the 100 most polarized words in each class. We train a couple of SVM and an RF classifiers, considering as features the words in this reduced vocabulary. As a result, the RF model achieved an Acc= 0.8833 and the SVM an Acc=0.8333. As can be noticed, the models achieve high classification scores. This analysis confirms that these words may have introduced a topic bias. However, we have also found out that stylistic aspects like emojis are also more used in the Irony class, which means that the topic bias might have been introduced by the strategy adopted to collect the data, or may have been also caused by a latent bias induced by the authors themselves.
In this subsection, we analyse the usage of Twitter elements such as hashtags, user mentions and URLs, as well as the average number of words used by the different types of users. The usage of these elements by the authors of the corpus is not normally distributed according to the Kolmogorov-Smirnov test (p<.001 for all the cases, both in the training and test sets). Therefore, we have performed the Mann-Whitney test to compare the corresponding distributions of these variables in the two classes (ironic vs. non-ironic users) in the two sets (training and test). The Mann-Whitney test relies on scores being ranked from lower to higher. Therefore, the group with the lowest mean rank is the group with the greatest number of lower scores and vice versa. For all the variables, the Mann-Whitney test is significant in both sets (see Table 5). In Table 5 we observe that non-ironic users (Mdn= 18.14) employ more number of words than ironic users (Mdn= 12.79). Both classes also differ in the use of typical Twitter elements. As we can see in Figure 4, ironic people use more hashtags, more mentions, and fewer URLs than non-ironic ones.
Statistics about differences between ironic and non-ironic users in the usage of Twitter elements.
To analyse whether there are differences between ironic and non-ironic users in their language style, we apply the POS-tagging FreeLing7 to each tweet of every user. A total of 33 morphological features are used to represent texts. A score of each morphological feature is computed as the percentage of this feature over the total words of the tweet. Then, the average score for each user in every morphological feature is calculated and normalized. With these scores, the categorical vs narrative index used by authors in previous research is calculated. Inspired in the work of Nisbett [99], the categorical versus narrative index is computed as a simple algorithm: nouns + adjectives + prepositions -verbs -adverbs -personal pronouns. Positive values in this index express more categorical style of language and negative values more narrative style. Categorical style is used to express ideas and concepts, whereas narrative is used to tell stories. The scores of this index are not normally distributed according to the Kolmogorov-Smirnov test (p<.001 in training and test data), then we perform the Mann-Whitney Test to compare the distributions of this variable in the two classes: ironic users vs non-ironic users. The Mann-Whitney test is significant in both sets of data (test and training), then we offer the statistics for the entire corpus in Figure 5. As we can see, non-ironic users (Mdn=0.71) utilize significative more than ironic users (Mdn= -0.99; U=15,834; p<.001) a categorical language style. Ironic users utilise more a narrative style. It is interesting to notice that these features are topic-agnostic, and we can conclude that, regardless of the topic, there are significant differences in the way ironic and non-ironic users employ language.
The new Dictionary of Affect in Language [100] is used to test if the ironic and non-ironic users differ in the expression of emotions. The new Dictionary of Affect in Language (DAL for short) is an instrument designed to measure the emotional meaning of words and texts. It compares individual words to a word list of 8,742 words that were originally rated by 200 naïve volunteers along three dimensions: activation (active vs. passive), imaginary (easy vs. difficult to imagine), and pleasantness (unpleasant vs. pleasant).
The scores of the three dimensions are not normally distributed according to the Kolmogorov- Smirnov test (p<.001 for all of them in training and test data), then we perform the Mann-Whitney Test to compare the scores distributions of these three dimensions in the two classes: ironic users vs non-ironic users. The Mann-Whitney test is significant for the three dimensions in both sets of data, then we offer the statistics for the entire corpus (Table 6). As we can see in Figure 6, non-ironic users present higher scores in the three emotional dimensions than ironic ones.
We use the Symanto API 8 to obtain the users' personality type and communication styles [101].
The personality type refers to the way the person behaves in a specific interaction from the emotional vs rational point of view. Regarding communication styles, it is composed of four traits: (i) action-seeking, defined as direct or indirect requests, suggestions, and recommendations that expect action from other people; (ii) fact-oriented, where the user utilises factual and objective statements; (iii) self-revealing, when the users share personal information or experiences; and (iv) information-seeking, defined as direct or indirect questions searching for information. For each of these traits, the Symanto API returns a value between 0 and 1, representing the confidence for the person to belong to some part of the continuous between the two extremes of the trait, for example, somewhere between completely emotional and completely rational.
The data of all these variables are not normally distributed according to the Kolmogorov-Smirnov test (p<.01 for all of them in training and test data, except for information-seeking where the p-value in training is p<.001 and in the test is .09). We perform the Mann-Whitney Test to compare the distributions of these variables in the two classes: ironic users vs non-ironic users in the training and in the test sets. For all variables, except action-seeking styles, the Mann-Whitney test is significant in the training and in the test data. Table 7 illustrates the statistics for the entire corpus. As we can see in Figure 7, ironic users use less the fact-oriented style and more a self-revealing and information-seeking style. In the personality type measure, the non-ironic users are more emotional and less rational than the ironic users.
For creating the IROSTEREO-Stance corpus, we selected those authors that were annotated as ironic and spreaders of stereotypes in IROSTEREO. Later, we performed a third annotation process on this data: for each author, only the tweets marked as ironic and using stereotypes in the IROSTEREO corpus were annotated with their stance. We did not provide any kind of guidelines for the annotation. Instead, we asked the annotators to rely on their own perspectives on whether the tweets are in favour or against the mentioned social category. The overall stance of an author is considered "in-favour" if the majority of the annotated tweets in her profile support the targets; in the other case, it is considered as "against". The overall stance of each ironic author that used stereotypes was initially annotated by two independent annotators. The IAA between the first two annotators was 0.645. Then, for those ironic authors where a disagreement existed, we asked third annotator for another annotation. Finally, a dataset with 58 ironic authors "in-favour" and 142 "against" was obtained. The distribution in training and test is showed in Table 8.
The performance of the systems is evaluated using the macro averaged F1 measure (F_Macro), although we also analyse the F1-measure per class to study more in depth +the behaviour of the systems (F1_A and F1_F for the "against" and "in-favour" class, respectively). We have evaluated three baselines in the profiling stereotype stance subtask:
• RF + char 3-grams character 𝑡𝑟𝑖𝑔𝑟𝑎𝑚𝑠 and Random Forest.
• SVM + word 2-grams 𝑏𝑖𝑔𝑟𝑎𝑚𝑠 of words with Support Vector Machine.
• LDSE method [42] In this subtask, we did not constrain the number of runs that a team could summit and the 7 teams submitted 15 runs in total. All results achieved by each team and the baselines are shown in Table 9. Most of the teams tested on the IROSTEREO-Stance corpus the systems previously submitted to the IROSTEREO task. The models submitted by dirazuherfa's team [96] employed emotIDM, an emotion-based approach, that comprises three groups of features for representing the tweets: (i) structural features: punctuation marks, length of words and chars, part-of-speech labels, Twitter marks (i.e., hashtags, mentions, etc.), and semantic similarity; (ii) sentiment features: an overall value of polarity is calculated in terms of how many positive or negative words a tweet contains (the sentiment intensity of each word was considered); (iii) emotions features: information regarding emotions from several lexicons. Moreover, an oversampling method was applied to address the imbalance in the training set. The runs differ from each other in the subset of features and in the classification model. Run1 and run4 considered all features in emoIDM combined with a 7-NN and 5-NN, respectively. In run2 and run3, only the structural features were considered for training a 5-NN and a 3-NN classifier, respectively. The toshevska's team trained a deep graph convolutional neural network (HinSAGE) to classify user nodes. For that, a heterogeneous graph was created. It comprises three types of nodes: user nodes, tweet nodes, and nodes, and three types of edges: user-tweet, tweet-word, and word-word. The system used by the JoseAGD's team [93] relied on four-faced representations. The feature representations are based on linguistic features from UMUTextStats, non-contextual sentence embeddings from FastText, contextual embeddings from BERT and contextual embeddings from RoBERTa. Later, a fully connected neural network was trained. Run1 submitted by the tamayo's team [89] used a prototype creation strategy for representing the profiles. Firstly, the tweets in the profile are encoded employing a pretrained RoBERTa-based model. Based on these representations, the profile is split into two groups, one where the tweets are strongly related, according to their inner similarity, and another group with more heterogeneous information. The representation of the profile is the sum of the tweet's encoding from the former group. For classifying a new author, a KNN method was applied. In run2 the tweets are encoded using three transformers models: BERT-base, Twitter-RoBERTa-base and LM HateXplain. The profile was modelled using a Spatial Graph Convolutional Neural Network, and a fully connected dense network was used as the classifier. The run1 and run2 submitted by AmitDasRup's team [91] used BERT combined with the TF-IDF representation, and the prediction was made by a logistic regression classifier. The runs differ in the parameters of TF-IDF used to build the vocabulary. The taunk's team [72] represented tweets using Bag of Word and TF-IDF weighting. Later, the profiles were built as the sum of the tweet vectors. Based on this representation of the profiles, several shallow machine learning models were trained. The authors experimented with Random Forest, Support Vector Machines, K Nearest Neighbors, Logistic Regression, and XGBoost. The best result was achieved using the SVM model. Finally, fernanda's team [81] proposed an ensemble method based on a hard voting scheme. Three distinct representations: char n-grams, word n-grams and Out of Vocabulary (OOV) were built. After that, SVM and RF were used as base classifiers, and their prediction was aggregated in the voting schema.
As it can be observed in Table 9 the results achieved by all participants are moderated. Five runs reached an F1_Macro>0.50, and no participants outperformed the LDSE baseline. Also, it can be noticed that the systems had a low performance in the "in-favour" class, whereas high F1 scores are achieved in the class "against". We hypothesize that the three main problems that have been faced by the participating systems are: i) the inherence complexity of profiling the stance of ironic authors that employ stereotypes, ii) the short size of the IROSTEREO-Stance corpus; and iii) the imbalance between "in-favour" and "against" classes which made challenging the learning process. Although the results were quite modest, this task opened a new way to study ironic language to perpetuate stereotypes and constitutes a starting point for profiling authors who frame aggressiveness, toxicity and messages of hatred towards social categories such as immigrants, women and the LGTB+ community, using an implicit way to convey hate speech employing stereotypes.
In this paper, we have presented the results of the 10th International Author Profiling Shared Task at PAN 2022, hosted at CLEF 2022. The participants had to discriminate on Twitter between irony and no-irony spreaders. The provided data cover the English language.
The participants used different features to address the task, mainly: i) 𝑛-grams; ii) stylistics; iii) personality and emotions; and iv) deep learning-based representations such as embeddings and transformers. Concerning machine learning algorithms, the most used ones were combinations and ensembles of different traditional algorithms such as SVM, Logistic Regression and Random Forest with deep learning techniques such as Fully-Connected Neural Networks, CNN, LSTM and Bi-LSTM, and transformer-based ones, mainly BERT and its variations.
The best result (99.44%) has been obtained with a BERT feature-based CNN model. The second best result (97.78%) has been achieved with a combination of SBERT and emojis, and the two ex aequo third best results (97.22%), respectively, with a Multilayer Perceptron trained with features extracted from a pre-trained BERT model, and a Random Forest fed with unigrams pre-selected with several techniques of feature selection.
The error analysis shows that the highest confusion is towards irony spreaders (false positives)
with almost double number of errors (15.45% vs 9.18%), which requires further research to prevent systems to bias their predictions. One of the main challenges of this task was to contemplate the use of stereotypes in a broad sense, that is, not focusing on a target group but considering those users who explain what happens in their environment by intensively using social categories. Behind this theoretical approach there is the idea that prejudice is fundamentally a vision of the world that homogenizes people on the basis of their groups of origin or affiliation. A vision of the world that considers that these group affiliations are the main cause of the people's behaviours and could explain social or economic problems. It is evident that to embrace stereotyping towards many social groups may have introduced a topic bias, although certainly when we analyse stereotypes towards a single group, the type of discourse changes if what is held is a stereotypical view of a group (certain social categories are brought up in order to present certain arguments). For example, gays are brought up in a moral discourse and immigrants are evoked in an economic or legal discussion.
Another conclusion derived from the corpus analysis is that ironic and non-ironic users differ significantly not only in the use of Twitter elements but also in the indices used to characterise language, use of emotions, and communication styles, which could explain the high scores obtained by the classifiers. These consistent differences in style open the door to future research in order to characterize better the use of irony.
Looking at the results, the corpus analysis and the error analysis, we can conclude that: i) it is feasible to automatically discriminate between irony and non-irony spreaders with high accuracy; ii) not only are the topics addressed by both types of users significantly different but also other elements such as the number of emojis they use, the number of users they mention, the number of hashtags they use, the number of URLs they share, their writing style, the emotions they convey or even their personality and communication style; iii) we have to bear in mind false positives since they are almost double than false negatives, and misclassifications might lead to ethical or legal implications [102].
The work of the third author has been partially funded by CDTI under grant IDI-20210776, IVACE under grant IMINOD/2021/72, and grant PLEC2021-007681 funded by MCIN/AEI/10.13039/ 501100011033 and by European Union NextGenerationEU/PRTR. The work of the fourth author was in the framework of the DeepPattern research project funded by the Generalitat Valenciana (PROMETEO/2019/121).
In this section, we aim to investigate the usage of irony to refer to stereotypes via the analysis of the authors' stance toward the targets. In fact, stereotypes may have been employed by ironic authors to hurt the targets (e.g. immigrants, women, the LGBT+ community, etc.) or to somehow support them. This subtask aims at detecting the stance of how stereotypes are used by ironic authors, whether in favour or against the target. Therefore, given the subset of ironic authors that employed stereotypes in some of their tweets, the goal is to detect their overall stance. In the four examples below, it can be observed how ironic messages can be used to support or hurt the target.