Introduction

With the proliferation of social media, irony has made it one of the most literary device utilized in this communication manner [1]. Several definitions have been provided in the literature about irony, but they concurred with the same binary classification, verbal and situational irony. The former has been conceived as the act of using words that mean the opposite of what you think, particularly to make funny [2,3]. The latter has been defined as a strange or funny situation because things happen in a way that seems to be the opposite of what you expected [4,5]. Both definitions highlight one of the primary features of this rhetorical device, to make something understandable by expressing the opposite [3]. Such rhetorical complexity makes dialogue that is occasionally arduous to comprehend by humans [6]. This challenge has attracted the research community's attention. In recent years, several approaches have been published addressing the detection of irony in natural language text obtained from different social media sources. In particular, we have focused on identifying whether its author spreads Irony and Stereotypes for the PAN shared challenge [7].

Irony and Stereotype identification is an essential task in social media applications since it enables the identification of online abuse and harassment [8]. The automatic detection in written discourse is a complex task where traditional text mining methods cannot be applied successfully [9]. This conventional method's drawback is that the identification requires semantics that cannot be inferred from word counts computed from document analysis [10]. To overcome this deficiency, more sophisticated Machine Learning methods are applied to solve the problem, but although the obtained results are quite competitive, there is still scope for improvement [11].

In this working note, we have faced the author profiling challenge proposed by constructing a supervised classification pipeline. The method comprises four stages: pre-processing stage to clean the provided dataset; the collecting features a stage, where contextual and non-contextual embeddings were utilized; training stage, where several machine models were used and finally, the evaluation stage, where we evaluated the designed models.

The remainder of this working note is organized as follows: Section 2 provides a brief review of the related work. It examines distinct approaches proposed in the literature that address the challenge thrown. Section 3 specifies the methods developed for addressing the challenge. In Section 4 the results achieved in the challenge are presented. Besides, we report separately our participation in a subtask concerning Stereotype Stance Detection in Section 5. Finally, Section 6 summarizes the findings obtained developing this work, and it also scrutinizes some of the future lines to explore.

Related work

Due to the complexity of recognizing verbal irony in a natural language test, we can find different approaches, from ones that utilize simple strategies to more complex ones. Barbieri and Saggion in [12] proposed two tree-based classifiers, Random Forest and Decision Tree. They represent each tweet by using the following seven groups of features: i) frequency to analyze the gap between rare and common words utilized by users; ii) written-spoken to capture the users' style; iii) intensity to measure the power of the adverbs and adjectives; iv) structure that analyzes the length, punctuation and emoticons; v) sentiments utilize SentiWordNet to measure the gap between positive and negative terms; vi) synonyms for comparing common vs rare synonyms utilized; and, finally, vii) ambiguity to analyze possible ambiguities. Furthermore, this approach explores the usage of a bag of word representation based on frequency analysis. Anchiêta et al. in [13] proposed two differentiated strategies in a more complex way. Firstly, they combined Term Frequency, Inverse Frequency (TF-IDF), and Linear Support Vector Machine (SVM). The former was used to extract the features from the datasets, and the latter was the classifier utilized for the identification task. The classifier was trained by using the Stochastic Gradient Descent (SGD) technique. Secondly, they combine embeddings created by using Distributed Bag of Words Paragraph Vector model and a Multi-Layer Perceptron (MLP) for tackling the classification task. With a different level of complexity, Wu et al. in [14] proposed the Dense-LSTM model based on a densely connected LSTM network with a multi-task learning strategy. It comprises an embedding layer to convert the inputs tweets into a sequence of dense vectors and four Bi-LSTM layers concatenated with 200-dim hidden states to learn different levels of information simultaneously. Furthermore, they combine two different pre-trained word embeddings that are concatenated and used.

Methodology

The IROSTEREO challenge consists of a binary classification from an author profiling perspective. The dataset proposed in this task is compiled from Twitter. The training dataset has a total of 420 different users. The users are grouped in those who are irony and stereotype spreaders (I) and those who are not (NI). For each user, there are 200 of their tweets written in English [15]. We separate a small subset from the training dataset to perform a custom validation. The statistics of the dataset are depicted in Table 1. We followed a typical pipeline of supervised classification for solving the proposed task. We started applying a pre-processing stage of the dataset. Then, we compile the feature sets, train several machine learning models, and evaluate them using a custom validation split.

The pre-processing stage consists in the creation of an alternative version of the documents by encoding them in lowercase, removing mentions, hyperlinks, digits, punctuation, and expressive lengthening. Besides, we expand texting language and fix misspellings. The alternative version is used to extract the majority of the feature sets based on sentence embeddings and linguistic features.

The feature sets involved in our experimentation consist into linguistic features from UMU-TextStats (LF) [16,17], and three sentence embeddings: non-contextual sentence embeddings from FastText (SE) [18], and two contextual embeddings from BERT (BF) [19] and RoBERTa (RF) [20]. These feature sets were used separately and combined using two approaches. One is based on knowledge integration, and another is based in ensemble learning. For the ensemble learning, we evaluate four strategies: i) soft voting, ii) hard voting, iii) average probabilities, and iv) highest probability. Concerning the hard voting strategy, it is a weighted mode with the weights based on the F1-score results of the custom validation split.

As we deal with author analysis, the results are reported at author level. Nevertheless, some of the described stages of our pipeline are performed at document level. For example, the features are compiled at the document level and then combined by each user to produce a unique vector per user.

For extracting the contextual sentence embeddings from BERT and RoBERTa we fine-tune the models with the IROSTEREO dataset, and then we obtained the value of the [CLS] token [21]. In order to find the best hyperparameters, we trained ten models for BERT and 10 models for RoBERTa. The hyperparameters are i) the weight decay, ii) the batch size, iii) the warm-up speed, iv) the number of epochs, and v) the learning rate. This step is performed using Tree of Parzen Estimators (TPE) [22], which is a method for choosing the hyper-parameters based on Bayesian reasoning and expected improvement.

Next, we train several neural networks for each feature set and for the combination of all feature sets using a knowledge integration strategy. These hyperparameters include the shape of the network, the dropout mechanism, the learning rate and the activation function. Table 2 depicts the best hyperparameters for this task. It can be observed that the majority of best results are obtained with shallow neural networks, with two hidden layers but a large number of neurons. The only exception is SE, which achieved its best result with 7 hidden layers and 27 neurons in a long funnel shape. Besides, all experiments achieved better results with high dropout mechanisms and a learning rate of 0.010 using no activation function (linear). The exception again is SE, which uses a smaller learning rate, a smaller ratio of the dropout and elu as an activation function.

Results and analysis

First, we report the results achieved with our custom validation split. These results include the label's precision, recall, and F1-score, and the macro and weighted average of the whole task. We report the results of the best feature set trained separately in Tables 3, 4, 5, and 6 for LF, SE, BF and RF respectively. The results for the KI strategy in Table 7, and the results for the four strategies using ensemble learning in Tables 8, 9, 10, and 11 for hard voting, soft voting, averaging probabilities and highest probability, respectively. From the results achieved with the custom validation split, that are reported at the user level, we can assume that determining if a user is an irony and stereotype spreader is somehow a trivial task. It is worth mentioning that these results at the document level will be more limited. The best results are achieved with BERT from the feature sets separately. However, it draws our attention to the limited results achieved with RoBERTa (see Table 6). We observed that all the incorrect predictions are from the I label, but the model reports the NI label. We also compared the predictions between BF and RF and observed that the BF model outputs probabilities near 100% whereas RF is less accurate.

We can observe that the features based on pure linguistics also achieve similar results to the ones obtained with state-of-the-art embeddings. The LF features include features related to stylometry, lexis, social media jargon, and Part-of-Speech features. In order to gain insights concerning the interpretability of the features, we calculate the Information Gain of the linguistic features and we normalize the top-ten that achieved a better coefficient for the I and NI labels (see Figure 1). It can be observed that the majority of the most discerning features are related to stylometry, including the number of words, the number of words per sentence, the usage of full stops, and some readability formulas. There are two linguistic features concerning morphology: the usage of interjections and the usage of words in singular. Because of the results achieved, for participating in this shared task, we sent one run based on the Knowledge Integration strategy, achieving the fifth best result an accuracy of 96.67% from a total of 65 participants. We selected the Knowledge Integration strategy over the two ensemble learning strategies that achieved the same results (hard and soft voting) because the Knowledge Integration have reported better results in other shared tasks in the past. Table 12 contains the best results along with the baselines proposed by the organizers. It is worth mentioning that these results were yielded from TIRA [23], an Integrated Research Architecture utilized by IROSTEREO organizers for managing the participants' algorithms executions.

Stereotype Stance Detection subtask

The organizers of the IROSTEREO shared task proposed a minor challenge that consisted in determining whether the stereotypes are used in favor of the target or against them. For this, they released a training dataset in which 94 authors were tagged against and 46 authors were tagged in favor.

To solve this challenge, we utilized the same pipeline described for the main challenge. Our results with our custom validation split are promising. We report the Knowledge Integration strategy and the four ensemble learning strategies in Table 13. We achieved a macro F1-score of 82.8753% with the Knowledge Integration strategy and a macro F1-score of 78.5714% with the ensemble learning based on soft-voting.

However, our results with the official leader board were limited. We achieved a macro F1-score of 53.12% (F1 with the In Favor label of 25% and F1 of the Against label of 81.25%).

Conclusions and future work

This working note describes the participation of the UMUTeam at IROSTEREO shared task concerning author profiling. This is a binary classification task in which the participants are challenged to identify which profiles from Twitter are spreaders of Irony and Stereotypes. Our proposal is grounded on the combination of several feature sets based on linguistic features and sentence embeddings. We achieved promising results with our custom validation split and achieved a final accuracy of 96.67% on the official leader board. One of the limitations of our work is the results achieved with RoBERTa (RF). Although we searched for common errors in our pipeline, we could identify the reason for the limited results. To address this issue, we suggest combining document level analysis with tools such as SHAP [24] in order to find the reason for the wrong predictions. Besides, we obtained wrong predictions with the highest probability strategy (see Table 11) as this ensemble outputs always the I label (100% of accuracy). We suspect this issue is related to an error in code while generating the final report.

As future work we will incorporate cross-validation techniques into our pipeline and dataaugmentation techniques to increase our models' generalization.