Introduction

The goal of the PAN '22 Author Profiling Shared Task is to classify authors on social media (Twitter) that disperse stereotypes by using irony, especially towards women and the LGBT community [1,2]. The task is unique in that it combines two of the commonly addressed tasks in the NLP community: hate speech (and sexism) detection and irony detection.

In this paper, we develop models for author profiling (defined as a binary classification task: "non-ironic" vs. "ironic") by using a variety of features and models. The models with the highest validation score were obtained by employing a lexicon-based feature extractor on traditional classifiers (Support Vector Machine and Random Forest Classifier), which beat the performance of state-of-the-art models. We also perform feature analyses to identify some of the most informative features for the author profiling task.

The organization of this paper is as follows. Section 2 summarizes some of the previous efforts in related tasks. Section 3 describes the proposed methodologies for the PAN '22 Author Profiling Shared Task. Section 4 describes the classification results and Section 5 provides feature analysis results.

CLEF 2022 -Conference and Labs of the Evaluation Forum, September 5-8, 2022, Bologna, Italy hye-won.jang@uni-konstanz.de (H. Jang)

Related Work

Irony Detection

(Verbal) Irony is often defined in the literature as a communicative act of expressing the opposite of literal meaning [3,4,5]. Although irony is not always associated with hostility, it certainly can be aggressive depending on the context [4,5]. In computational research, the type of task that is more common than irony detection is sarcasm detection. Sarcasm, which is often considered to be a subset of irony [6], is generally regarded as a more aggressive type of irony [7]. There have been many attempts to detect sarcasm computationally, using methods ranging from traditional classifiers [8,9,10] using extracted features [10,11,12] to state-of-the-art models [13,14,15,16,17].

Hate Speech (and Sexism) Detection

Hate speech is defined by the United Nations as "any kind of communication in speech, writing or behaviour, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, colour, descent, gender or other identity factor"

Proposed Methods

Data and Text Preprocessing

The training dataset for PAN '22 Author Profiling task was provided by the PAN 2022 shared task organizers [25]. The dataset consists of 200 tweets from 420 users, all in English. All the tweets of each user are contained in one xml file and each user is assigned a binary label of whether they are spreading irony and stereotypes ("NI": non-ironic, "I": ironic). Usernames and mentions of particular users in the body of tweets are anonymized before the dataset is shared with the task participants.

We extracted and grouped all tweets by each user and their ground-truth labels. Preprocessing was done minimally: mentions of users, hashtags, and url tags were removed. The resulting dataset consisted of tweets from 420 users, with an average word count of 4,338 per user.

Language Representation Methods

Traditional N-gram Features

As a baseline, we experiment with Term Frequency-Inverse Document Frequency features. After converting the data into a TF-IDF matrix using different N-gram sizes (N=1, 2), unigrams (N=1) proved to be the best in terms of classification performance and computational efficiency.

Sentence-Transformers Embeddings

As many state-of-the-art deep learning-based models have performed well in hate-speech (or sarcasm) detection and irony (sarcasm) detection [13,14,15,26,27,21,22,24], we experimented with some of the deep learning-based models. We represented the text using the pre-trained embeddings available on the Sentence Transformer library [28]. The pre-trained models used in our experiments were all-mpnet-base-v2 and average_word_embeddings_glove.840B.300d. The former (all-mpnet-base-v2) was chosen because it was reported to produce the best quality of sentence embeddings among the general-purpose models [28]. The latter (average_word_embeddings_glove.840B.300d), which provides average embeddings from GloVe [29] pre-trained embeddings, was selected because the computation speed is higher than other transformers-based models [28].

Lexicon-based Features

In order to obtain results with better explainability, we also developed a model using only lexicon-based features. We used the software Lexical Inquiry and Word Count (LIWC) (version 2015) to extract features belonging to various lexical categories such as Psychological Processes and Linguistic Dimensions [30]. This text analysis application extracts the percentage of words belonging to certain lexical categories based on its internal dictionary. A total of 93 features were extracted2 . To avoid issues arising from range differences between features, we scaled the features by subtracting the mean (u) and scaling them to unit variance by dividing them by the standard deviation (s) as in the following formula. We used the StandardScaler function from the sklearn library on Python for this.

z = (x -u) / s

The scaled features were used as input to the classifiers, which are described in Section 3.3.

Classifiers

Traditional Classifiers

Using the features mentioned in 3.2 as input, we experimented with several traditional classifiers: Random Forest Classifier (RF), Support Vector Classifier (SVC), Gaussian Process Classifier (GPC), Decision Tree Classifier (DT), Adaptive Boost Classifier (ABC). We only report results by SVC and RF, which produced the best results.

Transformers-Fine-Tuning

Instead of using contextualized embeddings extracted from transformer-based pre-trained models as input to subsequent classifiers, we also fine-tuned two transformer-based pre-trained models on our dataset: BERT [31] and distilBERT [32]. BERT was chosen because a myriad of research has proven its good performance on a range of NLP tasks [31,33,34,35,36,37] DistilBERT was chosen as it is reported to be a more efficient model compared to BERT [32].

Implementation Details

The original training dataset was split into the training and validation set with a 80 to 20 ratio after the data points were shuffled. With the traditional classifiers, the classification was implemented 100 times with random split and the average accuracy scores are reported as the final performance score to prevent biases stemming from the data split structure. For the fine-tuning approach, the text by each user was truncated to the maximum number of allowed tokens (512 tokens). The training was carried out for 3 epochs with the batch size of 64. The official metric used for the PAN '22 Author Profiling Task is accuracy score, which is the ratio of the sum of true positives and true negatives out of all the predictions. The evaluation of the models on the official test set was done on the Tira3 platform [38]. The implementation for the classification experiments was done using the sklearn, torch, and transformers libraries on Python.

Results

The full results on the validation data are presented in Table 1. Our best performances come from LIWC-SVC and LIWC-RF (described in 3.2.3 and 3.3.1), with a validation accuracy score of 0.88. We made a submission using the LIWC-SVC model, and the official test set accuracy for the model is 0.92. Our submission ranked the 45th place out of a total of 65 submissions. The submission with the highest accuracy score had the score of 0.99 and the submission with the lowest accuracy score had 0.53. Our winning models beat the transformers-based fined-tuned models. One possible reason behind this could be the length of each tweet being way over the limit of the maximum word allowed in the transformer models. Both BERT-base and DistilBERT allow up to 512 tokens for encoding text. Considering that the mean word count for the tweets is 4,338, a significant amount of information could have been lost in the process of text representation.

Post-hoc Feature Analyses

Since LIWC features yielded the highest classification scores, we performed further analyses to probe for any improvement or drop in the classification performance by subtracting certain features from the input set to the classifier. To select the most relevant features for the classification, we ran simple linear regressions between each of the 93 LIWC features (x) and the target labels (y). We used the software R [39] to run linear regression model (lm) as below.

lm (y ~x) where y denotes the dependent variable, which is the categorical labels coded as 0 or 1 and x denotes an independent variable, namely, one out of 93 LIWC feature vectors chosen at each time. We then calculated the R-squared value (R2) of the fitted model, which shows the proportion of the explained variance of the feature. The higher the R2, the more role the feature played in fitting the regression line. After obtaining the R2 values for each of the 93 feature vectors, we selected the features with the top 10 -90 percent of the R2 values (in the increment of 10). We ran the classification models (SVC and RF) again using the subset of the input features.

The experiment results show that while the classification performances do not improve by subtracting certain features, the performance starts dropping when only 20 percent of the features are used (Table 2). The performances of all the other combinations were comparable; using only the top 30 percent of the features (N=27) still yields similar results to using all the features (N=93). Table 3 shows the features that were absent in the top 10 and top 20 features but were included in the top 30 features. We assume that these features are more informative than others for the irony profiling task as the classification performance started dropping significantly when these features were excluded. The category certainty stands out, which is an observation aligned with general intuition: it is reasonable to assume that authors spreading irony and stereotypes would heavily rely on sentence structures using certainty (in an ironic way). Given that hyperbole is one of the tools people use for being ironic [40,41], it is expected that people use markers of certainty in the opposite direction of what they intend to say. Such linguistic patterns were commonly observed in instances correctly classified as ironic when we looked into the classification results as well.

• ... if you kill the opposing party you'll win right • ... only use twitter to harass people since all you dudes do is stare at our tits yeah we got the memo... • ... but it's totally cool when they call me a murderer i mean calling someone a murderer isn't abusive or anything • ... wow a democrat telling the truth for once that would be historic good for you • ... i have way more than two guns in my house and my family isn't scared or worried about them at all • ... he brought his young son with him right such a good father ... what kind of a parent allows their kid to be used ... like this As can be observed from the highlighted phrases, words that indicate certaintyall, totally, at all, such a -are often found in examples that were correctly classified as ironic. This observation aligns with the previous findings in the literature that identified hyperbolic words to be one of the markers for ironic comments [40,41].

Conclusions

In this paper, we described our models for the PAN '22 Author Profiling Shared Task and reported the results. We experimented with traditional n-gram features, contextualized sentence embeddings, and lexicon-based features with traditional classifiers. We also fine-tuned transformer-based models on our dataset. The best accuracy score on our validation data was achieved from traditional classifiers that were fed lexicon-based features. We identified some of the informative features for the task by performing post-hoc feature analyses and shared some insights about the usage of irony and stereotype spreading through qualitative analyses.