The contents of this paper describe our system which was submitted to the HUrtful HUmour Detection (HUHU) task at IberLEF 2023. The proposed system achieved a F1 score of 0.781 for Task 1 (binary classification) using an ensemble of pre-trained transformers which included DistilBERT Cased, XLMRoBERTa Spanish, RoBERTuito Cased, mBERT Cased and BERT Cased. A Macro F1 score of 0.739 was obtained for Task 2A (multi-label classification) using ensemble voting from mBERT Cased, BETO Cased, BETO Uncased, DistilBERT Spanish and RoBERTa. For the final Task 2B (regression), our proposed system achieved a RMSE score of 0.938 when the pre-trained transformers were fine-tuned with a fully connected layer (regression head) over the pre-final layer. For the regression task, an ensemble of BETO Cased, BETO Uncased, ALBERT Spanish, DistilBERT Spanish and RoBERTuito Uncased was used. Our proposed ensemble system ranked in the Top 10 systems team-wise for Subtasks 1 and 2A without using knowledge from a high performing baseline i.e. Bloom-1b1 and without any form of data pre-processing or augmentation.
HUrtful HUmour Detection 2023 [
Contemporary research in the fields of sentiment analysis and ensemble voting have widely
contributed to Hate Speech Detection [
Transformers are composed of layers of self-attention mechanisms and an encoder-decoder mechanism. The input corpus is converted to feature embeddings which capture semantic and contextual information. A self attention mechanism allows the system to analyze the impact of diferent tokens by recording long range dependencies. The transformer contains self-attention sub-layers and feed-forward neural network sub-layers which are responsible for parallel computation of the input sequence. In some tasks, such as language translation, transformers employ a decoder component. The decoder predicts the target sequence based on the encoded representation generated by the encoder. By giving consideration to other words in the input sentence, self-attention enables the transformer model to identify word dependencies. It stimulates attention scores that account for the significant connections between each word and its neighbours. The word representations are then updated using these attention scores.
Ensemble voting uses the concepts of weighted majority voting and soft voting [7]. Let ˆ denote the weighted majority vote. It can be computed by assigning a weight to a classifier . A denotes the set of distinctive class labels and is the characteristic discriminant.
ˆ = ∑︁{ }{ () = }
=1
In our tasks, we use hard voting & averaging. The classes which receive the majority of the votes from all the constituent models of an ensemble system are taken into account into the final prediction. The concept of soft voting can also be considered for analyzing the target variables. It contemplates the probability estimates or confidence scores provided by each model and the probabilities for each class label from all models are averaged. The class label with the highest averaged probability is selected as the final prediction. Soft voting is particularly beneficial when the models in the ensemble are well-calibrated and provide reliable probability.
The first batch of data provided by the organizers was the Training Set. This set incorporated 2671 spanish tweets, the 6 labels associated with the tweet and additionally an index label to uniquely identify the tweet. The data used to evaluate the fine-tuned models was a subset of this Training Set. The ratio of split used for Training and Evaluation sets was 80:20. The split was stratified in case of Task 1 (HUrtful HUmour Detection) to ensure that there were equal proportion of positive and negative samples in the training and evaluation sets. The actual Test Set which was used to determine the final ranks of proposed systems for all tasks had 778 tweets and the indexes associated with those tweets. The columns present in the corpus utilized for training and testing have been described in detail below. • index : a 17-bit integer acting as a unique identification to the actual tweet text. • tweet : the contents of the spanish tweet as plain text for which prediction is needed. • humor : a binary valued label indicating if a prejudicial tweet is intended to cause humour. • prejudice_woman : a binary label indicating if the tweet targets women and feminists. • prejudice_lgbtiq : a binary label indicating if the tweet targets LGBTIQ community. • prejudice_inmigrant_race : a binary valued label indicating if the tweet targets immigrants and racially discriminated people. • gordofobia : a binary valued label indicating if the tweet targets overweight people. • mean_prejudice : a real value between 1 and 5 depicting the average prejudice value.
The Training and Testing Data can be obtained from Zenodo (Record 7967255) [8].
The HuHu challenge [
The first subtask aims to determine whether a prejudicial tweet is intended to cause humour. The task in hand is to distinguish between tweets that use humour to express prejudice and the tweets that express prejudice without using humour. For this, the systems will be evaluated and ranked employing the F1-measure over the positive class. • Subtask 2A - Prejudice Target Detection
The second subtask takes into account the minority groups analyzed, i.e, Women and feminists, LGBTIQ community, Immigrants and racially discriminated people, and overweight people. The task aims to identify the targeted groups on each tweet as a multilabel classification problem. The metric employed for evaluation here is Macro-F1. • Subtask 2B - Degree of Prejudice Prediction
The third subtask aims to predict an average rating for each tweet which indicates the degree of prejudice. This regression task asks the systems to rate the tweet on a continuous scale (from 1 to 5) to evaluate how prejudicial the message is on average among minority groups. The evaluation metric employed here is the Root Mean Squared Error.
All models in our ensemble were fine-tuned using the AdamW optimizer with a learning rate of 4e-5 and a batch size of 8. These models were trained on the NVIDIA Tesla V100 GPU.
The scores obtained on the Evaluation Set (random stratified split of training set) for this binary classification task have been mentioned in Table 1 and Table 2. The baselines provided by the organizers for this task include the BLOOM-1b1 transformer, the BETO transformer [9] and SVM + Character Level n-gram which achieved F1 Scores of 0.789, 0.759 and 0.679 respectively on the Test Set. The general idea for submission was to encompass ensemble voting from various ifne-tuned pre-trained transformers instead of just using a single high performing transformer like BETO [9] which outperformed the ensembles in Task 2A and 2B by a considerable margin. The aggregated representation of [CLS] token is passed to a dense layer over which we apply a sigmoid activation function; thus converting logits into normalized probabilities for classification.
The proposed ensemble system is based on majority voting from 5 pre-trained transformers which have been listed in Table 2. We fine-tuned each of these models for 3 epochs and the whole training process took about 12-13 minutes on the V100. The ensemble achieved a F1 Score of 0.8 on the Evaluation Set. The voting mechanism used for this task has been briefly described below. =1 = ∑︁ {} , ˆ = ︂{ 0 if 0 ≤ < 0.5 1 if 0.5 < ≤ 1 k - real valued number, Pi - predicted class label from model i, m - number of models (m%2=1 always), y^- predicted ensemble voting output
Transformer
mBERT Cased mBERT Uncased
XLM-RoBERTa Spanish
F1 Score
[10]
[10]
[10]
[10]
[10]
[9]
[9]
[12]
[13]
[13]
[14]
[14]
[13]
[15]
[16]
[11]
[17]
[17]
[19]
[
The scores obtained on the Evaluation Set (random split of training set) for this multi-label classification task have been mentioned in Table 3 and Table 4. The baselines provided by the organizers for this task include the BETO transformer [9] and a SVM + Character Level n-gram which achieved Macro F1 Scores of 0.760 and 0.603 respectively on the Test Set. BETO Cased [9] outperformed all individual models with a Macro F1 Score of 0.93504 on the Evaluation Set as seen from Table 3. The ensemble system proposed in Table 4 achieved a Macro F1 Score of 0.94351 on the Evaluation Set. On the Test Set, our ensemble system had a F1 Score of 0.739 as opposed to the 0.760 Macro F1 score by BETO baseline. This also indicates that a system using ensemble voting from high performing pre-trained transformers might not always rank above the individual best transformer(s) in composition of the ensemble (BETO [9] in our case).
Transformer
Macro F1 Score
[10]
[10]
[10]
[10]
[9]
[9]
[12]
[13]
[13]
[14]
[14]
[13]
[15]
[
BETO Cased
CamemBERT mDistilBERT Cased
The proposed ensemble system is based on majority voting from 5 pre-trained transformers which have been listed in Table 4. We fine-tuned each of these models for 4 epochs and the whole training process took about 11-12 minutes on the V100. The voting mechanism used for this multi-label binary classification task has been briefly described below.
n ( )j=1 = ︃( ∑︁ , )︃n =1 j=1 (ˆ )j=1 = n ︂{ 0 if 0 ≤ < 0.5 1 if 0.5 < ≤ 1 - Real valued number denoting intermediary output for target label j computed across m systems, Pi,j - predicted class label from model i for category j, m - number of models (m%2=1 always), n - number of labels to be predicted, K^j- predicted voting output for target j
For the regression task of predicting a mean prejudice score for each tweet, we use a regression head i.e. a fully connected linear layer. This regression head uses pooled representations of the [CLS] token extracted from the hidden state sequence produced by various transformers. The scores obtained on the Evaluation Set (random split of training set) for this regression task have been mentioned in Table 5 and Table 6. The baselines provided by the organizers for this task include the BETO transformer [9], a SVM + Character Level n-gram and the BLOOM-1b1 transformer which achieved RMSE scores of 0.874, 0.907 and 0.915 respectively on the Test Set.
Transformer
[10]
[10]
[10]
[10]
[9]
[9]
[12]
[13]
[13]
[14]
[14]
[13]
[15]
[17]
[
The final submission was performed by the first ensemble combination in Table 6 i.e. the one with the lowest RMSE on Evaluation Set. We fine-tuned each of these models for 5 epochs and the whole training process took about 10-11 minutes on the V100 for a 5-ensemble and 6-7 minutes for a 3-ensemble system. The linear average voting schema has been briefed below.
BETO Cased
ˆ = ∑︁ {}
RMSE 0.78910 0.79969 0.73108 0.78555 0.69568 0.71792 0.73672 0.72136 0.78347 0.78780 0.80084 0.71911 0.82665 0.71735 0.74883 0.77367 0.85276 0.77892 0.74169 - Real valued number denoting the predicted output from model i for the regression task, m - number of models (m%2=1 always), y^- mean score from ensemble of m models
From Test Set Results of Subtask 2B, we infer that BETO had a RMSE of 0.874 as opposed to our best ensemble which had a RMSE of 0.938 despite BETO being given superior weightage in the ensemble due to it’s dual presence from cased and uncased versions. The same observation is reflected by results of Subtask 2A as well. This shows that Ensemble Voting might not always benefit tasks like Regression and Multi-Label Classification but can result in a significant performance boost for a rather simplified task of Binary Classification as seen from results of Subtask 1.
The run submitted for ranking was meant to demonstrate the performance of ensemble voting rather than to obtain a better rank from a single high performing baseline like BETO which was either way experimented with individually in all the subtasks. We present a comparative summary of results on the Test Set in Table 7, Table 8 and Table 9 for Subtasks 1, 2A and 2B respectively.
In all the Subtasks, we tried to gain a performance boost either in F1 Score or RMSE using ensemble voting from pre-trained transformers. For Task 1, i.e. the main task of Hurtful Humour Detection, we outperformed most of the systems proposed without using the knowledge from a significant baseline i.e. the Bloom-1b1 Transformer. We also had a decent margin between other baselines like BETO and SVM-3gram-char. For Subtask 2A as well, our ensemble performed decently well when compared to BETO and other proposed systems. In case of Subtask 2B, we see a sharp decrease in performance when seen relative to expectations from Subtask 1 and Subtask 2A. We also experimented with BETO for all subtasks which ranked highly in 2A and 2B subtasks. All the models were trained with raw text and no form of pre-processing /augmentation was used.
The proposed future work includes a comprehensive analysis of other voting methodologies that can be implemented. Weighted majority voting could be used to assign weights to a classifier based upon it’s confidence and reliability. Each classifier would generate a distinctive set of label predictions and the final prediction would be determined by analysing the predictions of various classifiers aggregated across their respective weights. Another voting method that can be implemented is the soft voting method which takes into account the probability scores of various systems. Each classifier predicts the probability of various labels which are computed in a way that the highest probability is assigned as the target prediction probability. Further scope of research could include the tasks of data preprocessing such as handling negations, removal of emojis, emoticons, slangs, abbreviations and stop words, handling imbalanced classes and performing stemming and lemmatization.
The authors of this paper would like to acknowledge the support of Pranav Vyas* for assisting in LATEX documentation and for advising in the context of Pre-Trained Transformers specifically for Sentiment Analysis and Similar NLP Tasks in Spanish. * - pranav.vyas2020 [at] vitstudent.ac.in [3] S. Cui, Y. Han, Y. Duan, Y. Li, S. Zhu, C. Song, A two-stage voting-boosting technique for ensemble learning in social network sentiment classification, Entropy 25 (2023). URL: https://www.mdpi.com/1099-4300/25/4/555. doi:10.3390/e25040555. [4] R. H. H. Aziz, N. Dimililer, Twitter sentiment analysis using an ensemble weighted majority vote classifier, in: 2020 International Conference on Advanced Science and Engineering (ICOASE), 2020, pp. 103–109. doi:10.1109/ICOASE51841.2020.9436590. [5] R. T. Mutanga, N. Naicker, O. O. Olugbara, Hate speech detection in twitter using transformer methods, International Journal of Advanced Computer Science and Applications 11 (2020). URL: http://dx.doi.org/10.14569/IJACSA.2020.0110972. doi:10.14569/IJACSA. 2020.0110972. [6] O. Sharif, M. M. Hoque, Tackling cyber-aggression: Identification and fine-grained categorization of aggressive texts on social media using weighted ensemble of transformers, Neurocomputing 490 (2022) 462–481. URL: https://www.sciencedirect.com/science/article/ pii/S0925231221018567. doi:https://doi.org/10.1016/j.neucom.2021.12.022. [7] S. Raschka, Mlxtend: Providing machine learning and data science utilities and extensions to python’s scientific computing stack, Journal of Open Source Software 3 (2018) 638. URL: https://doi.org/10.21105/joss.00638. doi:10.21105/joss.00638. [8] R. Labadie, B. Chulvi, P. Rosso, HUrtful HUmour (HUHU): Detection of humour spreading prejudice in Twitter, 2023. URL: https://doi.org/10.5281/zenodo.7967255. doi:10.5281/ zenodo.7967255. [9] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained bert model and evaluation data, in: PML4DC at ICLR 2020, 2020. [10] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv.org/abs/ 1810.04805. arXiv:1810.04805. [11] J. M. Pérez, J. C. Giudici, F. Luque, pysentimiento: A python toolkit for sentiment analysis and socialnlp tasks, 2021. arXiv:2106.09462. [12] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, ALBERT: A lite BERT for self-supervised learning of language representations, CoRR abs/1909.11942 (2019). URL: http://arxiv.org/abs/1909.11942. arXiv:1909.11942. [13] J. Cañete, S. Donoso, F. Bravo-Marquez, A. Carvallo, V. Araujo, ALBETO and DistilBETO: Lightweight Spanish language models, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 4291–4298. URL: https://aclanthology.org/2022.lrec-1.457. [14] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, ArXiv abs/1910.01108 (2019). [15] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [16] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, CoRR abs/1911.02116 (2019). URL: http://arxiv.org/abs/1911.02116. arXiv:1911.02116. [17] J. M. Pérez, D. A. Furman, L. Alonso Alemany, F. M. Luque, RoBERTuito: a pre-trained language model for social media text in Spanish, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 7235–7243. URL: https://aclanthology.org/2022.lrec-1.785. [18] M. García-Vega, M. Díaz-Galiano, M. García-Cumbreras, F. Del Arco, A. Montejo-Ráez, S. Jiménez-Zafra, E. Martínez Cámara, C. Aguilar, M. Cabezudo, L. Chiruzzo, et al., Overview Evaluation Forum (IberLEF 2020) Co-Located with 36th Conference of the Spanish Society in: International Conference on Learning Representations, 2021. URL: https://openreview. net/forum?id=XPZIaotutsD. autoregressive pretraining for language understanding, CoRR abs/1906.08237 (2019). URL: discriminators rather than generators, in: ICLR, 2020. URL: https://openreview.net/pdf?id= r1xMH1BtvB.
B. Sagot, Camembert: a tasty french language model, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
The Loss Function used in Subtask 1 (Binary Classification) and Subtask 2A (Multi-Label Binary Classification) are Variants of Cross-Entropy Loss which have been Described Below. =1 ℒ(, ˆ) = −
1 ∑︁[ log(ˆ) + (1 − ) log(1 − ˆ)] 1 ℒ(, ˆ) = − * =1 =1
∑︁ ∑︁[ log(ˆ ) + (1 − ) log(1 − ˆ )] The Loss used in Subtask 2B (Regression) can be computed using the Following Equation. (1) (2) (3) (4) The Equations for Descent using AdamW Optimizer have been Described Below. ℒ(, ˆ) = =1 1 ∑︁( − ˆ)2 ˆ = ˆ +
1 − 2 ˆ =
In (1) and (2), y represents the true binary labels for N examples, where each example can have L labels. Similarly, ˆ represents the predicted probabilities for the corresponding labels. The loss is calculated by summing the cross-entropy loss for each label and averaging it over the number of examples. In (3), y represents the true target values, ˆ represents the predicted values, and N represents the number of examples. In (4), and represent the first and second moments (mean and uncentered variance) of the gradients respectively. 1 and 2 are the decay rates for the first and second moments. represents the gradient at the current time step. ˆ and ˆ are the bias-corrected first and second moments. and − 1 represent the parameters at the current and previous time steps respectively. is the learning rate, is a small value to prevent division by zero, and is the weight decay parameter.