Every day, millions of information are shared on the internet through social media. The contents of the social media posts are based on the person's wishes, emotional expressions, ambitions, passions, and achievements. Among these posts there are possibilities of hurtful messages such as sexist contents, getting embedded. It may sometimes be intentional or unintentional, but also may disturb the mental well-being of the recipient. So automatic identification of these sexist languages and terms in social media posts has to be taken into immediate consideration. EXIST (sEXism Identification in Social Media Network) 2024, a shared task has addressed this issue. This shared task addresses binary classification(Task1), multiclass classification(Task2) and multilabel classification(Task3). We contributed Language Agnostic BERT Sentence Embeddings(LaBSE) based MultiLayer Perceptron (MLP) classifier, eXtreme Gradient Boosting (XGBoost) Classifier, and ensemble Convolutional Neural Network (CNN) model for Task1 and LABSE with MLP classifier and XGBoost Classifier for Task2.
Social media platforms have become a basic amenity for communication in the modern world[
EXIST 2024 aims to capture sexism in a broad sense, from explicit misogyny to other subtle expressions
that involve implicit sexist behaviors. The shared task EXIST 2024 was a part of CLEF 2024, based on
English and Spanish comments. The shared task intended to spot diferent categories of sexist content
[
Task 2 - Source Intention
This task aims to categorize the sexist messages according to the intention of the author in one of the following categories: (i) Direct sexist message (ii) Reported sexist messages and (iii) Judgemental sexist message
The second task is a multiclass classification problem, where the system needs to identify the intention behind the tweet. The possible intentions are directly addressing sexism, reporting sexism conditions about women, and judging/condemning sexism.
Various models including a MPL classifier with Language Agnostic Sentence Embeddings, XGBoost, and ensemble CNN were used for implementing the subtasks namely Task 1 and Task 2. The results of all these were submitted for ranking. Considering the two tasks the training and evaluation of the proposed models were carried out using the corresponding dataset provided by the EXIST 2024 task organizers. This model was then tested with the testing dataset provided for the shared task, based on which the task was evaluated.
This paper is organized as follows: Section 2 explains the related work, Section 3 describes the dataset, the methodology used is described in Section 4, the results and discussions are provided in Section 5 and Section 6 provides the Conclusion.
A machine learning model based on a bidirectional LSTM architecture was used for the classification of
sexist and non sexist tweets by [
Seoul metropolitan ciKNN, Naïve Bayes, SVM and GBDTvil complaint dataset in Korean language
had been classified using Random forest and XGBoost, the result had proven that XGBoost Classifier
outperformed Random forest classifier [
BLSTM-C, a hybrid model of BLSTM and Convolutional Neural Network performed well with the
Chinese language dataset for text classification. The BLSTM-C had been coded with two layers of LSTM
and one layer of CNN to obtain the accuracy of 0.962 [
Few research works were carried out on sexism identification and related text classification tasks had been explored. It is found that continuous research is being carried out in related fields like identifying insulting comments, hate speech, toxic comments, and intent classification which can be used as a base for identifying comments representing sexism from social media text. It could also be observed that the tweet and its contents have inconsistent structure, data preprocessing will helps to improve the accuracy of the training model. @ultimonomada_ Si comicsgate se parece en algo a gamergate pues muy bien por Spanish el acoso. Y si se está haciendo un sabotaje porque hay personajes que no os gustan entonces gracias por darme la razón. Sois unos lloricas ofendidos. $@Geek @ℎ@ ′ work for women who get assaulted at home or work. Also would give the government English the ability to track anyone for any reason.
The dataset used to implement Task 1 and Task 2 of EXIST 2024 was the training, evaluation and the test dataset, that were provided by the organizers of the shared task. All the datasets for the shared task Exist 2024 were given in the JSON format from which the important features required for implementing Task 1 and Task 2 were selected. This includes features like id_EXIST, tweet, annotators, and labels_task1 for Task 1 and id_EXIST, tweet, annotators and labels_task2 for Task 2. Other features like gender_annotators, age_annotators, ethnicities study_levels_annotators, countries were identified as unimportant features and were eliminated. Table 1 shows sample instances from the dataset considering both the languages English and Spanish. Twitter is the source of all the instances in the dataset.
The data distribution in the training, evaluation and testing dataset is represented by Table 2 and 3. The training dataset for Task 1 and Task 2 had 6920 instances of which 3260 tweets were in English and 3660 instances were in Spanish. Considering Task 1 there were 3553 instances under the Sexist category and 3367 instances under Non Sexist category. Considering the Task 2, the number of instances was 3141, 1298 and 1035 under the categories Direct, Reported, and Judgemental respectively. The test dataset had 12456 instances of which 5868 were in English and 6588 were in Spanish.
The proposed system uses XGBoost Classifier, LSTM-CNN Classifier, MLP classifier with Language Agnostic embeddings for Task 1 which is a binary classification problem to detect Sexist and Non Sexist comments. The Task 2 was implemented using XGBoost Classifier and MLP classifier with Language Agnostic embeddings, a multi class classification problem with three class labels namely Direct sexist message, Reported sexist message and Judgemental sexist message. The proposed architecture of the system is shown in Figure 1.
The dataset instances in JSON format, was read and cleaned by preprocessing techniques. Preprocessing is the technique of removing unimportant information from texts, which are not used during the classification process. It is performed by removing stop words, symbols, and special characters in addition to that root words are extracted using stemmer and lemmatization algorithms before the dataset is fed to the model.
The class label associated with each of the tweets was not provided directly. Instead the labels are provided by six diferent annotators as Hard Labels and Soft Labels. We chose Hard Label for our implementation. As a part of preprocessing, the approach of majority voting was applied to the provided information to decide the class label associated with the tweet. This was done for both Task 1 and Task 2.
The proposed system used a MLP classifier for which custom generated embedding was provided as input. Language agnostic sentence transformer was used to generate text embeddings. As the Language agnostic sentence transformer is multilingual in nature and support both English and Spanish languages, the same model was used to generate the embeddings for all the given tweets. Similarly Laser encoder pipeline was used to generate LASER embeddings for all the tweets. Both these embeddings were concatenated to generate a final set of embeddings using which the MLP classifier was trained. The hyper parameters associated with the MLP classifier are: random state was set as 42, the maximum iteration was set as 300, relu activation function was used, the parameter alpha was set as 0.05, learning rate as adaptive and solver as adam. The working of this model is represented in Figure 2.
The proposed model when evaluated using the evaluation dataset, it provided an accuracy and Macro F1 Score 0.77.
XGBoost is an optimized distributed gradient boosting library designed to be highly eficient, flexible,
and portable. It implements machine learning algorithms under the Gradient Boosting framework.
XGBoost provides a parallel tree boosting to solve many data science problems in a fast and accurate way.
The system uses XGBoost Classifier which gets the output from the TF-IDF(Term frequency-inverse
document frequency) model. The preprocessed text is fed to the TF-IDF model to find the term frequency
and document inverse frequency. TF-IDF algorithm [
, , ≡ ∑︀ ,
|| ≡ 2 |{ : }| + 1 TF-IDF of the word is calculated by the formula − = , *
The XGBoost Classifier model is tuned by the hyperparameters learning_rate, max_depth, n_estimators, use_label_encoder, eval_metric with the values 0.7,10,80, False, rmse respectively.
CNN model is one of the baseline models in Natural Language Processing and can be used to classify
sentences and text. It processes the data sequences and enables them to evaluate the perspective of a
given sentence and classify it based on the predefined labels [
The performance metrics associated with the evaluation of the diferent models using the evaluation dataset are represented in Table 4 and Table 5.
The metrics considered for the evaluation of Task 1 were ICM-Hard, ICM-Hard Norm and F1_Yes. The metrics considered to evaluation Task 2 are ICM-Hard, ICM-Hard Norm and Macro F1. The values of these performance metrics for the diferent models submitted are shown in Table 6 and Table 7.
On testing the model with the test dataset the MLP classifier with language agnostic embedding provided an ICM-Hard value of 0.3220, ICM-Hard Norm value of 0.6623 and F1_YES value of 0.7044 for Task 1. The same model applied for task2, it achieved a value of -2.0626 for ICM-Hard, 0.2115 for ICM-Hard Norm and 0.1200 for Macro F1. It could be found that the MLP classifier with Language Agnostic Embeddings outperformed the other models.
When the XGBoost Classifier model was tested using the test dataset, the model provided an ICMHard value of 0.2905, ICM-Hard Norm value of 0.6460 and F1_YES value of 0.6946 for Task 1. Considering Task 2, the same model achieved an ICM-Hard value of -0.8873, ICM-Hard Norm value of 0.2115 and Macro F1 value of 0.3148. The XGBoost Classifier outperformed other models for Task2.
When Task 1 is implemented using ensemble CNN model, it achieves an ICM-Hard value of -0.3410, ICM-Hard Norm value of 0.3286 and F1_YES value of 0.4922.
The MLP classifier with language agnostic embedding resulted in an F1_Yes score of 0.7044 based on which Task 1 was evaluated and we were ranked 48 on the leader board. Task 2 resulted in a macro-F1 score of 0.32 using the XGBoost Classifier, by which we were ranked 37 on the leader board.
Sexism detection has become a current research area as it is interlinked with diferent applications like sentiment analysis, opinion mining, ofensive and hate speech detection. Having this in mind CLEF 2024 had come up with the task of sexism detection, EXIST 2024. As per the requirement of shared task by EXIST 2024, the proposed system implemented the MLP classifier with Language Agnostic Embeddings, XGBoost Classifier, and ensemble CNN classification model for Task 1 and MLP classifier with Language Agnostic Embeddings and XGBoost Classifier for Task 2. It was found that MLP classifier with Language Agnostic Embeddings performed well for Task 1 compared to the other models with an F1 score of 0.70. In Task2 XGBoost Classifier model performed well with an F1 score of 0.32. Usage of hybrid approaches where diferent deep learning models are combined can also facilitate eficient detection of sexism from the text. Often it could be observed that sexism is not in the text, but could be detected from the intonation or facial expression, which has made multimodel sexism detection also a promising research area.