In this work, we use cutting-edge machine learning approaches to tackle the problem of sexism identification in memes. The study starts by importing and visualising a meme dataset, then pre-processing the images using techniques including cropping, scaling, and normalisation to get them ready for model training. A pre-trained model called CLIP is used to extract features, and the dataset is split into training and validation sets for memes in both Spanish and English. The collected features are used to train and assess a variety of machine learning models, such as Logistic Regression, SVM, XGBoost, Decision Trees, Random Forest, Neural Network, AdaBoost, and SGD. Accuracy scores, classification reports, and confusion matrices are used to evaluate performance. The Random Forest model performed the best out of all of them. After that, a JSON file containing the model's predictions about the occurrence of sexism in a test dataset is created. The results highlight how well-trained models and sophisticated machine learning approaches can identify hazardous content on social media, ofering insightful information for future studies and useful applications that will help create safer online spaces.
Social networks have developed into an essential communication tool in the current digital era, enabling
people to openly express their ideas and opinions but this transparency has also resulted in the spread of
ofensive material, such as sexism—a gender-based discrimination that mostly targets women. As sexism
on social media is so widespread, automated solutions must be developed to identify and remove such
ofensive content. In order to solve this problem, the EXIST 2024 shared task challenges participants to
develop models that can recognise sexist content in environments that are multilingual, specifically in
Spanish and English [
The complex and context-dependent nature of the language used makes it dificult to automatically
detect sexism. With difering degrees of efectiveness, conventional machine learning techniques like
logistic regression and support vector machines (SVM) have been used. Transformer-based models have
shown higher performance in natural language processing (NLP) tasks, such as sexism detection, more
recently. Examples of these models are BERT [
The dataset of memes used in this study has been annotated for sexism. To ensure high-quality input for model training, the dataset is subjected to a thorough pre-processing protocol. The proposed method intends to support the ongoing eforts to develop more inclusive and safe online environments by fusing powerful machine learning models with sophisticated feature extraction techniques.
Finding damaging and sexist content on social media has been a major field of study, with many
diferent strategies and techniques put forth. In the beginning, sexism was frequently studied as a type
of harassment or as a subcategory of hate speech. Character-level and word n-grams with logistic
regression to classify tweets as racist, sexist, or neither along with other research have used conventional
machine learning techniques like Random Forests, TF-IDF, and Support Vector Machines (SVM), utilising
manually chosen features like emotion ratings and Bag of Words (BoW) [
Deep learning has greatly improved the performance of NLP tasks, including sexism detection. In
particular, Transformer-based models like BERT [
A number of crucial processes are involved in the proposed method for detecting sexism in memes: importing and exploring datasets, pre-processing images, extracting features using the CLIP model, partitioning datasets, training and evaluating models.
The collection includes memes as shown in Figure 1 with sexism annotations in both Spanish and
English [
Preparing images for feature extraction and model training requires a crucial step called pre-processing,
which guarantees consistency and ideal input quality. In the pre-processing stage, images undergo
several critical transformations to ensure consistency and optimal input quality for the model. First,
the images are resized to a uniform dimension of 256 × 256 pixels, standardizing input sizes to reduce
computational complexity and enhance processing performance. The images are then centrally cropped
to 224 × 224 pixels, which helps eliminate extraneous background elements and focus on the main
content of the memes. Following cropping, the pixel values are normalized to a range typically between 0
and 1, ensuring uniform feature scaling which accelerates the convergence process during training. The
pre-processing pipeline also involves converting images to RGB format to maintain color consistency
and handling any potential image loading errors. These meticulously crafted steps, performed using
Python libraries such as PIL for image handling and Torchvision for transformations, are essential for
meeting the input specifications of the CLIP model [
For feature extraction, we employ the state-of-the-art pre-trained model CLIP (Contrastive
LanguageImage Pre-training) [
To guarantee an even distribution of memes in Spanish and English, the dataset is carefully divided into training and validation sets. In order to maintain the representativeness of the training data and ensure that the models trained on it can efectively generalise to new examples across other languages, stratified splitting is essential. The validation set acts as an impartial set to assess how well the machine learning models perform; the training set is utilised to fit the models. This process is necessary to determine how efectively the models will function in practical situations and to adjust hyperparameters to avoid overfitting. Additionally, by preventing the models from becoming biased in favour of any one language, the balanced distribution improves the models cross-linguistic applicability.
Diferent machine learning models were trained to categorise the memes as sexist or non-sexist after feature extraction and dataset splitting. Logistic regression, Support Vector Machines (SVM), XGBoost, Decision Trees, Random Forest, AdaBoost, Neural Networks, and Stochastic Gradient Descent (SGD) are among the models that were assessed. After a thorough training process using the collected features, each model is assessed using confusion matrices, accuracy scores, and classification reports. These measurements ofer a thorough evaluation of each model’s efectiveness, pointing out both its advantages and disadvantages in terms of sexist content detection.
The Random Forest model outperformed the other models that were assessed, exhibiting the best classification accuracy and robustness as shown in Figure 6. This model is excellent at managing complicated datasets and reducing overfitting. It is well-known for its ensemble learning method, which integrates many decision trees. The Random Forest model is then used to predict if sexism will be present in the test dataset after it has been determined to be the top performer. The forecasts are then stored in a JSON file for additional examination, resulting in a structured output that is simple to understand and apply to reports and more study.
These intricate procedures, which include pre-processing, feature extraction, dataset splitting, model training, and evaluation, guarantee a strong and all-encompassing solution to the problem of sexism detection in memes. Every stage is meticulously crafted to optimise the eficacy and applicability of the models, hence augmenting the system’s total eficiency in practical scenarios.
Creating a system to recognise sexist material in memes was the focus of Task 4 of the EXIST 2024 shared task. The ICM-Hard metric, normalised ICM-Hard, and the F1 score for the positive class (F1_YES) are used to assess the performance of the model. With a focus on various evaluation situations, these metrics ofer a thorough understanding of the model’s capacity to detect sexist content. The outcomes of our contribution "Umera Wajeed Pasha_1.json" in three distinct evaluation contexts are shown below: all instances, Spanish instances, and English instances.
In terms of the comprehensive assessment of every case, the system produced the following outcomes: Based on these findings, the model is ranked 36th out of all the participants. The system’s capacity to manage intricate, hierarchical classification tasks is shown by the ICM-Hard score, a metric that takes into account the information content of both correct and incorrect classifications. A standardised view of this performance is given by the normalised ICM-Hard score (ICM-Hard Norm), while the F1_YES score emphasises the recall and precision for the positive class—in this example, the identification of sexist memes.
The system’s performance increased when tested on Spanish instances, proving its capacity to manage multilingual data successfully: The model placed 30th in this category with these results. The preprocessing and feature extraction strategies appear to be especially beneficial for Spanish language memes, based on the enhanced scores in the Spanish context. When it comes to identifying sexist content in Spanish memes, a higher F1_YES score denotes improved memory balance and precision.
In contrast, the evaluation on English instances highlighted areas for improvement in the system’s performance: Here, the model ranked 37th. The model’s dificulties in managing English language memes are indicated by the lower scores in the English context, which also point to possible areas for improvement in the pre-processing or feature extraction for English content. In this language area, there has to be a greater balance between recall and precision, as indicated by the relatively lower F1_YES score. The performance measures in various scenarios highlight the advantages and disadvantages of this methodology. Although it showed diversity among languages, the Random Forest model, which was shown to be the best-performing model during training and validation, exhibited resilient performance overall. Future improvements, concentrating on customised feature extraction and pre-processing strategies to handle the unique qualities of English and Spanish memes, might be guided by the insights gained from these results.
It is clear from the insights from the existing results that improving the methods might greatly increase the resilience and eficacy of the sexism detection system. This section describes a number of possible improvements, with particular attention on sophisticated feature extraction methods, dynamic pre-processing pipelines, and hybrid approaches that combine numerous techniques for better outcomes.
1. Improved Feature Extraction: Sophisticated feature extraction methods can be quite helpful in
extracting the semantic and contextual details from memes, which frequently contain nuanced and
intricate sexism indications. Using more complex models and methods can improve the quality of the
features that are retrieved from memes textual and visual components. By aligning visual and textual
elements into a shared embedding space, models that are built to handle both visual and textual data,
such as VisualBERT or ViLBERT [
2. Dynamic Pre-processing: Preserving the consistency and quality of input features requires building dynamic pre-processing pipelines that can adjust to various data formats and language combinations. More eficiently, the heterogeneity in meme formats and content can be handled by a pre-processing architecture that is adaptable and versatile. By employing techniques like object identification to recognise and preserve the relevant portions of an image, adaptive scaling and cropping algorithms can guarantee that important portions of the images are not destroyed. Additionally, by handling various alphabets, special characters, and idiomatic expressions with customised approaches, normalisation techniques that take into account the unique characteristics of diferent languages can improve text processing performance. Increasing the diversity of the training data through the application of data augmentation techniques like random cropping, rotation, and colour modifications can also help to create more resilient models that perform better when applied to previously unseen data.
3. Hybrid Approaches: Rule-based systems and machine learning models together can handle edge cases more skillfully and increase the system’s overall accuracy. By combining the best features of probabilistic and deterministic techniques, hybrid approaches can ofer a more complete solution. By using specified terms, phrases, or patterns that are suggestive of sexist content, rule-based filters might assist in identifying explicit and evident occurrences of sexism that machine learning models would overlook. By combining the predictions from various models, ensemble methods can increase overall performance by combining the strengths of various models, hence increasing the system’s robustness and accuracy. Contextual data, such as user interaction patterns and social network metadata, can also ofer extra insights that improve the detection of sexist content by illuminating the environment in which memes are shared and their possible efects.
To significantly enhance the eficacy and robustness of the sexism detection system, several advanced strategies and techniques can be explored. These improvements focus on various aspects of the model development lifecycle, from data augmentation and pre-processing to model architecture and evaluation metrics.
Using Generative Adversarial Networks (GANs) is one interesting way to increase the model’s robustness. The current dataset can be enhanced by using GANs to produce artificially realistic yet synthetic meme images. By resolving class imbalance and broadening the pool of training samples, this strategy can improve the model’s capacity to generalise across various forms of sexist material. Furthermore, the textual material within memes can be made more diverse by utilising textual data augmentation techniques like synonym replacement, paraphrasing, and back-translation. By ensuring that the model picks up strong features from a variety of linguistic expressions, these techniques raise the accuracy of the model even further.
Text analysis performance in the system can be greatly improved by using transformer-based models,
such as BERT [
Efective management of multilingual text data requires the use of cross-lingual embeddings, such as Multilingual BERT (mBERT). These embeddings improve the systems worldwide applicability by guaranteeing consistent performance across many languages. Creating shared embedding spaces for text and images through multimodal learning can greatly enhance the models comprehension of memes, in addition to its cross-lingual capabilities. By capturing complex interactions between textual and visual aspects, pre-training models on big multimodal datasets strengthens the systems ability to interpret meme content.
General-purpose models can be tailored to the specifics of the target domain by fine-tuning pre-trained models on domain-specific datasets pertaining to sexism detection and social media analytics. This task-specific fine-tuning increases the relevance and accuracy of the models. Furthermore, the models performance on sexism identification can be improved by utilising layer-wise transfer from models that have already been pre-trained on comparable tasks, such hate speech detection, to leverage transfer learning. This method, which makes use of shared features across related domains, cuts down on training time and costs while ofering a strong basis for the new work.
Experimenting with fusion strategies, such as early, late, and hybrid fusion, is essential to capturing
complementing information from memes textual and visual aspects. By fusing textual and visual elements,
these methods ofer a thorough comprehension of memes. Furthermore, the models comprehension of
memes in the context of social media can be improved by employing contextual embedding that take
into account the memes larger context, including user metadata and engagement metrics. By using
this method, the model is guaranteed to capture the entire range of information included in memes,
increasing the accuracy of detection [
It is crucial to keep assessing hierarchical and multilabel classification problems using sophisticated metrics like ICM and ICM-Soft. These measures capture the complexities of sexism detection and ofer a detailed assessment of model performance. Furthermore, user studies that assess the systems functionality in real-world situations and collect input for future improvement might yield insightful information. Through a user-centric evaluation, it is ensured that the model meets user expectations and works well in real-world applications.
This system for detecting sexist content in memes demonstrated moderate performance in the EXIST 2024 Task 4 shared task. The results indicate that while the model is competitive, there is considerable room for improvement, particularly in the English instances where it ranked lower. The system performed better on Spanish instances, which suggests that the pre-processing and feature extraction steps might be more efective for Spanish language content.
The outcomes highlight the intricacy of the task and the subtlety of sexist content, which presents serious dificulties for automated detection systems. Although this method, which fused sophisticated feature extraction with the CLIP model with conventional machine learning models, provided a strong basis, further improvements will be needed to increase its accuracy and durability.
We appreciate the platform that EXIST 2024 shared task organisers provided to further study on the identification of sexist content in social networks. I also like to express my appreciation to the annotators for their work in labelling the dataset, which made it possible to perform this study.