In this paper, we describe our submission to the EXIST-2024 contest. We tackled Task 1 - “Sexism Identification in tweets" in English and Spanish. To classify the tweets as texts containing sexism, we created diferent set up of models, changing the ML classifier, the feature type(word/char), the feature amount and the preprocessing of the text. With this set up, we vectorized the text data using tf-idf embedding technique. After training all these set-ups on the training dataset, we chose the best models according to their accuracy and F1-score on the dev set, and used them to predict the test labels. The best model got a F1 score of 72.23 and the rank of 39 out of 70.
Sexism identification in social networks has emerged as a significant challenge within the field of
Natural Language Processing. This task involves detecting and classifying sexist content within social
media posts, which is crucial for maintaining respectful and inclusive online environments. The
identification of sexist remarks is not only important for individual platforms to manage content
but also for broader societal implications, such as monitoring and mitigating the spread of harmful
stereotypes and promoting gender equality[
Social networks have become the primary platforms for social complaints, activism, and widespread
movements such as MeToo, 8M, and Time’sUp. These movements have gained momentum quickly, with
countless women around the world sharing their experiences of abuse, discrimination, and other forms
of sexism encountered in their daily lives. While social networks play a crucial role in amplifying voices
against injustice, they also serve as conduits for the transmission of sexism and other disrespectful and
hateful behaviors[
In this context, the development of automatic tools for sexism identification is essential. These tools can aid in detecting and flagging sexist behaviors, providing real-time alerts to help manage and moderate online content. Furthermore, they enable the estimation of the prevalence of sexist and abusive situations on social media platforms. By analyzing patterns and forms of sexism, these tools can ofer insights into how sexism is expressed and propagated in these digital spaces.
The significance of this task lies in its potential to enhance the safety and inclusivity of social media environments. Efective sexism identification tools can not only assist in immediate content moderation but also contribute to long-term strategies for reducing the spread of harmful stereotypes and fostering a more respectful online community. The eforts in this area, including the contributions from this lab, are pivotal in developing robust applications aimed at detecting and mitigating sexism in social networks.
In this paper, we describe our participation in the EXIST-2024 contest[
The importance of this task is underscored by the growing volume of user-generated content on social media platforms, where the rapid identification and mitigation of sexist content can significantly impact user experience and safety. By leveraging a combination of preprocessing, feature selection, and various machine learning models, our approach contributes to the ongoing eforts in developing robust automated systems for sexism detection.
Feature selection is a critical process in text classification tasks, significantly impacting model performance by identifying the most informative attributes from the text data. In our approach to sexism identification, we meticulously focused on selecting features based on two primary types: word n-grams and character n-grams.
Word n-grams represent contiguous sequences of words within the text, capturing contextual relationships and syntactic structures. By considering sequences of words, n-grams facilitate the model’s understanding of semantic meaning conveyed through word combinations. For instance, in a bigram model, pairs of consecutive words are considered, while a trigram model examines sequences of three words. Using the sentence “The quick brown fox jumps over the lazy dog” as an example, the bigrams include “The quick”, “quick brown”, “brown fox”, “fox jumps”, “jumps over”, “over the”, “the lazy”, and “lazy dog”. Trigrams, on the other hand, include sequences like “The quick brown”, “quick brown fox”, “brown fox jumps”, “fox jumps over”, “jumps over the”, “over the lazy”, and “the lazy dog”. This granular approach helps in capturing the syntactic structure and semantic nuances of word combinations, which are pivotal for understanding context-dependent expressions of sexism.
However, word n-grams have limitations, particularly when dealing with sparse data and out-ofvocabulary words, which are prevalent in social media texts. To mitigate these issues, we implemented techniques such as TF-IDF weighting to emphasize the importance of rare but informative n-grams and reduce the impact of common but less informative ones.
Character n-grams, especially those with word boundaries (char-wb), segment the text into sequences of characters while respecting word boundaries. This method is adept at capturing morphological patterns and handling variations such as typos, slang, and informal language, which are ubiquitous in social media. For instance, character n-grams of length six in the word “identification” might include “identi”, “dentif”, “entifi”, and so on. By incorporating word boundaries, char-wb n-grams can maintain the integrity of individual words while allowing the model to learn from character-level patterns.
Our experiments demonstrated that character n-grams, particularly of medium length (around six characters), consistently outperformed word n-grams. This indicates their superior ability to capture the nuanced morphological features and informal linguistic variations typical in sexist language. The lfexibility of character n-grams in handling diferent morphological structures and idiomatic expressions was particularly beneficial in our dataset, which included diverse and colloquial expressions of sexism.
Through extensive experimentation, we observed that models utilizing character n-grams with word boundaries achieved higher accuracy and F1 scores compared to those relying solely on word ngrams. This suggests that character n-grams provide a richer and more robust feature set for sexism identification in tweets, capable of capturing subtle and context-dependent expressions of sexism that may be overlooked by word n-grams alone.
To optimize the feature selection process, we employed the Term Frequency-Inverse Document Frequency (TF-IDF) technique. TF-IDF helps in quantifying the importance of each n-gram by balancing its frequency within a document against its frequency across all documents in the dataset. By doing so, it highlights the most informative features that are likely to contribute to the classification task.
Text embeddings are representations of textual data in a continuous vector space, enabling algorithms to process and analyze text efectively. These embeddings capture both semantic and syntactic similarities between words or documents, facilitating various Natural Language Processing (NLP) tasks such as sentiment analysis, document classification, and information retrieval.
There are several types of text embeddings, each with its unique characteristics and applications: Word Embeddings Word embeddings, such as Word2Vec and GloVe, map each word to a highdimensional vector, capturing semantic relationships based on the context in which words appear. For instance, words with similar meanings (e.g., "king" and "queen") are located close to each other in the vector space, while unrelated words are far apart. Word embeddings are particularly useful for tasks that require understanding word semantics, such as word analogy tasks and semantic similarity. Contextualized Word Embeddings Contextualized word embeddings, such as those generated by models like ELMo, BERT, and GPT, provide representations that vary depending on the word’s context in a sentence. Unlike static word embeddings, these embeddings can capture the polysemy of words (i.e., words with multiple meanings). For example, the word "bank" will have diferent embeddings in the sentences "I sat on the bank of the river" and "I deposited money in the bank." This context-awareness significantly improves performance in tasks like named entity recognition, question answering, and machine translation.
Document Embeddings Document embeddings extend the concept of word embeddings to larger text units, such as sentences, paragraphs, or entire documents. Techniques like Doc2Vec and Universal Sentence Encoder create fixed-length vectors that represent the overall meaning of a text segment. These embeddings are valuable for tasks such as document classification, clustering, and information retrieval, where the goal is to compare and analyze entire documents rather than individual words.
The use of text embeddings represents a significant advancement in NLP, as they provide a dense and continuous representation of text that traditional bag-of-words models cannot achieve. Embeddings allow for the eficient handling of large vocabularies and capture intricate relationships between words and phrases. This has led to substantial improvements in various NLP tasks, making embeddings a crucial component of modern NLP systems.
In our study, we utilized Term Frequency-Inverse Document Frequency (TF-IDF), as an embedding
method. [
TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical measure used to evaluate the importance of a word in a document relative to a collection of documents. It calculates a weight for each word based on its frequency in the document and its inverse frequency across all documents. Words with high TF-IDF scores are considered more informative for distinguishing documents (Ramos, 2003).
The TF-IDF (Term Frequency-Inverse Document Frequency) score is calculated as follows: TF-IDF(, , ) = TF(, ) × IDF(, ) (1) Where:
TF(, ) = IDF(, ) = log
Number of times term appears in document
Total number of terms in document ︂( Total number of documents in the corpus || )︂
Number of documents containing term
By employing these diverse embedding techniques, we aimed to capture the rich semantic and syntactic features of the text, enhancing the performance of our models in identifying and classifying sexist content in social media posts.
In the approach to sexism identification, we experimented with a variety of machine learning
classiifers to determine the most efective model for our task. Each classifier brings unique strengths and
characteristics, making them suitable for diferent aspects of the classification problem. The classifiers
where chosen from highest accuray models from Lazy Predict. Below, we describe the key classifiers
we employed:
1. Random Forest Classifier ( RandomForestClassifier):
• The Random Forest Classifier is another ensemble learning method that constructs
multiple decision trees during training and outputs the mode of the classes for classification
tasks[
By evaluating these classifiers using LazyPredict, we was able to quickly identify which models performed best on our dataset. This informed our decision-making process and guided us in selecting and fine-tuning the models that ultimately provided the highest accuracy and F1 scores for sexism identification in social networks.
3.1. EXIST 2024
The EXIST 2024 competition[
For instance, the following tweets illustrate examples of sexist and non-sexist messages: • “Alguien me explica que zorra hace la gente en el cajero que se demora tanto.” • "@messyworldorder it’s honestly so embarrassing to watch and they’ll be like ’not all white women are like that’" 3.2. Task 1 In Task 1, participants are required to develop models that can accurately classify tweets into these two categories. The challenge lies in creating a system that can efectively discern the subtle nuances of language and context that indicate sexism. The objective is to build models that are not only precise
• “Mujer al volante, tenga cuidado!” • “People really try to convince women with little to no ass that they should go out and buy a body.
Like bih, I don’t need a fat ass to get a man. Never have.” in detecting overtly sexist remarks but also adept at identifying more covert and context-dependent expressions of sexism.
The development and evaluation of these models involve several stages, including data preprocessing, feature extraction, and the application of various machine learning algorithms. The ultimate goal is to create robust and reliable tools that can contribute to the broader efort of mitigating sexism on social media platforms, thereby promoting a healthier and more respectful online discourse.
Our methodology for identifying sexism in social media posts was based on a systematic approach using
training and development datasets exclusively. The primary objective was to train various machine
learning models on the training dataset and then select the best-performing models based on their
accuracy and F1 score, as stipulated by the competition requirements, on the development dataset.
our approach to solving the task was based on a previous study that dealt with a similar sentiment
classification task [
we began by employing text embedding techniques to represent the textual data in a vectorized format. Specifically, we utilized the Term Frequency-Inverse Document Frequency (TF-IDF) method for each language in our dataset. TF-IDF transforms text into numerical vectors based on the frequency of terms within documents relative to a collection of documents. we experimented with diferent configurations, including: • Various feature types such as words, characters, and character n-grams (e.g., bigrams, trigrams). • Diferent feature ranges, ranging from single words to sequences of characters of varying lengths. • Various amounts of features were chosen, ranging from 1,000 to 20,000, to determine the optimal number of features for classification.
Text preprocessing is a critical step in Natural Language Processing, especially in tasks such as Sexism Identification. In both general and social media text documents, various types of noise are commonly present. This noise can include typos, emojis, slang, HTML tags, spelling mistakes, and repetitive letters. If the text is not properly preprocessed, it can lead to incorrect analysis outcomes and significantly impact the performance of the models.
Former researchers[
In our work, we adopted a comprehensive preprocessing strategy to clean and standardize the text data before applying further analytical techniques. This approach ensured that the models received high-quality input, thereby enhancing their ability to accurately identify and classify sexist content in social media posts.
LazyPredict is an open-source Python library designed to streamline the process of building and comparing multiple machine learning models. It is particularly useful for quickly benchmarking diferent algorithms without the need for extensive manual coding. By providing a simple interface, LazyPredict allows data scientists to eficiently identify the most promising models for their specific tasks[18].
In the context of sexism identification task, LazyPredict proved to be a valuable tool during the initial model selection phase. Given the variety of machine learning classifiers available, we needed a systematic way to evaluate their performance on the dataset. LazyPredict facilitated this by automatically training and testing a wide array of models using default hyper parameters, enabling us to gain a broad understanding of which algorithms might be most efective for our problem.
LazyPredict compared the following ML classifiers: AdaBoostClassifier, BaggingClassifier, BernoulliNB, CalibratedClassifierCV, DecisionTreeClassifier, DummyClassifier, ExtraTreeClassifier, ExtraTreesClassifier, GaussianNB, KNeighborsClassifier, NuSVC, PassiveAggressiveClassifier, Perceptron, QuadraticDiscriminantAnalysis, RandomForestClassifier, RidgeClassifier, RidgeClassifierCV, SGDClassifier, SVC, LGBMClassifier. The results of the LazyPredict model on the data is presented in Table 1 (Appendices).
With the vectorized representations of the text data, we proceeded to train multiple machine learning models using the training dataset. we explored a diverse range of classifiers, including but not limited to: • Extra Trees Classifier • LightGBM Classifier • Random Forest Classifier • AdaBoost Classifier • Bernoulli Naive Bayes • Support Vector Classifier (SVC)
For each model, we evaluated its performance on the development dataset based on accuracy and F1 score. we experimented with diferent feature combinations to optimize model performance. The models that demonstrated the highest accuracy and F1 score on the development dataset Ire selected as our best-performing models for further evaluation.
Finally, we got a list of all the best models. To choose the models that will label the test pool and the labels that will be accessible to the competition, we chose 3 groups of models. The 10 best models, the 50 best models, and the 100 best models. we asked each group of models to tag the test database, for each tweet we chose the majority of tags (yes or no) and created a JSON file that contains all the answers.
Table 2 (Appendices) presents the Accuracy rank and F1 score of the models for Task 1. The table shows for each language the ideal model we received, feature type, range and amount, whether it performed pre-processing, which classifier it used, what was the score we received in the dev phase.
The most prominent classifiers in the best models are the ExtraTreesClassifier, RandomForestClassifier, LGBMClassifier. They are based on classic machine learning algorithms - Random Forest and boosting, and Naive Bayes which are recognized as classic classifiers but strong and good in many ML tasks. Despite the well-known advantages of preprocessing methods in ML tasks, it seems that there is an overall balance between models that were quicker to preprocess their text and models that worked better on the raw text. It may be that more advanced preprocessing methods such as stemming or lemmatization will be more helpful for learning.
With respect to the type of features, sequences of characters seem to work much better than sequences of words. And precisely a medium group of about 6 characters was better than low ranges of 3 or high ranges of 9.
Regarding the amount of features, it seems that more than 10000 features were often required to obtain the good models, and low amounts converged to lower accuracy.
The best model we sent was the combination of the results of the top 50 models and it came in 39th place in the competition. The second model was a combination of the 100 best models in the competition and it was ranked 41st. The model of the 10 best models was ranked 47th.
In this paper, we described our participation in the EXIST-2024 competition, focusing on the task of sexism identification in tweets. Our approach involved experimenting with various models, text preprocessing techniques, feature types, and feature amounts. Through systematic experimentation and evaluation, we identified the most efective models based on accuracy and F1 score on the development dataset.
Our findings revealed several key insights. First, the ExtraTreesClassifier, RandomForestClassifier, and LGBMClassifier emerged as the top-performing models. These classifiers, based on ensemble learning techniques such as bagging and boosting, demonstrated strong performance across various configurations. Additionally, we observed a balance between models that utilized text preprocessing and those that did not. While preprocessing methods like stemming and lemmatization can potentially enhance model performance by normalizing text, their impact varied, suggesting the need for more advanced and context-specific preprocessing techniques.
Moreover, character sequences generally outperformed word sequences, with character n-grams of medium length (around six characters) providing better results compared to shorter or longer sequences. This finding highlights the efectiveness of character n-grams in capturing the nuances of sexist language. Furthermore, models with more than 10,000 features tended to perform better, underscoring the importance of a rich feature set for capturing the subtleties in tweets.
Overall, our study underscores the complexity of sexism identification in social media posts and the importance of leveraging diverse techniques and models to achieve robust performance. These insights contribute to the ongoing development of more accurate and reliable models for sexism detection in online platforms.
Our current work opens several avenues for future research and improvements. One significant direction is the investigation of advanced preprocessing techniques, such as stemming, lemmatization, and context-aware normalization. These sophisticated methods could enhance the robustness and generalization of our models by better handling linguistic variations and subtleties.
Additionally, enriching the training dataset with more examples from diverse sources and languages is essential. This augmentation could improve the models’ ability to generalize across diferent contexts and cultural nuances, thereby enhancing their performance.
Conducting in-depth error analysis is another crucial area for future work. By thoroughly analyzing recurrent misclassifications and patterns, we can understand the root causes of these errors, such as sarcasm, irony, and cultural references. This understanding can inform the development of more accurate and reliable models.
Exploring additional feature types and combinations is also recommended. This includes investigating domain-specific features that better capture the nuances of sexist language. Incorporating semantic and syntactic features, as well as external knowledge sources, could provide a more comprehensive understanding of the data.
Lastly, extending our research to include deep learning models, such as BERT and Transformers, for sexism identification is a promising direction. Addressing the unique challenges posed by diferent languages, such as varying morphological structures and idiomatic expressions, will be critical in this endeavor.
By addressing these future directions, we aim further to enhance the efectiveness and applicability of sexism identification models, contributing to the broader goal of combating sexism and promoting equality in online spaces. disorders, in: Proceedings of the 19th Industrial Conference on Data Mining (ICDM 2019), New York, 2019. [17] Y. HaCohen-Kerner, D. Miller, Y. Yigal, The influence of preprocessing on text classification using a bag-of-words representation, PLOS ONE 15 (2020) e0232525. [18] M. I. J. Putra, V. Alexander, Comparison of machine learning land use-land cover supervised classifiers performance on satellite imagery sentinel 2 using lazy predict library, Indonesian Journal of Data and Science 4 (2023) 183–189.