Knowledge-based Neural Framework for Sexism Detection and Classi cation

Abstract

Sexism, a prejudice that causes enormous su ering, manifests in blatant as well as subtle ways. As sexist content towards women is increasingly spread on social networks, the automatic detection and categorization of these tweets/posts can help social scientists and policymakers in research, thereby combating sexism. In this paper, we explore the problem of detecting whether a Twitter/Gab post is sexist or not. We further discriminate the detected sexist post into one of the ne-grained sexism categories. We propose a neural model for this sexism detection and classi cation that can combine representations obtained using RoBERTa model and linguistic features such as Empath, Hurtlex, and Perspective API by involving recurrent components. We also leverage the unlabeled sexism data to infuse the domain-speci c transformer model into our framework. Our proposed framework also features a knowledge module comprised of emoticon and hashtag representations to infuse the external knowledge-speci c features into the learning process. Several proposed methods outperform various baselines across several standard metrics.

Keywords

• Sexism detection
• Sexism classi cation
• Transformers
• Knowledge-base

-

Sexism encompasses stereotypes, prejudice, or discrimination based on a person's sex or gender, most often women. It occurs in various subtle and overt forms, causing immense su ering to women and girls. Inequality and sexism against women prevalent in society are constantly being mirrored on the internet. Women are a ected in many aspects of their lives, including domestic and parental responsibilities, work prospects, sexual appearance, life desires, by the most nuanced manifestations of sexism. So, the automatic detection and categorization of sexism into well-de ned categories may help to analyze sexism to improve sensitization programs and put in place other mechanisms to combat this oppression. It could also aid in the development, design, propagation of new equality policies and as well as to promote positive societal action.

The detection of sexism varies from and may supplement the classi cation of sexism. Sexism detection will be used to identify the sexist posts where instances of sexism are mixed with other posts unrelated to sexism on which to perform sexism classi cation. Some hate speech classi cation works [ 4, 25 ] detect sexism as a type of hate; however, it does not conduct sexism classi cation. The prior work on classifying sexism [ 3, 13, 11, 25 ] has identi ed explicit hatred or violence towards women and classi ed the given tweet into two to ve categories. In this paper, we examine the problem of detecting and classifying sexism in a broad context, from overt misogyny to other indirect phrases that include latent sexist behaviours. More speci cally, we attempt to solve two classi cation problems using tweets and gab posts in Spanish and English. Firstly, we assign a binary label to a tweet indicating whether it is sexist or not. Secondly, if the tweet is classi ed as sexist, we classify it further into ne-grained sexism categories.

We develop a novel neural framework for sexism detection and classi cation that enables a exible combination of text representations generated by the domain-speci c transformer model with the linguistic and semantic feature representations through recurrent operations. Our model can be better equipped to capture the semantic aspects by using general-purpose transformer models like RoBERTa [ 17 ] since it is trained on text data that is much larger than the domain-speci c labeled data we have. However, in order to incorporate domain-speci c elements into our model, we further retrain the RoBERTa model using the unlabeled instances of sexism. The representations from this domain-speci c model complement the representations built from linguistic and semantic features such as Empath [ 10 ], Hurtlex [ 5 ] and Google's Perspective API https://www.perspectiveapi.com/ as a function of end-to-end trainable neural network parameters. Further, to fully comprehend the style of text, we infuse external knowledge information into our framework by leveraging the pragmatics of emojis, smileys, and the speci c context of the hashtags as additional context representations. Our experimentation has shown that multiple instances of the proposed framework outperforming several diverse baselines on established metrics.

Our key contributions are summarized below. { We propose a neural framework that can combine post representations built from di erent linguistic features with those created using the transformer model through learnable model parameters. { Our proposed framework is also aided by external knowledge information by leveraging the hashtags and emojis present in the Tweets or Gab posts. { The proposed methods outperform numerous baselines across established metrics.

Related Work

In this section, we rst look at work on hate speech detection and classi cation since some of it is related to our work in some ways, such as detecting sexist hate. We then explain previous research on sexism classi cation.

The detection of sexism is performed by some hate speech classi cation approaches that include sexism as a category of hate [ 25, 4, 7, 29 ]. [ 11 ] presents an approach for detecting sexism and misogyny from tweets. [ 6 ] build a data-driven model of cyberhate to identify disability, race, and sexual orientation using bagof-words, dictionary, and text parser to extract typed dependencies. [ 25, 24 ] classi ed tweets as sexist, racist, or neither using character n-grams along with extra-linguistic features. Deep learning algorithms such as fastText, RNN, and CNN are investigated by [ 4 ] to classify the given tweet as racist, sexist, and neither. [ 30 ] proposed deep learning ensemble techniques on the existing datasets. [ 20 ] provide a hierarchical Conditional Variational Autoencoder model for negrained hate speech classi cation. [ 29 ] explored the word embeddings with a combination of GRU and CNN and skipped CNN to classify tweets as sexism, racism, both, and non-hate.

In [ 13 ], tweets are classi ed as benevolent, hostile, or non-sexist using biLSTM with attention, SVM, and fastText. Using features such as Part of Speech (POS) identi ers, n-grams, and text embedding, tweets described as misogynist, are categorized as stereotype and objecti cation, discredit, threats of violence, sexual harassment, and dominance, or derailing in [ 3 ]. [ 15 ] has investigated deep learning strategies for classifying tweets of sexual violence but has not directly focused on developing a comprehensive method to detect recollections of personal stories of abuse. [ 12 ] use ConceptNet and Wikidata to classify the tweets related to sexual harassment by text augmentation and text generation. The rst dataset of sexist phrases and attitudes in Spanish on Twitter (MeTwo) is created [ 21 ] and investigates the possibility of using both conventional and novel deep learning models for automatically detecting various forms of sexist conduct.

In [ 27 ], a density matrix encoder inspired by quantum mechanics is used for the classi cation of personal stories of sexual harassment. [ 14 ] explores CNN, RNN, and a combination of them for categorizing personal experiences of sexual harassment into one or more of three classes. [ 18, 1 ] explores multi-label categorization of accounts reporting any kind(s) of sexism. They developed supervised and semi-supervised methods for classifying sexism at the ne-grained level using transformer models such as BERT [ 8 ]. While their study focuses on what an incident of sexism entails, where it happens, and who perpetrates it, but our work focuses on detection and categorization of sexism pertains to how it is stated. [ 2 ] propose a multi-task approach to perform multi-label sexism classi cation. They use sexism detection as one of the auxiliary tasks in a multi-task setup. In contrast, our work performs the sexism detection task explicitly. Our work also makes use of cutting-edge transformer models [ 23 ] that have been trained on vast amounts of data to produce stable and semantically rich embeddings which can be used for downstream tasks such as sexism detection and classi cation.

Dataset

In this section, we will explain both labeled and unlabeled datasets that are used for the sexism detection and classi cation tasks. We also provided the label distribution across the languages for both the tasks. 3.1

Labeled data

The dataset for the tasks were provided by the organizers of the EXIST shared task [ 22 ]. The dataset is composed of tweets and gab posts in two languages: English and Spanish. It consists of 6977 tweets for training and 3386 tweets for testing. Test set also has 492 gabs in English and 490 in Spanish from the uncensored Gab social network.

The rst subtask is a binary classi cation, predicting whether the given text (tweet or gab post) is sexist (i.e., it is sexist itself, describes a sexist situation or criticizes a sexist behaviour) or non-sexist. Table 1 shows the number of posts present in each class for both languages. Once a message has been classi ed as sexist, the second task aims to categorize the given text into ve sexism categories. The number of posts in each class for both languages is shown in table 2. 1. Ideological and Inequality: This class includes text that criticises the feminist movement, condemns gender discrimination, or portrays men as victims of gender-based oppression. 2. Stereotyping and Dominance: The text presents false views about women, such as that they are more desirable for certain positions or that they are inadequate for certain activities, or that men are somehow superior to women. 3. Objecti cation: The text portrays women as subjects separate from their integrity and personal aspects, or de nes those physical characteristics that women must have in order to meet conventional gender norms. 4. Sexual Violence: Sexual suggestions, requests for sexual favors or harassment of a sexual nature (rape or sexual assault) are made.

5. Misogyny and Non-Sexual Violence: The text expresses hatred and

violence towards women.

In this work, we converted all Spanish posts to English using the Microsoft Translator API and use the entire data to perform the experiments. 3.2

Unlabeled data

We also investigate unlabeled data in order to construct the domain-speci c transformer model. We make use of both unlabelled (crawled) and labeled data created by [ 18 ]. They crawled over 90,000 unlabeled examples of sexism from the Everyday Sexism Project, which includes hundreds of thousands of sexism accounts from witnesses and survivors. Along with these unlabeled instances, we also use the 13023 labelled data after removing the labels. We used a total of 1,03,023 unlabeled posts in our work. 4

Proposed Sexism

Approach

Detection and Classi cation

In this section, we detail our approaches for carrying out the detection and classication of sexist posts. We begin the section with the description of the features explored. Next, we discuss the proposed neural architecture, which enables combining di erent post representations by infusing external knowledge information into the learning process. 4.1

Features Explored

We investigated linguistic and semantic features to see if they could aid in detecting and classifying sexist posts.

Perspective API (Pe): Google's Perspective API is an API that uses machine learning algorithms to predict the perceived e ect of a text by analysing di erent emotional concepts. The API provides scores in real numbers between 0 and 1. The API provides the following attributes for English text: toxicity, severe toxicity, identity attack, insult, profanity, threat, sexual explicit, obscene, and irtation. For each sentence in a post, we created 9-dimensional feature vector.

HurtLex (Hu ): HurtLex is a lexicon of aggressive, o ensive, and hateful words/phrases. These hateful words are divided into 17 categories. It also has a 2-level structure named conservative and inclusive. Conservative lemmas are obtained by translating o ensive senses of the words in the original lexicon. Inclusive lemmas are obtained by translating all the potentially relevant senses of the words in the original lexicon.

We consider nine categories that are relevant to our problem; negative stereotypes ethnic slurs (PS), professions and occupations (PA), physical disabilities and diversity (DDF), cognitive disabilities and diversity (DDP), female genitalia (ASF), words related to prostitution (PR), words related to homosexuality (OM), with potential negative connotations (QAS), and derogatory words (CDS). We consider both conservative and inclusive levels results in an 18dimensional feature vector. For each sentence in a post, we check if the lexicon's phrases/words present in these categories and create a feature vector with a frequency of each category.

Empath (Em ): Empath is a tool for analyzing text across lexical categories (similar to LIWC) and can also generate new lexical categories. Empath draws connotations between words and phrases by deep learning across more than 1.8 billion words of modern ction.

We ltered out the built-in human-validated categories relevant to our task. The categories chosen were sexism, violence, money, valuable, domestic work, hate, aggression, anticipation, crime, weakness, horror, swearing terms, kill, sexual, cooking, exasperation, body, ridicule, disgust, anger, and rage result in a 21 dimension feature vector. 4.2

Proposed Architecture

Figure 1 depicts our proposed architecture. Each tweet or gab post (raw input text) is passed to the Perspective API, Empath tool, Hurtlex lexicon, and transformer model to generate the sentence representations. We employ a

Output probabilities Fully connected

layer Fully connected layer R '

H '

E '

T ' Concatenate tweet representations

Raw input text

Knowledge Module Roberta-t egmenbeerdadtiionng hSaesghmtaegnst emoji2vec generation

Extract Hashtags Extract Emojis biLSTM + Attention biLSTM + Attention biLSTM + Attention biLSTM +

Attention

Perspective feature generation

Hurtlex feature Empath feature generation generation

Embedding generation

RoBERTa-t

Re-training Module

Domain Specific retraining

Pre-trained RoBERTa Unlabeled accounts of sexism domain-adapted RoBERTa variant named RoBERTa t for generating more effective sentence representations than those produced by o -the-shelf RoBERTa models. RoBERTa t is created by further training a pre-trained RoBERTa model in an unsupervised manner using unlabelled accounts of sexism. We incorporate RoBERTa t into our end-to-end training and the weights are updated during the training. Further, these sentence representations are passed to the bidirectional LSTM and an associated attention mechanism [ 28 ] to generate the entire text representation. For each sentence in a text, the biLSTM layer produces an h-dimensional output length. These output lengths are aggregated into a vector representation by the attention layer. Overall, this results in four di erent text representations, which are then concatenated to generate the nal post representation T 0.

To improve the e ciency of our model, we add external information to the current architecture, which includes knowledge of the pragmatics of emojis and smileys, as well as the context in which those hashtags are used. The motivation behind this is that the model may not capture the characteristics and true meaning of emojis and hashtags present in the text. We utilize emoji2vec [ 9 ] to obtain a semantic vector representing the particular emoji. The hashtags are segment into meaningful tokens using the ekphrasis segmenter https://github.com/cbaziotis/ekphrasis. We generate the segmented hashtag embeddings using the RoBERTa t such that the text representations and hashtag embeddings are grounded in the same latent space. For each input text, to obtain the centralized emoji representation E0 and hashtag representation H0, we average the vector representations of all the individual emojis and the segmented hashtags. Then we concatenate all three representations T 0, E0, and H0 to get the nal tweet representation R0. This R0 is passed to the fully connected layer. Finally, a fully connected layer with nonlinearity generates the output probabilities. 5

Experiments

This section provides the experimental evaluation of the proposed methods present analysis. Our code, as well as all hyper-parameter values, are available at GitHub https://github.com/Harikavuppala1a/EXIST_shared-task.

For sexism detection we reported results on F score (F ) and Accuracy (Acc) and for sexism classi cation results are reported on F weighted (Fw), F macro (Fmacro), and Accuracy (Acc). 5.1

Baselines

Random

For each test sample, labels are selected randomly as per their normalized frequencies in the training data.

Traditional Machine Learning (TML)

We report the performance using Support Vector Machine (SVM), Logistic Regression (LR), Gradient Boosted Trees (GBT), and Random Forests (RF), each applied on three feature sets, namely the word n-grams, character n-grams, and the average of the ELMo vectors [ 19 ] for a post's words (ELMO). Deep Learning (DL) { biLSTM: The word embeddings correspond to each post are fed through a bidirectional LSTM. { biLSTM-Attention: The biLSTM-Attention is similar to biLSTM, but with the attention scheme from [ 28 ]. { C-biLSTM: This architecture is somewhat similar to approach [ 14 ]. After the convolution operation has been carried out on the word vectors of each post, the lter dimensions are stacked to create a series of window vectors that are then transmitted through biLSTM. { CNN-Kim: Word vectors of a post are passed through convolutional and max-over-time pooling layers similar to [ 16 ]. { BERT and RoBERTa : Sentence embeddings are generated using BERT via bert-as-service [ 26 ] and RoBERTa and passed through a biLSTM with attention separately. We set aside 15% from original labeled data for validation. During the testing phase, the validation set was merged with the training set. For all the methods, the mean of the results obtained over three runs is given for each metric. Most proposed methods outperform all baselines across all metrics.

Table 3 shows results produced using four traditional ML methods (SVM, LR, GBT, and RF) across three di erent feature sets (word n-grams, character

Random

sebiLSTM linbiLSTM-Attention seCNN-Kim aBC-biLSTM

BERT

RoBERTa

biL-att applied on

RoBERTa t

RoBERTa t

RoBERTa t sdRoBERTa t, Em ohRoBERTa t, Em teRoBERTa t, Hu mRoBERTa t, Pe edRoBERTa t, Pe, Hu soRoBERTa t, Pe, Hu opRoBERTa t, Em, Pe rPRoBERTa t, Em, Pe

RoBERTa t, Pe, Em, Hu

RoBERTa t, Pe, Em, Hu

RoBERTa t, Hu, Em

RoBERTa t, Hu, Em

n-grams, and ELMo average) for both the tasks. For SVM and LR, we apply class imbalance correction across both tasks. Among these combinations, RF with word n-grams emerges as the top sexism detection method. SVM with character n-grams produces the best F scores for sexism classi cation.

Table 4 and Table 5 provides sexism detection and sexism classi cation results for random, deep learning and various combinations of proposed framework. For proposed methods, the sub-columns in each row specify which neural method or linguistic features are used to generate post representations and the knowledge information employed. We note that the results are reported for a subset of the possible outcomes of the proposed neural framework.

In Table 4, for sexism detection task, the random method performs poorly as expected. The best deep learning baseline is C-biLSTM based on Acc, and it outperforms its traditional ML counterpart. Our best method involves bil-att processing on RoBERTa t, Hu, and Em separately to generate the nal post representations. When external information such as Emoji and Hashtag representations are concatenated with this best method, the performance is marginally reduced. In Table 5, also the random method performs poorly as expected, reecting the challenging nature of the sexism classi cation. The best deep learning baseline is biLSTM-Attention based on Fmacro, and it outperforms its traditional ML counterpart. Among the various combinations of proposed methods, our best

Random

sebiLSTM linbiLSTM-Attention seCNN-Kim aBC-biLSTM

BERT

RoBERTa

biL-att applied on

RoBERTa t

RoBERTa t

RoBERTa t sdRoBERTa t ohRoBERTa t, Em teRoBERTa t, Em mRoBERTa t, Pe edRoBERTa t, Hu soRoBERTa t, Em, Pe opRoBERTa t, Em, Pe rPRoBERTa t, Hu, Em

RoBERTa t, Pe, Em, Hu

RoBERTa t, Pe, Em, Hu

RoBERTa t, Pe, Hu

RoBERTa t, Pe, Hu

RoBERTa t, Pe, Hu

External knowledge

Emoji

Hashtag

Emoji, Hashtag

Emoji

Emoji, Hashtag

Hashtag

Emoji

Hashtag

Emoji, Hashtag

Emoji, Hashtag

Hashtag

Emoji, Hashtag

Emoji

Hashtag

method uses the biL-att processing on RoBERTa t, Pe, and Em features along with the external hashtag context vector.

Overall, we observed that linguistic features, when combined with the domainspeci c transformer model involving recurrent components, help to improve the detection and classi cation performance to some extent. It has also been noted that in most cases, the external knowledge information (Emoji and Hashtag features) helps to improve the performance even further. Furthermore, several variants of the proposed framework outperform all baselines across all the metrics.

Figure 2 compares the class-wise performance of our best method (Roberta t, Pe, Hu, Hashtag ) with that of the best baseline (biLSTM-Attention) for sexism classi cation. For each class, the average of the F scores over three runs are shown for both methods. For all the classes, the F score of the proposed method outperforms the baseline F score.

We analyze the impact of our proposed sexism detection and classi cation methods on di erent social networks, including Twitter (\with consent control") and Gab.com (\without content control"). Table 6 shows the total number of posts present in the test set for each network and the number of correctly predicted posts by baselines and proposed methods for both tasks. Our proposed approaches perform e ectively on both social media posts compared to the baselines with a reasonable margin for both tasks. In this paper, we explored the sexism detection and ne-grained classi cation of tweets and Gab posts. We developed a knowledge-based neural framework that combines representations created using linguistic features and those obtained using the RoBERTa model trained in an end-to-end manner. We capitalized on unlabeled data to build the domain-speci c transformer model. Our experiments show that the external knowledge representations fed into the neural framework aided in boosting the performance. All the variants of the proposed approach outperforms several deep learning and traditional machine learning baselines. Our analysis showed that our proposed approach is also e ective at detecting and classifying the posts in Gab.com where the abusive content is not restricted. Directions for future work include developing approaches that conduct sexism classi cation more accurately as well as exploring the multilingual sexism detection and classi cation.

References

1. Abburi , H. , Parikh , P. , Chhaya , N. , Varma , V. : Fine-grained multi-label sexism classi cation using semi-supervised learning . In: International Conference on Web Information Systems Engineering . pp. 531 { 547 . Springer ( 2020 )

2. Abburi , H. , Parikh , P. , Chhaya , N. , Varma , V. : Semi-supervised multi-task learning for multi-label ne-grained sexism classi cation . In: Proceedings of the 28th International Conference on Computational Linguistics . pp. 5810 { 5820 . International Committee on Computational Linguistics, Barcelona, Spain (Online) ( Dec 2020 )

3. Anzovino , M. , Fersini , E. , Rosso , P. : Automatic identi cation and classi cation of misogynistic language on twitter . In: International Conference on Applications of Natural Language to Information Systems . pp. 57 { 64 . Springer ( 2018 )

4. Badjatiya , P. , Gupta , S. , Gupta , M. , Varma , V. : Deep learning for hate speech detection in tweets . In: Proceedings of the 26th International Conference on World Wide Web Companion . pp. 759 { 760 . International World Wide Web Conferences Steering Committee ( 2017 )

5. Bassignana , E. , Basile , V. , Patti , V. : Hurtlex: A multilingual lexicon of words to hurt . In: 5th Italian Conference on Computational Linguistics , CLiC-it 2018 . vol. 2253 , pp. 1 { 6 . CEUR-WS ( 2018 )

6. Burnap , P. , Williams , M.L. : Us and them: identifying cyber hate on twitter across multiple protected characteristics . EPJ Data Science 5 ( 1 ), 11 ( 2016 )

7. Davidson , T. , Warmsley , D. , Macy , M. , Weber , I. : Automated hate speech detection and the problem of o ensive language . In: Eleventh international aaai conference on web and social media ( 2017 )

8. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : Bert: Pre-training of deep bidirectional transformers for language understanding . arXiv preprint arXiv: 1810 . 04805 ( 2018 )

9. Eisner , B. , Rocktaschel, T., Augenstein , I. , Bosnjak , M. , Riedel , S.: emoji2vec: Learning emoji representations from their description . arXiv preprint arXiv:1609.08359 ( 2016 )

10. Fast , E. , Chen , B. , Bernstein , M.S. : Empath: Understanding topic signals in largescale text . In: Proceedings of the 2016 CHI conference on human factors in computing systems . pp. 4647 { 4657 ( 2016 )

11. Frenda , S. , Ghanem , B. , Montes-y Gomez , M. , Rosso , P. : Online hate speech against women: Automatic identi cation of misogyny and sexism on twitter . Journal of Intelligent & Fuzzy Systems 36 ( 5 ), 4743 { 4752 ( 2019 )

12. Jafarpour , B. , Matwin , S. , et al.: Boosting text classi cation performance on sexist tweets by text augmentation and text generation using a combination of knowledge graphs . In: Proceedings of the 2nd Workshop on Abusive Language Online (ALW2) . pp. 107 { 114 ( 2018 )

13. Jha , A. , Mamidi , R.: When does a compliment become sexist? analysis and classi cation of ambivalent sexism using twitter data . In: Proceedings of the second workshop on NLP and computational social science . pp. 7 { 16 ( 2017 )

14. Karlekar , S. , Bansal , M. : Safecity: Understanding diverse forms of sexual harassment personal stories . In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP) . pp. 2805 { 2811 ( 2018 )

15. Khatua , A. , Cambria , E. , Khatua , A. : Sounds of silence breakers: Exploring sexual violence on twitter . In: 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) . pp. 397 { 400 ( 2018 )

16. Kim , Y. : Convolutional neural networks for sentence classi cation . In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP) . pp. 1746 { 1751 ( 2014 )

17. Liu , Y. , Ott , M. , Goyal , N. , Du , J. , Joshi , M. , Chen , D. , Levy , O. , Lewis , M. , Zettlemoyer , L. , Stoyanov , V. : Roberta: A robustly optimized bert pretraining approach . arXiv preprint arXiv: 1907 . 11692 ( 2019 )

18. Parikh , P. , Abburi , H. , Badjatiya , P. , Krishnan , R. , Chhaya , N. , Gupta , M. , Varma , V. : Multi-label categorization of accounts of sexism using a neural framework . In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . pp. 1642 { 1652 ( 2019 )

19. Peters , M.E. , Neumann , M. , Iyyer , M. , Gardner , M. , Clark , C. , Lee , K. , Zettlemoyer , L. : Deep contextualized word representations . In: Proc. of NAACL ( 2018 )

20. Qian , J. , ElSherief, M. , Belding , E. , Wang , W.Y.: Hierarchical cvae for ne-grained hate speech classi cation . In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . pp. 3550 { 3559 ( 2018 )

21. Rodr guez-Sanchez , F. , Carrillo-de Albornoz , J., Plaza , L. : Automatic classi cation of sexism in social networks: An empirical study on twitter data . IEEE Access 8 , 219563 { 219576 ( 2020 )

22. Rodr guez-Sanchez , F. , de Albornoz , J.C. , Plaza , L. , Gonzalo , J. , Rosso , P. , Comet , M. , Donoso , T. : Overview of exist 2021: sexism identi cation in social networks . Procesamiento del Lenguaje Natural 67 ( 0 ) ( 2021 )

23. Shazeer , N. , Parmar , N. , Uszkoreit , J. , Jones , L. , Gomez , A.N. , Kaiser , L. , Polosukhin , I. : Attention is all you need . Advances in Neural Information Processing Systems pp. 5998 { 6008 ( 2017 )

24. Waseem , Z. : Are you a racist or am i seeing things? annotator in uence on hate speech detection on twitter . In: Proceedings of the rst workshop on NLP and computational social science . pp. 138 { 142 ( 2016 )

25. Waseem , Z. , Hovy , D. : Hateful symbols or hateful people? predictive features for hate speech detection on twitter . In: Proceedings of the NAACL student research workshop . pp. 88 { 93 ( 2016 )

26. Xiao , H.: bert-as-service . https://github.com/hanxiao/bert-as-service ( 2018 )

27. Yan , P. , Li , L. , Chen , W. , Zeng , D. : Quantum-inspired density matrix encoder for sexual harassment personal stories classi cation . In: 2019 IEEE International Conference on Intelligence and Security Informatics (ISI) . pp. 218 { 220 . IEEE ( 2019 )

28. Yang , Z. , Yang , D. , Dyer , C. , He , X. , Smola , A. , Hovy , E.: Hierarchical attention networks for document classi cation . In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . pp. 1480 { 1489 ( 2016 )

29. Zhang , Z. , Luo , L. : Hate speech detection: A solved problem? the challenging case of long tail on twitter . Semantic Web pp. 1 { 21 ( 2018 )

30. Zimmerman , S. , Kruschwitz , U. , Fox , C. : Improving hate speech detection with deep learning ensembles . In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018 ) ( 2018 )