Systems that can automatically detect ofensive content are of great value, for example, to provide protective settings for users or assist social media supervisors with removal of odious language. In this paper, we present three machine learning models developed at University of Tripoli, Libya, for the detection of misogyny in Arabic colloquial tweets. We present the results obtained with these models in the first Arabic Misogyny Identification shared task ArMI'21, a sub track of HASOC@FIRE2021. With our first model (optimized BERT-based pipelines), we placed as the second-ranked team on sub-task A: Misogyny Content Identification, and as the third-ranked team on sub-task B: Misogyny Behavior Identification.
extraction of text-based features a dificult task. Tweets are short and often consist of few words.
In this paper, we present three models to detect misogyny in Arabic tweets, for the first Arabic Misogyny Identification (ArMI) shared task, a sub track of Hate and Ofensive Content Identification (HASOC) at the 2021 Forum for Information Retrieval Evaluation (FIRE@2021).
Though misogyny detection in Arabic text is a recent topic, some previous work has been done on ofensive language and hate speech detection in Arabic.
The first study on abusive language detection in Arabic was done by
Mulki and Ghanem (2021b) built a levantine data set of 6,603 tweets collected from the
Twitter accounts of several female journalists who covered the Lebanese protests of October
2019. Tweets in the data set are annotated as misogynous or none. Misogynous tweets are
further classified to diferentiate between, for example, a threat of violence versus a derailing
comment. They used several models to detect misogyny and found that BERT is the best model
to classify tweets as misogynous, with an F1 score of 0.88. In the categorical classification they
reported that
Our experiment was part of the ArMI shared task, a sub track of the HASOC at FIRE 2021. The task aims at identifying misogynistic tweets and recognizing diferent misogynistic categories in a collection of Arabic (MSA/dialectal) tweets.
ArMI 2021 used a data set composed of (7,866) tweets written in Modern Standard Arabic
(MSA) and several Arabic dialects, including Gulf, Egyptian and Levantine
We have participated under the name of University of Tripoli (UoT) with three diferent runs in each sub-task. Below is the description of these runs in each task:
Full pipelines with BERT models
We also created a classical machine learning pipeline based on sklearn library
We started by removing all non-Arabic characters for the text, then we removed repeated characters leaving only two of the following characters ( 0ð", " è", "@", "È", "ø", "¬" ) We then removed the dot character, normalized the diferent forms of Hamza to a bare Hamza " @", and split the starting combination of "AK" from any word in the text. This last adjustment was made since this combination is used to address someone in Arabic, is widely used in Arabic hate speech, and is often attached to the following word. We have also normalized wrongly written Arabic phrases widely used in damming someone, such as: " é<Ë@ iJ.¯", " é<Ë@ ½jJ.¯", " é<Ë@ àX@", " é<Ë@ IJªË", " é<Ë@ ZA @", " é<Ë@ àX@" . We then normalized " è" to " è" and final " ø" to "ø". Finally, we replaced the female addressee pronoun "úaeK@" with "IK@" since most tweets are addressing females. Using word frequency in the training data set, we have removed a list of 29 tokens chosen based on their frequency in the training data set. The remaining words are transformed to a matrix of numbers based on their tf-idf score in tweets. A 2-layer feedforward neural network was implemented in keras. We trained the model with a batch of size 100, and trained the model for 4 epochs. The final F1 score we got is 0.838.
We treated Sub-task B as a multi-class classification problem, using essentially the same strategy and system for each of the 3 runs, respectively, but training on the more fine-grained descriptions of misogyny behavior: damning, discredit, dominance, sexual harassment, stereotyping & objectification, and threat of violence. Based on the full-pipeline comparisons for UoT_run 1, we selected the highest-performing pipeline, which included basic tokenization and a BERT-large model from Koc University. For run 3, we used the same experimental setup used in sub-task A, however, we used the "binary" mode to transform words to either 0 or 1 based on their presence in the tweets. Results on the training data set are shown in Table 2.
The test set has been released and above runs have been submitted for evaluation to the organizing committee. Table 3 shows results obtained by participating teams including our submitted runs. Our UoT_run1 scored in the 4th position (2nd-best team), while the UoT_run3 and UoT_run2 scored the 12th and the 14th respectively. Our results show that BERT algorithm is the best performer in our runs.
Table 4 shows results of the participants’ submitted runs for the sub-task B. Our best performer is again the UoT_run1 at the 8th position (3rd-best team). UoT_run3 and UoT_run2 were in the 11th and the 13th positions respectively.
We have tested three machine learning algorithms in classifying Arabic tweets. AraBert, feedforward networks, and traditional machine learning models have been tested on classifying Arabic tweets as a non-misogynous or misogynous and additionally on classifying misogynic tweets into six predefined categories. By far the AraBERT algorithm was the best performer with an F1 score of 90% in the first task and 51.7% in the second. In future work, we plan to test the combination of the preprocessing steps we made with the Keras model and the AraBERT approach.