English. The problem of online misogyny and women-based offending has become increasingly widespread, and the automatic detection of such messages is an urgent priority. In this paper, we present an approach based on an ensemble of Logistic Regression, Support Vector Machines, and Naïve Bayes models for the detection of misogyny in texts extracted from the Twitter platform. Our method has been presented in the framework of the participation in the Automatic Misogyny Identification (AMI) Shared Task in the EVALITA 2018 evaluation campaign.
Italiano. Il problema della misoginia online e dell'odio diretto verso le donne si sta diffondendo sempre più, e così il riconoscimento automatico di tali messaggi è una priorità importante. In questo articolo, presentiamo un approccio basato sui classificatori Logistic Regression, SVM e Naive Bayes per il riconoscimento automatico della misoginia in testi estratti da Twitter. Il nostro metodo è stato presentato attraverso la nostra partecipazione allo shared task AMI presso la campagna di valutazione EVALITA 2018. 1
It is hard to miss the fact that an intensive growth of social networking has led not only to the rise of personal communication opportunities, but also to an increase in aggression on social media. Hate speech can be aimed at sexual orientation, race, religion as gender as a whole. In particular, when the target of hate speech is women, we could say that this is misogyny. Nowadays, more and more attention is paid to this problem, and one of the directions for the hate speech recognition is the women-oriented aggression detection in social networks.
It is important to work with hate speech and misogyny detection now, because over the course of time the data from social networks will grow and this problem will become more and more serious. It is necessary to create a range of systems which allow us to detect and control the number of hate speech messages, and we need to understand how to classify this type of information and how we could reduce the number of it. So, it is a big challenge to find the way of misogyny data detection and processing.
This paper describes our participation in
the Automatic Misogyny Identification
(AMI) Shared Task, in EVALITA 2018
Subtask B - Misogynistic Behavior and Target Classification: the idea of the target classification was to define misogynous tweet which offends a specific person (Active) and tweets which insult a group of people (Passive).
Misogynistic behavior task was intended to divide misogynous tweets into different groups: - Stereotype & Objectification: a widely held but fixed and oversimplified image or idea of a woman, description of women’s physical and/or comparisons to narrow standards. - Dominance: to assert the superiority of men over women or to highlight gender inequality. - Derailing: to justify abuse of women, rejecting male responsibility and an attempt to disrupt the conversation in order to redirect women’s conversations on something more comfortable for men. - Sexual Harassment & Threats of Violence: to describe actions as sexual advances, requests for sexual favours, harassment of a sexual nature, intent to physically assert power over women through threats of violence. - Discredit: slurring of women with no other larger intention.
There were two datasets for the task, one of which contained tweets in the English language and another containing Italian tweets. Our team worked with English dataset only. The English dataset was composed of 4,000 tweets for training and 1,000 tweets for testing. The results were evaluated using the accuracy performance for Task A and macro Fmeasure performance for Task B.
This paper presents our approach to solve the above problems. The main thrust of our approach is to build a model that allows us to assess the classification of any tweet to its assigned group.
The paper is organized as follows. Some relevant related works in the area are described in Section 2. Section 3 presents the way we conducted data preprocessing and the approach we chose for building the desired model. In Section 4 the results are described and analyzed. In Section 5 we summarize our work. 2
There are a number of approaches in the area
of text processing by machine learning
methods which allow us to deal with misogyny and
harassment in texts. Some of these were
presented in the
AMI@IBEREVAL2018 shared task
Some works which could help us to
understand the way to hate speech messages
classification were published in recent years. In
In the research reported in
In our approach we perform a number of sequential actions including preprocessing, model design, and finally embedding the constructed models in one ensemble. 3.1
Preprocessing In the first step, we prepared the data for the classification. To clean the data we removed the string punctuation and converted words to lower case. For the vectorization we used the tf-idf (term frequency–inverse document frequency) method which allows us to reduce the weight of frequently occurring in many documents words and to increase the weight of frequently occurring words in the documents. These were carried out for the first run. For the subsequent two runs, we added some extra preprocessing steps: the replacement of all links with the string "URL" the replacement of all references to Twitter users (i.e, terms starting with the "@" symbol) with the term "USER". we marked some combinations of symbols which were used often in messages such as "!!! ", "??? " and other emotional expressions, and replaced them with the term "emoji". 3.2
Models The main idea of the modeling was to create an ensemble of different models which could complement each other to achieve the best results. The final blended model assigns the tweet to a specific class by majority voting. We used a number of simple models which include:
- Logistic regression model. Logistic
regression involves the construction of a
discriminant model, which calculates the
probability from a function of a weighted set of
observation features and assigns a class to each
observation. The classifier based on logistic
regression applies an exponential function to a
linear combination of objects obtained from
the input data
- Support Vector Machines classifier. As it
was shown in
- Naive Bayes classifier. One of the
advantages of this method is the high speed of
calculations
In the next step we combined the Naive
Bayes approach and Logistic regression
approach in one model, as presented in the work
In the final step we combined the models we have mentioned, Logistic regression (LR), Support Vector Machines (SVM), Naive Bayes and Logistic Regression (NB+LR), into one ensemble. In this blended model the probabilities of belonging to different classes from the simple models were summed and averaged. We marked as a final choice the class which had the highest average probability. 4
We chose three different runs for the evaluation: one of them was implemented by using the simplest type of preprocessing (we just deleted punctuation symbols and changed all letters to the low case) and this variant supposed that we marked a tweet as misogynistic one in case that two of three types of classification marked this tweet as misogynous (Misogyny+Target or Misogyny+Misogynistic Behavior or Target+Misogynistic Behavior).
In the next step, we carried out a more intricate preprocessing as described in Section 3.1 and applied the type of tweets labeling such a way as we detected a tweet as misogynistic each time when at least one classifier worked.
The last run was implemented by using the most complicated preprocessing and the type of tweets labeling such as at the first run.
Table 1 shows the results of all three classification types. As can be seen, the fourth type of selection was the most successful. It could be concluded that the blended model which contained more simple models (Logistic Regression, Naive Bayes + Logistic Regression and Support Vector Machines) allows us to achieve the best results for all classification types: Misogyny Identification, Target Classification and Misogynistic Behavior classification.
It should be noted that we used the F-Measure for the results’ evaluation because this assessment allows bringing together both recall and precision and because of the imbalance within both the Misogynistic Category Classification and the Target Classification.
Task
Classifier
F1-score
Misogyny Identification
Target Classification Misogynistic
Behavior
LR NB+LR
SVM Blend
LR NB+LR
SVM Blend
LR NB+LR
SVM Blend 0.78 0.72 0.71
Also note that the results of our model increase when the number of different classes decreases, thus an efficiency of the blended model is reduced from the Misogyny Identification classification results to the Misogynistic Behavior classification ones.
The results of all 3 runs for the blended model with the testing dataset are presented in Table 2.
Rank 8 9 10
Subtask A - English
Team ITT.c.run2.tsv ITT.c.run3.tsv ITT.c.run1.tsv
Accuracy 0.638 0.636 0.636
It can be concluded by the results on the test data, the best run is the one with the most complicated preprocessing and the type of labelling, when we mark tweet as misogynistic every time when at least one of classifiers worked. 5
A negative aspect of the increased usage of platforms like Twitter is that incidents of aggression and related activities like harassment and misogyny have increased significantly. Nowadays it is an urgent problem to deal with such type of text information and messages, and there are a lot of challenges that have a connection with this task. In this article we have described our approach to misogyny detection and classification of tweets. The method was presented for evaluation in the framework of the Automatic Misogyny Identification (AMI) Shared Task at EVALITA 2018. We built an ensemble of models that includes Logistic regression, Naive Bayes and Support Vector Machines approaches, which classified the data taking into account the probabilities of belonging to classes calculated by simpler models. It was shown that it is possible to achieve quite good results using the final blended model and our model showed the best results for the binary classification of misogynistic tweets and non-misogynistic ones.
We observed preprocessing to be a very important part of the data handling and it has a high impact on the results of all models. From our results it could be concluded that the highest accuracy has been produced with maximum additional work at the preprocessing stage. It was important to pay attention to the replacement of links and references with special symbols, because the run with this type of alteration demonstrated the best results. Also, the best type of labelling misogynistic tweets was to mark the message as misogyny if any one of the type of classification worked. At first we had an idea that it could be more reliably if we mark tweet when 2 of 3 classifications mark it, but the real results disproved that hypothesis. We are currently investigating the addition of more features and models for the blended model to improve our results in the future.
Shushkevich E., Cardiff J. 2018. Classifying Misogynistic Tweets Using a Blended Model: The AMI Shared Task in IBEREVAL 2018. CEUR Workshop Proceedings. CEUR-WS.org.