This paper explains the author's approach to the task Automatic Misogyny Identification at IberEval 2018 whose objective is to identify cases of aggressiveness and hate speech towards women. It describes the system submitted to the task, which is based on a support vector machine model, its development and the results obtained.
The objective of the task on Automatic Misogyny Identification at IberEval 2018 is to
identify cases of aggressiveness and hate speech towards women. Nowadays in social
media people can develop and express their hatred easily without endangering
themselves, leaving many cases of harassment. One regular target of this hatred are women,
and there are investigations that study how and in which forms this misogyny is
expressed [
The task of misogyny identification shares some similarities with sentiment analysis
or stance detection [
In the following sections this paper will describe the task (Section 2), the submitted system’s approach (Section 3), and the results obtained on the task (Section 4). At the end (Section 5) some conclusions will be presented. The task Automatic Misogyny Identification is performed over a collection of tweets, two corpora are available in different languages, Spanish and English, whose results are evaluated separately. The task includes two subtasks. Subtask A is to classify the tweets as either misogynous or not misogynous, it is evaluated using the standard accuracy measure. Subtask B is to classify according to the type of misogyny, and the target of the message. The categories of misogynistic behavior are five: Stereotype & Objectification, Dominance, Derailing, Sexual Harassment & Threats of Violence, 2 Discredit. The target classification is binary, Active if the target is specific and Passive if the target is generic. The evaluation of the second subtask is the macro average F1 score as follows: 1 = 1( ℎ ) + 1(
) 2 (1) The distribution of misogynistic tweets, behavior categories and target for both Spanish and English tweets in the training corpora is shown in Table 1. Some behavior categories are not very represented, <2% of misogynistic tweets pertain to the category Derailing. There are some differences between the two languages in the presence of misogynistic tweets of the category Dominance, and >85% of the Spanish misogynistic tweets are Active, i.e. refer to a specific target. The system’s approach starts with a preprocessing of the tweets, then each tweet is vectorized by tf-idf features. At the next step the system classifies each tweet for subtask A, the tweets classified as misogynistic in this first subtask are then finally classified for subtask B, a classification by type of misogyny and target of the message. 3
Replace multiple concatenated exclamation and question marks by the keyword MIXED_MARKS.
Replace URLs by the keyword URL.
Replace user mentions by the keyword USER.
Replace misogynistic hashtags by the keyword MISO_HASHTAG. A hashtag was considered misogynistic if it appears only in several misogynistic tweets of the training corpora. Misogynistic hashtags in English are those that contain any of the words: bitch, whore, hoe, cunt, womenare, womensuck. Misogynistic hashtags in Spanish are those that contain any of the words: feminazi, perra.
Replace the rest of the hashtags by the keyword HASHTAG. The system model uses tf-idf feature vectors, that are built from the tweets’ unigrams. For subtask A, that is to classify if a tweet is misogynistic, all tweets are used to extract the vocabulary. On the other hand, for subtask B, that is to classify the misogynistic behavior and the target, only the misogynistic tweets are used to extract vocabulary. Which percentage of terms form the vocabulary according to its document frequency was empirically decided after the development of the system and its experimentation.
The system’s classifier is based on support vector machine (SVM), it uses a linear kernel, and, except for the English part of misogynistic behavior classification, it uses a one-vs-one classifier instead of one-vs-rest. The system classifies if a tweet is misogynistic for the first subtask, only if the tweet is classified as misogynistic then the system classifies its behavior category and its target. The system employs a separately trained classifier for misogyny, behavior and target. 4
During its development the system was evaluated using 10-fold cross-validation. The
range of terms that form the vocabulary according to its document frequency was tuned
for each language to maximize accuracy and F1 score. In Table 2, for each subtask, the
percentages of minimum and maximum document frequency among the rest of terms
are presented. If the value for minimum document frequency is 20%, the terms that
formed the vocabulary must not be the 20% least document frequent, analogously with
the maximum.
4
The exceptional case where a one-vs-rest classifier offered best results than the
one-vsone classifier was for the language English in the misogynistic behavior classification.
This can be due to the use of a much larger vocabulary. Only one run of the system
over the test corpora was submitted to the shared task [