AMI is the first task on automatic misogyny identification [2]. Its aim was to identify cases of aggressiveness and hate speech towards women in social media [1]. Poland's work [3] was the first attempt to manually classify misogynous tweets. Now this shared task will consider two subtasks for this classification:
-subtask1: Misogyny identification.
-subtask2a: Misogynistic Behaviour.
-subtask2b: Target Classification.
The aim of subtask1 is to identify whether a tweet is misogynous or not, and the second subtask2a aims to identify the category, if a misogynous tweet belongs to: discredit, dominance, sexual harassment, stereotype, and derailing. Finally, subtask2b is in charge to identify whether a misogynous tweet is active or passive i.e. if its target is generic (women in general) or individual. In this work, each of these tasks is approached as a classification task. We will use natural language processing (NLP), machine learning and feature engineering to identify patterns and learn classification models respectively.
This section tries to describe the main approaches that have been used. Generally, misogyny can be expressed written, orally, in a subtle or explicit way, also directly or indirectly addressed to someone. In order to investigate how people may express misogyny in tweets, we propose an approach that allows us to discover some aspects about how misogyny is expressed in the corpus provided by the organizers. Hence this approach takes into account some features that we considered important in order to understand if some of them contribute to recognizing misogynous content and its respective category.
Structure (str): Basically, knowing how many words are used in a tweet or if most of those words are written in capital letters, even if some of them use excessively punctuation marks could reveal important information. As we know a tweet is composed of words, punctuations, mentions, URLs, etc. In this approach, we will pay attention to these aspects to see if all of them in some way help to better discriminate between misogynous tweets and not misogynous one. A summary of these features is given below:
-The number of symbols or punctuation marks (!' ?,.").
-The number of words written in capital letters.
-The number of words and characters, including stop words.
-Mean of the numbers of words and characters.
-The number of mentions, URLs, and hash-tags.
Another component that we consider important is the possibility to get features from Linguistic Inquiry and Word Count (LIWC)2 . We have just taken into account some categories related to misogynous emotions such as: angry, sexual, swear, positive, negative, etc. [4] The idea behind this component is to calculate for instance the percentage of positive or negative emotions, or even if a tweet has sexual content as we can see in Figure 1.
In this component Term frequency -Inverse document frequency based on Words (TFIDFW) or Chars (TFIDFC) schemes are used. For instance in misogynous TFIDFW (see Table 1) the term bitch (first place) is more used among the misogynous tweets than among the ones that are not misogynous (fourth place), e.g. in our case the uni-gram bitch has a different weight, it means that this word has a specific weight in a misogynous tweet and has another weight in a non misogynous one. The same logic is followed for subtask2a and subtask2b using TFIDFW of their categories and targets respectively.
The last component of our approach takes into account part of speech information, which has the task of tagging each word in a sentence with its appropriate part of speech. We decide whether each word is a noun, adjective, verbs, etc. Using this component we can identify some patterns, for instance in our corpus some nouns are followed by punctuation marks e.g. bitch!!!!!!.
Thanks to the organizers we count with a dataset of 3307 Spanish and 3251 English tweets respectively. Each tweet is labeled as misogynous (1) or nomisogynous (0) and both datasets are balanced. Regarding the type of misogyny and target, each tweet is labeled as: discredit, dominance, sexual harassment, stereotype, derailing and active or passive in case of the target. With respect to the category and target information, the corpus is unbalanced. The first one is biased in favor of discredit (60%) and regarding the target is biased in favor of active (almost 75%). Moreover, to evaluate our system, a test dataset with 831 and 726 unlabeled tweets in Spanish and English respectively was provided.
For the experiments, we employed a set of feature combinations which has been used to feed some classifiers: Support Vector Machine (SVM), Multi-layer Perceptron (MLP) and MultinomialNB (MNB). SVM and 10 K-fold cross-validation were used. The first one was chosen because its performance was good enough with thousands of features, and the second one allows us to avoid over-fitting in all the experiments. Firstly, the main goal was to face the classification of misogynous tweets in Spanish in order to apply the best performing approach to the rest of subtasks Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018) in English or Spanish. Table 2 shows how the experiments were set up. Approaches ap1 and ap2 had the aim to find out whether features created by TFIDFW, TFIDFC, Bag of word-grams (BOW) or Bag of char-grams (BOC) are useful. ap1 uses the whole group of features (thousands of them) created by TFIDFW or TFIDFC, while ap2 obtains the 60 best features using truncated singular value decomposition (SVD) on TFIDFW and TFIDFC then combines with BOW. Unfortunately, those approaches were interesting but we did not obtain results over our baselines with any of the classifiers (MLP, SVM, MNB). ap3 tries to reduce the number of features: firstly we classified a tweet using MNB and then we obtained their respective probabilities to use them as features (2), additionally we got the best 20 features using SVD on TFIDFW and lastly, we added the features str (5) and lc (10). Unfortunately, with these 37 features we did not achieve results over our baselines in subtask1 and subtask2ab respectively. Now we proceed to analyze the results that we got with the approach proposed in Section 2. ap4 and ap5 follow the same logic, but ap5 obtains better results than ap4 because it uses TFIDFW. Tables 3 and 4 show the best val- ues that we achieved: run4 in Table 3 uses TFIDFW plus structure, category and weight of ngrams(unigrams+bigrams+trigrams) as features and we obtained 0.782 of accuracy applying linear SVM on subtask1 . While with respect to the subtask2a, we added part of speech as feature and we obtained 0.370 of F1-macro. Looking at Table 4, we may observe that in run2 we obtained 0.780 of F1-macro in subtask2b just using lc as feature. Also, that just using str and ng(bigram) we obtained 0.503 of F1-macro on subtask2a.
We did not expect good results in English (see Table 5), but we obtained scores slightly above the average F1-baseline (0.3374) in subtask2a and sub-task2b (see run3 and run4). While in subtask1 we were below the accuracy baseline (0.7837). These results can be due to a bad combination of our features. Table 6 shows the better results we obtained in Spanish (between the first five teams). However, we think that classifying misogynous tweets in this corpus was quite difficult because the performance of the teams was approximately 80% in terms of accuracy. Similarly, in subtask2a and subtask2b, mostly the teams were not far from the baseline.
In this work, we proposed an approach that takes into account some aspects: weights of ngrams, LIWC categories, structural information and lexical analysis. We observed that each aspect contributes in some way to the different subtasks. Moreover, we notice that the four aspects contributed to obtaining a better accuracy and F1-macro in the corpus of English tweets. However, only the first three aspects were useful for the Spanish tweets. As future work, it is interesting to use some techniques to face unbalanced dataset and explore other features. Moreover, we plan deep learning to see what performance this technique could achieve.