We present our systems for misogyny identification on Twitter, for Italian and English. The models are based on a Support Vector Machine and they use n-grams as features. Our solution is very simple and yet we achieve top results on Italian Tweets and excellent results on English Tweets. Furthermore, we experiment with a single model that works across languages by leveraging abstract features. We show that a single multilingual system yields performances comparable to two independently trained systems. We achieve accuracy results ranging from 45% to 85%. Our system is ranked first out of twelve submissions for sub-task B on Italian and second for sub-task A.
In questo articolo presentiamo i nostri modelli per il riconoscimento automatico di testi misogini su Twitter: abbiamo addestrato lo stesso sistema prima su un corpus italiano e poi su uno inglese. Il modello si basa su una macchina a vettori supporto e usa n-grammi come feature. La nostra soluzione e` molto semplice e tuttavia ci permette di raggiungere lo stato dell’arte sull’italiano e ottimi risultati sull’inglese. Presentiamo inoltre un sistema che funziona con entrambe le lingue sfruttando una serie di feature astratte. Il nostro livello raggiunge livelli di accuratezza tra il 45% e l’85%: con questi risultati ci piazziamo primi nel task B per l’italiano e secondi nel task A.
With awareness of violence against women
growing in the public discourse and the spread of
unfiltered and possibly anonymous communication
on social media in our digital culture, the issue
of misogyny online has become compelling.
Violence against women has been described by the
UN as a “Gender-based [..] form of
discrimination that seriously inhibits women’s ability to
enjoy rights and freedoms on a basis of
equality with men”1. On the web this often takes the
form of female-discriminating attacks of
different types, which undermine the women’s rights
of freedom of expression and participation2.
Following erjavec2¸012you’s understanding of hate
speech, reported in
3http://www.voxdiritti.it/wpcontent/uploads//2018/06/mappa-intolleranza-3-donne.jpg
nity has addressed the issue of automatic
detection of hate speech in general
To summarise, the following are the contributions of this paper:
We propose a simple and yet strong misogyny detection system for English and Italian (ranked first out of twelve systems for misogynistic category detection) We show how a single system can be trained to work across languages We release all the code4 and our trained systems for reproducibility and for a quick implementation of language technology systems that can help detect and mitigate cybersexism phenomena.
Task Description The AMI task is combined binary and multi-label, short text classification task. Given a Tweet, we have to predict whether it contains or not misogyny (Task A) and if it does, we have to classify the misogynistic behaviour and predict who is the subject being targeted (Task
https://github.com/anbasile/AMI/. be found at
B). The misogynistic behaviour’s space consists of five different labels:
The target can be either Active when the message refers to a specific person or Passive when the message expresses generic misogyny. The setup is the same for both Italian and English. 2
We use only the data released by the task organisers: they consist of Italian and English Tweets. The organisers report that the corpus has been manually labelled by several annotators. We provide an overview of the data set in Table 1. As it can be seen from the table, the data for Task A is more or less balanced, while the data for Task B is highly skewed. 3
In this section we describe the feature extraction process and the model that we built. 3.1
Pre-processing We decide not to pre-process the data in any way, since we do not have linguistic (or non-linguistic) reasons for doing so. To tokenize the text we simply split at every white space. 3.2
Model and Features We built a sparse linear model for approaching this task.
We use n-grams extracted at the word level as
well as at the character level. We use 3-10
ngrams and binary tf-idf. We feed these features
to a Support Vector Machine (SVM) model with a
linear kernel; we use the implementation included
in scikit-learn
Passive
and 2) abstracts from meaning by still approximating the vowels and characters that compose the word; then, we compute the length of the word and its frequency (while taking care of padding the first one with a zero in order to avoid feature collision); finally, we use a Boolean label for explicitly distinguishing words from non-alphanumeric token (e.g. emojis). Table 2 shows an example of this feature abstraction process.
We propose three different multi-lingual experiments:
TRAIN Italian ! TEST English TRAIN English ! TEST Italian
TRAIN Ita. & Eng. ! TEST Ita. & Eng.
For the last experiment, we use half the data set for each language. We report scores obtained by training on the whole training set and testing on the official test set, using the gold labels released by the organisers after the evaluation period. 4
Since the data set labels for the sub-task B are not evenly distributed across the classes, we use f1score to evaluate our model. First we report results obtained via a 10-fold cross-validation on the training set; then, we report results from the official test set, whose labels have been released. The official evaluation does not take into account the joint prediction of the labels, however here we report results considering the 0 label: since we train different models for the different label sets, we make sure that the models trained on Task B are able to detect if a message is misogynistic in the first place. 4.1
Development Results We report the development results obtained by using different text representations. Table 3 presents an overview of these results. We note that all four representations — words, characters, a combination of these two and the bleached representation — all yield comparable results. The combination of words and characters seems to be the best format. Overall, we note that the system performs better on the Italian corpus than on the English corpus. 4.1.1 Cross-lingual Results In Table 4 we present the results of our crosslingual experiments. We train and test different systems using lexical and abstract features. We note that the abstract model trained on Italian outperforms the fully lexicalized model when tested on English, but the opposite is not true. The English data set seems particularly hard for both the abstract and the lexicalized model. Interestingly, the abstract model trained on both corpora shows good results.
Words (W) Chars (C) W+C Bleaching 0.68 0.71 0.70 0.68
ENGLISH
CAT. TGT. 0.88 0.88 0.88 0.85
CAT. TGT.
In Table 5 we present official test results
IT EN
TASK A 0.843 0.617
TASK B CATEGORY
TARGET 0.579 0.293 0.423 0.444
AVG. 0.501 0.369 A warning to the reader: this section contains explicit language. In the attempt to understand better the big difference in performance between the English and Italian models, we show the importance of words as learned by the model: we print the ten most important words, ranked by their learned weights. The result is shown in Table 6. From the output we see that the model trained on Italian learned meaningful words for identifying a misogynist message, such as zitta [shut up], tua [your] and muori [die!]: these words stand out from the rest of the profanity for directly referring to someone, while the rest of the words and almost all the most important English words could be used as interjections or could be more generic insults.
RANK
ITA
The results of the abstract system are satisfactory for eventually building a light, portable model that could be adapted to different language. In the future we will try training on English and Italian and testing on a third corpus (such as the Spanish version of the AMI data set).
In this paper we described our participation to the AMI - Automatic Misogyny Identification for Italian and English. We proposed a very simple solution that can be implemented quickly and we scored a state-of-the-art result for classification of misogynistic behaviours in five classes.
The authors would like to thank the two anonymous reviewers who helped improve the quality of this paper. The first author has conducted this research as he was still part of the Erasmus Mundus master in Language and Communication Technology, a shared master program between the University of Groningen (NL) and the University of Malta (MT).