English. In this paper, we present a system able to classify hate speeches in Italian messages from Facebook and Twitter platforms. The system combines several typical techniques from Natural Language Processing with a classifier based on Artificial Neural Networks. It has been trained and tested on a corpus of 3000 messages from the Twitter platform and 3000 messages from the Facebook platform. The system has been submitted to the HaSpeeDe task within the EVALITA 2018 competition and the experimental results obtained in the evaluation phase of the competition are presented and discussed.
Italiano. In questo documento presentiamo un sistema in grado di classificare messaggi di incitamento all’odio in lingua italiana presi dalle piattaforme Facebook e Twitter. Il sistema combina diverse tecniche tipiche del Natural Language Processing con un classificatore basato su una Rete Neurale Artificiale. Quest’ultimo stato allenato e testato con un corpus di 3000 messaggi presi dalla piattaforma Twitter e 3000 messaggi presi dalla piattaforma Facebook. Il sistema stato sottomesso al task HaSpeeDe relativo alla competizione EVALITA 2018, e sono presentati e discussi i risultati sperimentali ottenuti nella fase di valutazione della competizione.
In the last years, social networks have
revolutionized in a radical way the world of communication
and the publication of contents. However, if on
one hand social networks represent an instrument
of freedom of expression and connection, on the
other hand they are used for propagation and
incitement to hatred. For this reason, recently, many
softwares and technologies have been developed
to reduce this phenomenon
Specifically, approaches based on machine
learning and deep learning are used by large companies
to stem and stop this widespread fact. Despite the
efforts spent to produce systems for the English
language, there are very few resources for Italian
In this paper we describe the system submitted
by the Vulpecula team. The system works in
four phases: preprocessing of the initial dataset;
encoding of the preprocessed dataset; training
of the Machine Learning model; testing of the
trained model. In the first phase, the comments
were cleaned by applying text analysis techniques
and some features have been extrapolated from
these; then in the second phase, using a trained
Word2Vec model
The system has a structure similar to
In the first phase the corpus of 3000 Facebook comments, and the corpus of 3000 Twitter comments are cleaned and prepared to be encoded. In parallel within the cleaning we have extrapolated some interesting features for each comment. The entire phase is explained in details in section 4 and 5.
In the second phase we trained a Word2Vec model, starting from 200k comments we download from some Facebook pages known to contain hate messages. Each of the initial data set comment has been encoded in a vector of real values by submitting it to the Word2Vec model. This phase is explained in details in the section 5.
In the third phase we trained a multi-layer feed-forward neural network using the 3000 encoded comments and the respective features we extracted in the first phase. The description of the ANN is in the section 6.
In the last phase the test set comments provided by EVALITA were classified and we joined the competition. This phase is explained in details in the section 7.
The source code of the project is provided online1. 3
The entire project was developed using the Python programming language, for which several libraries are available and usable for the purpose of the project. Specifically, the following libraries were used for the preprocessing phase of the dataset: nltk: toolkit for natural language processing; unicode emoji: library for the recognition and translation of emoticons; treetaggerwrapper: library for lemming and word tagging; textblob: another library for natural language processing; gensim: library that contains word2vec; sequence matcher: library for calculating the spelling distance between words;
For the training phase of the ML model the
following libraries were used:
keras
SentiWordNet
In the Knowledge Discovery in Databases (KDD) one of the crucial phases is data preparation. In this project this phase was tackled and the comments given to us were processed and prepared. Specific text analysis techniques have been applied in order to prepare the data in the best possible way in order to extract the most important information from them. All the operations performed for the data cleaning and for the extrafeature extraction are listed below. Each operation is iterated for all the 3000 comments of the data set.
Extraction of the first feature: length of the comment.
Extraction of the second feature: percentage of words written in CAPS-LOCK inside the comment. Calculated by the number of words written in CAPS-LOCK divided by the number of words in the comment.
Conversion of disguised bad words. An interesting function added to the preprocessing is the recognition of censored bad-words, i.e. bad-words where some of their middle letters are replaced by special character (symbol, punctuation...) to make it recognizable by an human but not by a computer. At this scope we don’t use a large vocabulary but it’s better a simple list of most common badwords censored (because only a small group of bad words is commonly censored). At this python function we pass an entire sentence creating a list splitting this by space. We scan the list of sentence words and we control if the first and last characters are letters and not number or symbols. Then we take this word without first and last letters and control if this middle sub-word is formed by special symbols/punctuation or by letter x (because ”x” is often used for hiding bad-words). If yes, this middle sub-word is deleted from the censored bad-word, taking the top and end part of this formed by letters. At the end we scan the list of bad-words and we control if this top and end part matching with one of this scanned bad-words. If yes, this is replaced by the real word.
Hashtag splitting. One of the most difficult cleaning phases is the Hashtag Splitting. For this we used a large dictionary of italian words in .csv format. First, we scan every word in this file and we control if these word is in the hashtag and then (for convenience we avoid the words of lenght 2) saving it in a list. In this phase will be taken also useless words not contextualized to the hashtag, so we will need to filter them. For this, first we sort all found words in decreasing length and we scan the list. So, starting to the first word on, we delete it from the hashtag. In this way the useless words in the list contained in larger words are found, saved in another list, and deleted from the beginning list containing all the words (both useful and useless) in the hashtag. In the final phase for each word in the resulting list we find its position within the hashtag and with this we create the real sentence, separating every word with a space. Removal of all the links from the comment. Editing of each word in the comment by this way: removal of nearby equal vowels, removal of nearby equal consonants if they are more than 2. Examples: from ”caaaaane” to ”cane”, from ”gallllina” to ”gallina”.
Extraction of the third feature: number of sentences inside the comment. By sentence we mean a list of words that ends with ’.’ or ’?’ or ’!’.
Extraction of the fourth feature: number of ’?’ or ’!’ inside the comment.
Extraction of the fifth feature: number of ’.’ or ’,’ inside the comment.
Translation of emoticons (for Twitter messages). Given the large presence of emoticons in Twitter messages, it was decided to translate the emoticons with the respective English translations. To do this, each sentence is scanned and if there are emoticons, these are translated into their corresponding meaning in English. Using the library unicode emoji,
Replacement of the abbreviations with the respective words, using a list of abbreviations created by ourselves.
Removal of articles, pronouns, prepositions, conjunctions and numbers.
Replacement of accented characters with their unaccented characters.
Lemmatization of each comment with the treetaggerwrapper library.
Extraction of the sixth feature : polarity of the message. This feature is compute using the SentiWordNet corpora and his APIs. Since SentiWordNet was created to find the polarity of sentences in English, each message is translated using TextBlob in English and the polarity is then calculated.
Extraction of the seventh feature : Percentage of spelling errors in the comment. To calculate a spelling error a word is compared with all the words of the Italian Vocabulary corpora; if the word is not present in the corpora there is a spelling error. Calculated by the number of spelling error divided by the number of words in the comment.
Replacement of spelling error: In parallel with the previous step every spelling error is replaced with the most similar word in the Italian Vocabulary corpora. The similarity between the wrong word and all the other is calculated using a function of SequenceMatcher library. The wrong word is replaced with the most similar word in Italian Vocabulary corpora.
Extraction of the eighth feature: number of bad words in the comment. Every word in the comment is compared with all the word in the Bad Words corpora; if the word is in the corpora it’s a bad word.
Extraction of the ninth feature: percentage of bad words. Calculated by the number of bad words divided by the number of words in the comment.
Extraction of the tenth feature : Polarity TextBlob. This value is compute using a TextBlob function that allows to calculate the polarity. Also in this case the message is translated into English.
Extraction of the final feature : Subjectivity TextBlob. Another value computed with a function in TextBlob. 5
Very briefly, Word Embedding turns text into numbers. This transformation is necessary because many Machine Learning algorithms don’t work with plain text but they require vectors of continuous values. Word Embedding has fundamental advantages in particular, it is a more efficient representation (dimensionality reduction) and also it is a more expressive representation (contextual similarity). So we have created a Word2Vec model for word embedding. For the training of the model, 200k messages were downloaded from several Facebook pages. These messages were preprocessed as explained in the previous section 4 and (in addiction with the messages provided by EVALITA’s team) were used to train the Word2Vec model. The trained model encode each word in a vector of 128 real numbers. Each sentence is instead encoded with a vector of 256 real numbers divided into two components of 128 elements: the first component is the vector sum of the coding of each word in the sentence, while the second component is the arithmetic mean. At this point each of the 3000 comments of the starting training set is a vector of 265 reals: 256 for the coding of the sentence and 9 for the previously calculated features. 6
The vectors obtained by the process described in
section 5 were used as input for training an
Artificial Neural Network - ANN.
Number of nodes of the hidden layers: 56, 128, 256, 512; Activation function of the hidden layers: sigmoid, relu, tanh, softplus;
Furthermore, the dropout was essential to prevent over-fitting. In fact, dropout consists to not consider neurons during the training phase of certain set of neurons which is chosen randomly. The dropout rate is set to 45%, meaning that the 45% of the inputs will be randomly excluded from each update cycle. As methods of estimation, cross-validation was used, partitioning the data into 10 disjoint subsets. As metrics for performance evaluation, the goodness of the model was analyzed by calculating True Positive, True Negative, False Positive and False Negative. From these the cost-sensitive measures precision, recall and f-score were calculated. These are the best results achieved with the training dataset of Facebook comments obtained during the cross validation:
Accuracy: 83.73%;
Recall: 0.817%; F1-Score: 0.856%;
F1-Score Macro: 0.856%; 7
After the release of the unlabelled test set, the new 2000 messages (1000 of them from Facebook and 1000 of them from Twitter) were cleaned as explained in section 4 and the respective features were extrapolated. Then, these new comments were added to the comment’s pool used to create the Word2Vec model, and a new Word2Vec model was created with the new pool. Finally, the 2000 comments were encoded as previously explained in section 5 in the 265 component vectors and these were the input of the neural network that classified them. From the training phase, two neural network models were built: one trained with the dataset of 3000 Facebook messages and the other trained with the dataset of 3000 Twitter messages. We call the first model VTfb and the second one VTtw. EVALITA’s task consisted in four sub-tasks that were:
HaSpeeDe-FB: test VTfb with the 1000 messages taken from Facebook; HaSpeeDe-TW: test VTtw with the 1000 messages taken from Twitter;
1000 messages taken from Facebook;
1000 messages taken from Twitter;
HaSpeeDe-FB HaSpeeDe-TW Cross-HaSpeeDe-FB Cross-HaSpeeDe-TW
VTfb VTtw VTfb VTtw In Table 1 we report the Macro-Average F1 score for each sub-task together with the differences with the best result obtained in the competition (column ”Distance” in the table). Compared with the results we had in the training phases (section 6), we would have expected better results in the HaSpeede-FB task. However, our system appears to be more general and not specifically targeted to a platform, in fact the differences in the other tasks are minimal. 8
In this paper we presented a system based on neural networks for the hate speech detection in social media messages in Italian language. Recognizing negative comments is not easy, as the concept of negativity is often subjective. However, good results have been achieved that are not so far from the results obtained by the best within the competition. The proposed system can certainly be improved, an idea can be to use clustering techniques to categorize the messages (cleaned and with the related features) in two subgroups (positive and negative) and then, for each comment, calculate how much this is more similar to negative comments or positive comments and add it as a feature.
The authors would like to thank prof. Valentina Poggioni who has helped and supported us in the development of the whole project. A special thanks to Manuela Sanguinetti, our shepherd in EVALITA competition for all the support she has given to us.