English. The detection of hate speeches, over social media and online forums, is a relevant task for the research area of natural language processing. This interest is motivated by the complexity of the task and the social impact of its use in real scenarios. The task solution proposed in this work is based on an ensemble of three classification strategies, mediated by a majority vote algorithm: Support Vector Machine (Hearst et al., 1998) (SVM with RBF kernel), Random Forest (Breiman, 2001), Deep Multilayer Perceptron (Kolmogorov, 1992) (MLP). Each classifier has been tuned using a greedy strategy of hyper-parameters optimization over the ”F1” score calculated on a 5-fold random subdivision of the training set. Each sentence has been pre-processed to transform it into word embeddings and TF-IDF bag of words. The results obtained on the cross-validation over the training sets have shown an F1 value of 0.8034 for Facebook sentences and 0.7102 for Twitter. The code of the system proposed can be downloaded from GitHub: https: //github.com/marcopoli/ haspeede_hate_detect
Italiano. L’individuazione di discorsi
di incitamento all’odio sui social media
e sui forum on-line e` una sfida
rilevante per l’area di ricerca riguardante
l’elaborazione del linguaggio naturale.
Tale interesse e` motivato della complessita`
del processo e dell’impatto sociale del
suo utilizzo in scenari reali. La soluzione
proposta in questo lavoro si basa su un
insieme di tre strategie di classificazione
mediate da un algoritmo di votazione per
maggioranza: Support Vector Machine
In the current digital era, characterized by the large use of the Internet, it is common to interact with others through chats, forums, and social networks. Common is also to express opinions on public pages and online squares. These places of discussion are frequently transformed into ”fight clubs” where people use insults and strong words in order to support their ideas. The unknown identity of the writer is used as an excuse to fell free of consequences derived by attacking people only for their gender, race or sexual inclinations. A general absence of automatic moderation of contents can cause the diffusion of this phenomenon. In particular, consequences on the final user could be psychological problems such as depression, relational disorders and in the most critical situations also suicidal tendencies.
A recent survey of state of the art approaches
for hate speech detection is provided by
The hate speech detection strategy proposed in
HAnSEL has been developed for HaSpeeDe (Hate
Speech Detection) task organized within Evalita
2018
Facebook dataset can be used to classify the Facebook test set
Twitter dataset can be used to classify the Twitter test set Task 3: Cross-HaSpeeDe, which can be further subdivided into two sub-tasks:
only the Facebook dataset can be used to classify the Twitter test set
where only the Twitter dataset can be used to classify the Facebook test set The Facebook and Twitter datasets released for the task consist of a total amount of 4,000 comments/tweets, randomly split into development and test set, of 3,000 and 1,000 messages respectively. Data are encoded in a UTF-8 with three tab-separated columns, each one representing the sentence id, the text and the class (Fig. 1). id 8 36 text hs Io votero NO NO E NO 0
Matteo serve un colpo di stato. 1 The system proposed in this work is HanSEL: a system of Hate Speech detection through Ensemble Learning and Deep Neural Networks. We decided to approach the problem using a classic natural language processing pipeline with a final task of sentences classification into two exclusive classes: hate speech and not hate speech. The data provided by task organizers are obtained crawling social network, in particular, Facebook and Twitter. The analysis of the two data sources showed many possible difficulties to face in the case of using approaches based on Italian lexicons of hate speeches. In particular, we identified the following issues: Repeated characters: many words includes characters repeated many times for emphasizing the semantic meaning of the word. As an example, the words ”nooooo”, ”Grandeeeee”, ”ccanaleeeeeeeeeeeeeeee” are found in the training of Facebook messages.
Emoji: sentences are often characterized by emoji such as hearts and smiley faces that are often missing in external lexicons.
this particular elements are typical of the social network language and can introduce noise in the data processing task.
composed by only one word or in general, they are very short. Consequently, they are not expressive of any semantic meaning. The complexity of the writing style used in hate speech sentences guided us through the idea to do not use an approach based on standard lexicons and to prefer supervised learning strategies on the dataset provided by the task organizers.
We decide to represent each sentence as a
concatenation of a 500 features word embedding
vector and a 7,349 size bag of words for Facebook
messages and 24,866 size bag of words for Twitter
messages. In particular, the word-embedding
procedure used is word2vec introduced by Mikolov
We follow the same step of pre-processing
applied by Tripodi
Data cleansing: we transformed the words is lowercase and following we removed from each sentences links, hashtags, entities, and emoji
The normalized sentences are consequently tokenized using the TweetTokenizer of the NLTK library 1. For each sentence we averaged the word2vec vectors correspondent of each token, removing during each sum the centroid of the whole distributional space. This technique is used for mitigating the problems of loss of information due to the operation of averaging the semantic vectors.
The two bags of words (Facebook and Twitter) are, instead, created directly on the sentences without any pre-processing, also if during the tuning of the architecture we had tried some configurations that include bag of words without stop words, with lowercase letters and processed by Snowball stemmer algorithm 2 without obtaining breaking results. The n-gram size considered for the construction of the bag is in the range of 1 to 3. The final representation of each sentence of the dataset is consequently obtained concatenating the word2vec vector and the correspondent bag of words. Sentences too shorts that cannot be transformed into word2vec as a consequence of the absence of all the tokens of the sentence have been classified using only the bag of words representation.
HAnSEL is based on a classification process that uses three different classification strategies mediated by a hard majority vote algorithm. A stacking of classifiers with a Random Forest blender 1https://www.nltk.org/data.html 2http://snowball.tartarus.org/texts/quickintro.html has also been considered during the design of HAnSEL architecture but, the internal evaluation runs on 5-fold cross-validation of the training set showed us low performances of the approach. This analysis is not detailed more in this work due to the page limitations of it. In order to design the ensemble, we analyzed the performances of some of the most popular classification algorithms for the text categorization task. In particular, we considered:
Logistic regression with stochastic gradient descendent training (SGD). It has the advantage to be very efficient with large datasets considering, during the training, one instance per time independent by others. It uses the gradient descendent as optimization function for learning the optimal weight of the separation function of the distributional space of items. In literature, it has been successfully used for tasks of text classification, especially with binary classification problems.
C-Support Vector Classification (SVC). It is the standard Support Vector Machine algorithm applied for the classification task. It is a powerful approach that supports linear and non-linear classification function. Moreover, through the C parameter, it is possible to decide how much the margin of classification could be significant and consequently sensitive to outliers. The implementation is based on libsvm, and we evaluated different configurations of the algorithm: polynomial function with 2 and 3 degree, RBF kernel and different values of the C parameters.
K-nearest neighbors vote (KNN). This classic and versatile algorithm is based on the concept of similarity among items according to a distance metric. In particular, for an unseen item, the k most similar items of the training set are retrieved, and the class, provided as output, is obtained by the majority vote of the neighborhoods. Despite the simplicity of the algorithm it is often used in tasks of text classification.
A decision tree classifier (DT). This approach is another popular strategy of classification used especially when it is required to visualize the model. The DT algorithm works splitting items into a different path of the tree according to their feature values. In order to classify an unseen item, the three is navigated until reaching the leaf and then the ration of training items of class k in that leaf is used as a class probability.
Random forest classifier (RF). It is an ensemble of Decision Trees trained on different batches of the dataset that uses averaging to improve the predictive accuracy and control over-fitting. A typical parameter is the number of threes to use in order to balance the precision of the algorithm and the randomness to obtain a good level of generalization of the model.
Multi-layer Perceptron classifier (MLP). This model is a classical architecture of a deep neural network. It is composed by one layer of inputs, one layer of linear threshold units (LTU) as output and many hidden layers of an arbitrary number of LTU plus one bias neuron fully connected each other. The weights learned by each neuron (perceptron) are updated through back-propagation using a gradient descendent strategy. Important parameters to configuring are the number of hidden layers, the number of training epochs and the L2 penalty (regularization term) parameter.
We evaluated the performance of the algorithms just described using a default configuration and a 5-fold cross validation over the Facebook training set. Moreover, we set the random seed equal to 42 for obtaining at each run always the same folder subdivision.
Tab. 3 shows the results obtained by the different classification algorithms during their preliminary analysis considering the macro F1 score as in the task specifications. The values obtained do not point out significant statistical differences among the approaches, but we decided to investigate more the top three scored algorithms: SVM with an RBF kernel, Random Forest with 300 trees, MLP with 2,000 hidden layers. In general, we observed that linear algorithms obtain a high score for the task supporting our idea that linguistic features are enough for defining a clear separation among the sentences of hate and not hate speeches. In order to identify an optimal configuration of the algorithms, we trained our models using a greedy search approach. For each algorithm, we performed 100 training runs with pa
Task 3.1 Task 3.2 rameters randomly selected from a range of values preliminary defined. Each run has been evaluated, considering the macro F1 score, on the training set using the same strategy of cross-validation already described before. At the end of the 100 runs the model that achieve the best results has been stored and later used in our final ensemble of classifiers. The final configurations obtained for the three strategies are the following:
SVC(C=1.76800710431488, gamma=0.1949764030136127, kernel=’rbf’) RandomForestClassifier(bootstrap=False, max depth=30,max features=’sqrt’, min samples leaf=2,min samples split=2, n estimators=200, warm start=False) MLPClassifier(alpha=0.5521952082781035, early stopping=False, hidden layer sizes=2220, learning rate init=0.001, max iter=184,solver=’adam’, warm start=False)
The models are consequently used in a voting classifier configured for using a hard majority vote algorithm. The ensemble obtains an F1 value of 0.8034 for Facebook sentences and 0.7102 for Twitter using the 5-fold subdivision of the training sets. The implementation of the system has been realized into Python language and using the scikitlearn 0.20 machine learning library 3. 4
HanSEL has been used for classifying the data provided as a test set for each of the three specialized tasks of HaSpeeDe competition. Tab. 2 shows the final results obtained by our system in the challenge. It is possible to observe that the system well performed for Task 1 and Task 3.2 which involve the classification of Facebook messages. In particular, it emerges that HanSEL performs better fot hate speeches sentences than for not hate speeches probably a consequence of the presence of many clear hate words used in this type of messages such as ”sfigati” and ”bugiardo” in that category of textual sentences. A symmetrical situation is obtained for Task 2 and Task 3.2 that involves Twitter messages. In this scenario, the significant use of specific hashtags, irony, and entities instead of clear hate words has made difficult the identification of hate speeches. The cross-classification task has, moreover, stressed the generalization of the system. It has been observed that the writing style of the two social networks strongly influences the classification performance, especially when the models are trained on a small training set, as in our case. Finally, the optimization of the models inside the ensemble has been stressed more on the Facebook dataset consequently overfitting on the characteristics of that type of messages. The outcomes achieved for the challenge 3http://scikit-learn.org/stable/ allow us to deduce important consideration for further developments of the system. In particular, we consider essential to mix the two datasets in order to allow the models to generalize better considering the two different sources of data. Moreover, extra features regarding hashtags, entities, and links can be helpful for obtaining better results with Twitter messages. 5
The HaSpeeDe competition has been a perfect scenario for developing and testing solutions for the social problem of hate speeches on social media and, in particular, for them in the Italian language. In our work, we presented HAnSEL a system based on an ensemble of classifiers that includes the Support Vector Machine algorithm, Random Forests, and a Multilayers Perceptron Deep Neural Network. We formalize messages as a concatenation of word2vec sentence vectors and a TF-IDF bag of words. Results showed the efficacy of the solution in a scenario that uses clear offensive words such as Facebook messages. On the contrary, there is a large margin of improvements for the classification of Tweets. The future direction of the work will surely investigate the use of more data and semantic features for allowing classification methods to create a more general model.