By1510 @ HaSpeeDe 2: Identification of Hate Speech for Italian Language in Social Media Data Tao Deng, Yang Bai, Hongbing Dai† School of Information Science and Engineering Yunnan University, Yunnan, P.R. China Dtao.top@gmail.com baiyang.top@gmail.com hbdai it@126.com Abstract 1 Introduction With the continuous development of computer and English. Hate speech detection has be- networks, social media users have increased year come a crucial mission in many fields. by year, social media has entered people’s daily This paper introduces the system of team life and becomes an indispensable part. More and By1510. In this work, we participate in more people use the Internet to express their opin- the HaSpeeDe 2 (Hate Speech Detection) ions and ideas on social media platforms. Some shared task which is organized within E- offensive, abusive, defamatory contents are easy to valita 2020(The Final Workshop of the 7th spread and incite hatred, and these negative con- evaluation campaign). In order to obtain tents can cause some bad effects. The simplest more abundant semantic information, we way is that people mark the report and then delete combine the original output of BERT-Ita the system warning, which can not be solved ef- and the hidden state outputs of BERT-Ita. ficiently. Therefore, an efficient way is urgently We take part in task A. Our model achieves needed to eliminate these negative effects. This an F1 score of 77.66% (6/27) in the tweets paper proposes a hate speech detection system, test set and our model achieves an F1 score which can better detect and mark these annoy- of 66.38% (14/27) in the news headlines ing contents. The HaSpeeDe 2 (Sanguinetti et al., test set. 2020) (Hate Speech Detection) shared task is orga- nized within Evalita 2020 (Basile et al., 2020), the 7th evaluation campaign of Natural Language Pro- Italiano. L’ individuazione dell’ in- cessing and Speech tools for Italian, which help citamento allodio diventata una mis- to detect whether the Italian language on Twit- sione cruciale in molti campi. Questo ter contains hate language, with the aim to reduce articolo introduce il sistema del team the spread of hate speeches and online harassment. By1510. In questo lavoro, partecipiamo (Waseem and Hovy, 2016) al task HaSpeeDe 2 che stato organiz- In this paper, we take part in task A in the zato allinterno di Evalita 2020. Per ot- HaSpeeDe 2 task. The BERT model we use is tenere informazioni semantiche pi abbon- dbmz1 trained on Italian data. In order to obtain danti abbiamo combinato loutput origi- more abundant semantic information, we extrac- nale di BERT Ita e gli output di hidden t the state of hidden layer outputs and we provide state di BERT Ita. Il sistema presentato a reference for the detection of the hate speech in partecipa al task A. Il nostro modello ot- the Italian language. The rest of the paper is orga- tiene un punteggio F1 di 77.66% (6/27) sui nized as follows. Section 2 briefly shows the re- dati di test da Twitter e un punteggio F1 di lated work for the identification of hate speeches. 66.38% (14/27) sui dati di test contenenti Section 3 elaborates on our approach. It shows the titoli di quotidiano. data set officially provided and architecture of our model. Section 4 describes the hyper-parameters and our results. Finally, Section 5 concludes our Copyright c 2020 for this paper by its authors. Use work. permitted under Creative Commons License Attribution 4.0 1 International (CC BY 4.0). https://huggingface.co/dbmdz Figure 1: our model. L12 H0 is hidden-state of the first token of the sequence(CLS token) at the output of the 12th hidden layer of the BERT-Ita. Similarly, L11 HO and L10 HO are the 11th and 10th hidden layers outputs of BERT-Ita respectively. [32, 768]/[32, 3072] is the output shape (batch size, hidden size) 2 Related Work classifier integration, which was verified by cross validation on Facebook and Twitter data sets, and Previously, machine learning (Davidson et al., the effect was obvious in offensive words. (Coraz- 2017; MacAvaney et al., 2019a), Bayesian method za et al., 2018) used recurrent neural network, n- (Miok et al., 2020; Fauzi and Yuniarti, 2018), sup- gram neural network and support vector machine port vector machine (MacAvaney et al., 2019b; to classify Twitter data sets, and its recurrent mod- Del Vigna12 et al., 2017), neural network (Bad- el had achieved good results. (Bianchini et al., jatiya et al., 2017; Zhang et al., 2018) and oth- 2018) proposed artificial neural network to anno- er methods were proposed for the identification tate and classify 3000 message data from Face- of hate speech. In the Hindi-English mixed lan- book and Twitter, and achieved good results. guage, (Bohra et al., 2018) et al. in parentheses used a supervised classification system to detect 3 Methodology the hate speech in the text in the code-mixed lan- 3.1 Data Description guage. The classification system used Character N-Grams, Word N-Grams, Punctuations, Negation In this work, we take part in task A, which is a Words, Lexicon and other feature vectors for clas- binary classification task aimed at determining the sification and training. The accuracy could reach presence or the absence of hateful content in the 71.7% with SVM, which proved to be a very ef- text towards a given target (among Immigrants, fective method for classification tasks. In Danish Muslims or Roma people). The organizers pro- language, (Sigurbergsson and Derczynski, 2019) vide the training set and test set. For the training developed four automatic classification systems set, it is from Twitter. For the test set, the organiz- to detect and classify hate speech in English and ers provide in-domain data and out-of-domain da- Danish, and proposed a method to automatically ta, which come from Twitter and news headlines, detect different types of the hate speech, which respectively. It can be seen from Table 1 that the achieved good results for the detection of English data set is slightly imbalanced. and Danish hate speeches. In English language, (Aroyehun and Gelbukh, 2018) used a linear base- 3.2 Our approach line classifier (nbsvm with n-grams) and improved As the train data is very limited we resort to a deep neural network model. transfer learning approach. That is, we take an For the Italian language, (Polignano et al., NLP model pre-trained(Peters et al., 2018; Rad- 2019) proposed an AlBERT o model based on ford et al., 2018; Devlin et al., 2019) on a large Hate Speech hidden layer of BERT-Ita. We concatenate pooler No HS (HS) output, L12 H0 , L11 H0 and L10 H0 into the train data 2766 4071 classifier. test data 622 641 (tweets) 4 Experiments and Results test data 181 319 (news headlines) 4.1 Preprocessing and Experiments Setup Table 1: Distribution of data set in the Task A. In the experiment, we try to preprocess the tex- t but we did not achieve the desired results. We Hyperparameters find that after preprocessing the Twitter data, the output hidden states=True F1-score of the model decreased on the validation max sequences length=100 set. We do not preprocess the data and we do not learning rate=1e-5 use an extra data set. In this work, the training adam epsilon=1e-8 set is split into the new training set and the val- Our Model per gpu train batch size=32 idation set by using the Stratified 5-Fold Cross- gradient accumulation steps=1 validation4 .The random seed is set 42 in Cross- epoch=8 validation. Due to the imbalance of datasets, the dropout=0.1 Stratified 5-Fold Cross-validation ensures that the proportion of samples in each category in each Table 2: Hyperparameters of the model in our ex- fold data set remains unchanged. During the train- periments. ing, the best weight of the model is saved in 8 e- pochs. Table 2 shows the hyperparameters used in corpus of texts and fine-tune it for a specific task our model. at hand. In this work, we used BERT-base-Italian- uncased(BERT-Ita)2 from Transformers library. It 4.2 Results and analysis is trained on the recent Wikipedia dump and vari- ous texts from the OPUS corpora3 collection. The In the experiment, we find that with the increase final training corpus has a size of 13GB and 2050 of the extra semantic features, the model can ob- million tokens. For classification tasks, the out- tain more abundant semantic information. Table 3 put of BERT-Ita (pooler output) is obtained by its shows the performance of the model for different last layer hidden state of the first token of the se- semantic features after getting the labels of the test quence (CLS token) further processed by a linear set.5 . layer and a Tanh activation function. However, the pooler output is usually not a good summary of the Task A semantic information. Therefore, we extract the test set of tweets(100%) hidden layer output of BERT-Ita to obtain more No HS HS abundant semantic information. No HS 489 152 (Jawahar et al., 2019) pointed that the hidden HS 119 503 layer of BERT encodes a rich hierarchy of linguis- Task A tic information, with surface features at the bot- test set of news headlines(100%) tom layer, syntactic features in the middle layer No HS HS and semantic features at the top layer. Therefore, No HS 312 7 we get abundant semantic information by extract- Hs 133 48 ing the extra semantic features which is the last three hidden layer outputs(L12 H0 , L11 H0 and Table 4: The confusion matrix of BERT- L10 H0 ) of BERT-Ita. We propose the following Ita+L12 HO in test sets. model which is shown in Figure 1. In the mod- el, we get L12 H0 , L11 H0 , L10 H0 from the top 4 https://scikit-learn.org/stable/modules/generated/sklearn .model selection.StratifiedKFold.html#sklearn.model selecti 2 https://huggingface.co/dbmdz/bert-base-italian-uncased on.StratifiedKFold 3 5 http://opus.nlpl.eu/ https://github.com/msang/haspeede/tree/master/2020 Task A Task A test set of tweets(100%) test set of news headlines(100%) Precision/Recall/Macro F1-score Precision/Recall/Macro F1-score BERT-Ita+L12 HO 78.61/78.58/78.54 78.69/62.16/61.18 BERT-Ita+L12 HO +L11 HO 75.50/77.27/77.16 78.13/62.23/62.76 BERT-Ita+L12 HO +L11 HO +L10 HO 77.80/77.72/77.66 72.07/65.74/66.38 (Our submitted model) Table 3: The performance of the model for these test sets. Task A sults on the tweets test set, but the results of our test set of tweets(100%) model are not good on the news headline data set. No HS HS There are many differences between the syntactic No HS 478 163 features of tweets and news headlines. For exam- HS 119 503 ple, there are many irregular expressions in tweet- Task A s, while news expressions are very standard. Our test set of news headlines(100%) model is only fine-tuned on the tweets data set, so No HS HS we think this affects the performance of the model No HS 289 30 on other types of data. Hs 107 74 5 Conclusion Table 6: The confusion matrix of BERT- In this work, this paper introduces the system pro- Ita+L12 HO +L11 HO F+L10 HO in test sets. posed for HaSpeeDe 2 shared task for identifying and classifying hate speeches on social media. We Task A enriched BERT-Ita with semantic information by test set of tweets(100%) extracting the extra semantic features. We find No HS HS that with the increase of semantic information, the No HS 463 178 performance of the model for identifying the hate HS 110 512 speech is also increasing. Finally, in the official e- Task A valuation, our model rank 6th (6/27) in the tweets test set of news headlines(100%) test set and 14th (14/27) in the news headlines test No HS HS set. In the future, we will focus on how to make No HS 310 9 the model learns more semantic information. Hs 128 53 Table 5: The confusion matrix of BERT- References Ita+L12 HO +L11 HO in test sets. Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseu- The confusion matrices (actual values are rep- do labeling. In Proceedings of the First Workshop resented by rows) are shown in Table 4, Table 5, on Trolling, Aggression and Cyberbullying (TRAC- Table 6. These tables show the performance of the 2018), pages 90–97. model on the test set as the extra semantic features Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, increase. In the tweets test set, we can see from and Vasudeva Varma. 2017. Deep learning for hate these tables that the ability of the model to detect speech detection in tweets. In Proceedings of the the hate speech is increasing as the extra seman- 26th International Conference on World Wide Web Companion, pages 759–760. tic features increase. Similarly, in the news head- lines test set, the ability of the model to detect the Valerio Basile, Danilo Croce, Maria Di Maro, and Lu- hate speech is also increasing. We think that with cia C. Passaro. 2020. Evalita 2020: Overview of the 7th evaluation campaign of natural language the increase of these extra semantic features, the processing and speech tools for italian. In Valeri- model can learn more semantic information. In o Basile, Danilo Croce, Maria Di Maro, and Luci- addition, we find that our model achieve good re- a C. Passaro, editors, Proceedings of Seventh Eval- uation Campaign of Natural Language Processing Kristian Miok, Blaz Skrlj, Daniela Zaharie, and Marko and Speech Tools for Italian. Final Workshop (E- Robnik-Sikonja. 2020. To ban or not to ban: VALITA 2020), Online. CEUR.org. Bayesian attention networks for reliable hate speech detection. arXiv preprint arXiv:2007.05304. Giulio Bianchini, Lore nzo Ferri, and Tommaso Giorni. 2018. Text analysis for hate speech detection in i- Matthew Peters, Mark Neumann, Mohit Iyyer, Mat- talian messages on twitter and facebook. EVALI- t Gardner, Christopher Clark, Kenton Lee, and Luke TA Evaluation of NLP and Speech Tools for Italian, Zettlemoyer. 2018. Deep contextualized word rep- 12:250. resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- Aditya Bohra, Deepanshu Vijay, Vinay Singh, ation for Computational Linguistics: Human Lan- Syed Sarfaraz Akhtar, and Manish Shrivastava. guage Technologies, Volume 1 (Long Papers), pages 2018. A dataset of hindi-english code-mixed social 2227–2237, New Orleans, Louisiana, June. Associ- media text for hate speech detection. In Proceedings ation for Computational Linguistics. of the second workshop on computational modeling of peoples opinions, personality, and emotions in so- Marco Polignano, Pierpaolo Basile, Marco de Gemmis, cial media, pages 36–41. and Giovanni Semeraro. 2019. Hate speech detec- tion through alberto italian language understanding Michele Corazza, Stefano Menini, Pinar Arslan, model. In NL4AI@ AI* IA. Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Comparing different super- Alec Radford, Karthik Narasimhan, Tim Salimans, and vised approaches to hate speech detection. Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech Manuela Sanguinetti, Gloria Comandini, Elisa Di Nuo- detection and the problem of offensive language. vo, Simona Frenda, Marco Stranisci, Cristina Bosco, arXiv preprint arXiv:1703.04009. Tommaso Caselli, Viviana Patti, and Irene Russo. 2020. HaSpeeDe 2@EVALITA2020: Overview of Fabio Del Vigna12, Andrea Cimino23, Felice DellOr- the EVALITA 2020 Hate Speech Detection Task. In letta, Marinella Petrocchi, and Maurizio Tesconi. Valerio Basile, Danilo Croce, Maria Di Maro, and 2017. Hate me, hate me not: Hate speech de- Lucia C. Passaro, editors, Proceedings of Seventh E- tection on facebook. In Proceedings of the First valuation Campaign of Natural Language Process- Italian Conference on Cybersecurity (ITASEC17), ing and Speech Tools for Italian. Final Workshop (E- pages 86–95. VALITA 2020), Online. CEUR.org. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Gudbjartur Ingi Sigurbergsson and Leon Derczynski. Kristina Toutanova. 2019. BERT: Pre-training of 2019. Offensive language and hate speech detection deep bidirectional transformers for language under- for danish. arXiv preprint arXiv:1908.04531. standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Zeerak Waseem and Dirk Hovy. 2016. Hateful sym- Computational Linguistics: Human Language Tech- bols or hateful people? predictive features for hate nologies, Volume 1 (Long and Short Papers), pages speech detection on twitter. In Proceedings of the 4171–4186, Minneapolis, Minnesota, June. Associ- NAACL student research workshop, pages 88–93. ation for Computational Linguistics. Ziqi Zhang, David Robinson, and Jonathan Tepper. M Ali Fauzi and Anny Yuniarti. 2018. Ensemble 2018. Detecting hate speech on twitter using a method for indonesian twitter hate speech detection. convolution-gru based deep neural network. In Eu- Indonesian Journal of Electrical Engineering and ropean semantic web conference, pages 745–760. Computer Science, 11(1):294–299. Springer. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3651–3657, Florence, Italy, July. As- sociation for Computational Linguistics. Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019a. Hate speech detection: Challenges and solutions. P- loS one, 14(8):e0221152. Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019b. Hate speech detection: Challenges and solutions. P- loS one, 14(8):e0221152.