Hate Speech Detection through AlBERTo Italian Language Understanding Model

Abstract

The task of identifying hate speech in social networks has recently attracted considerable interest in the community of natural language processing. This challenge has great importance for identifying cyberattacks on minors, bullying activities, misogyny, or other kinds of hate discriminations that can cause diseases. Identifying them quickly and accurately can, therefore, help to solve situations that are dangerous for the health of the attacked people. Numerous national and international initiatives have addressed this problem by providing many resources and solutions to the problem. In particular, we focus on the Hate Speech Detection evaluation campaign (HaSpeeDe) held at Evalita 2018. It proposes an evaluation campaign with the aim of developing strategies for identifying hate speeches on Twitter and Facebook written in the Italian language. The dataset released for the task has been used by the classi cation approach proposed in this work for demonstrating that it is possible to solve the task e ciently and accurately. Our solution is based on an Italian Language Understanding model trained with a BERT architecture and 200M of Italian Tweets (AlBERTo). We used AlBERTo for ne-tuning a classi cation model of hate speech, obtaining state of the art results considering the best systems presented at the HaSpeeDe workshop. In this regard, AlBERTo is here proposed as one of the most versatile resources to be used for the task of classi cation of Social Media Textual contents in the Italian Language. The claim is supported by the similar results obtained by AlBERTo in the task of sentiment analysis, and irony detection demonstrated in previous works. The resources need for ne-tuning AlBERTo in these classi cation tasks are available at: https://github.com/marcopoli/AlBERTo-it

Keywords

• Language Understanding Model
• AlBERTo
• Hate Speech
• Classi cation
• Machine Learning
• Deep Learning

Introduction and Motivations

Hate speeches are characterized by their wide di usion on the web and by the anonymity of the author, which makes this type of problem risky and relevant for the community. These messages can be against groups of people, such as those concerning discriminations about religion, race, and disability, or to a speci c person. In addition, hate messages are characterized by di erent facets that are very di erent from each other and which give rise to a wide and varied problem. The interpretation of a message as hate or not is subject to a strong cultural and social in uence by making the same message hatefully for some subjects (e.g. them from a speci c country) and not hatefully for others. Hate Speech (HS) is, consequently, a multi-faceted problem with strong cultural and social intersections. The lexicon used in that messages is di cult to be found in a standard dictionary and with many lexical variations making approaches of classi cation based only on dictionaries unsuccessful. Therefore, the automatic identi cation of hate messages is often a complex and intrinsically multidisciplinary task, including the research areas of natural language processing (NLP), psychology, law, social sciences, and many more. The hate speech detection is a challenging task that gains hight interest by private industries and public institutions to be able to remove potentially illegal contents quickly from the Web and to reduce the connected risk to remove legal content unjustly. This has made it interesting for us to apply an innovative classi cation model based on a language understanding model for the Italian language (AlBERTo [ 29 ].) to obtain promising results for the task.

The classi cation model of this work is part of a wider national project, \Contro l'odio"1, that aims to monitor, classify and summarize in statistics the hate messages in Italian identi ed via Twitter. \Contro l'odio" is a project for countering and preventing racist discrimination and HS in Italy, in particular focused against immigrants. On the one hand, the project follows and extends the research outcomes emerged from the `Italian Hate Map project' [ 23 ], whose goal was to identify the most-at-risk areas of the Italian country, that is to say, the areas where the users more frequently publish hate speech, by exploiting semantic analysis and opinion mining techniques. On the other hand, \Contro l'odio" bene ts from the availability of annotated corpora for sentiment analysis, hate speech detection and related phenomena such as aggressiveness and o ensiveness, to be used for training and tuning the HS detection tools [ 31,27 ]. The project brings together the competences and active participation of civil society organizations Acmos2 and Vox3, and two academic research groups, respectively from the University of Bari and Turin. 2

Related Work

The interest of the scienti c community in the task of identifying hate speech and related phenomena such as misogyny, cyberbullying, and abusive language has been growing since 2016. Events such as HatEval 2019 [ 3 ], AMI at IberEval 2018 [ 15 ], HaSpeeDe 2018 [ 9 ] and AIM 2018 [ 16 ] at EVALITA 2018 have contributed to the emergence of a strong community of reference, methods, resources and tools to address this complex task. For what concerns Italian a few resources

1 https://controlodio.it/ 2 http://acmos.net/ 3 http://www.voxdiritti.it/

have been recently developed drawn from Twitter [ 31,27 ] and Facebook [ 13 ], where the annotation of hateful contents also extends the simple markup of HS. A multilingual lexicon of hate words has also been developed [ 5 ] called Hurtlex4. It is divided into 17 categories such as homophobic slurs, ethnic slurs, genitalia, cognitive and physical disabilities, animals, and more.

A recent survey of state of the art approaches for hate speech detection is provided by Schmidt et al. [ 33 ]. The most common systems of speech detection are based on algorithms of text classi cation that use a representation of contents based on "surface features" such as them available in a bag of words (BOW) [ 11,37,36,34 ]. A solution based on BOW is e cient and accurate, especially when n-grams have been extended with semantic aspects derived by the analysis of the text. [ 11 ] describe an increase of the classi cation performances when features such as the number of URLs, punctuations and not English words are added to the vectorial representation of the sentence. [ 35 ] proposed, instead, to add as a feature the number of positive, negative, and neutral words found in the sentence. This idea demonstrated that the polarity of sentences positively supports the classi cation task. These approaches su er from the lack of generalization of words contained into the bag of words, especially when it is created through a limited training set. In particular, terms found in the test sentences are often missing in the bag. More recent works have proposed word embeddings [ 19 ] as a possible distributional representation able to overcome this problem. This representation has the advantage to transform semantically similar words into a similar numerical vector (e.g. Word2Vec). Word embeddings are consequently used by classi cation strategies such as Support Vector Machine and recently by deep learning approaches such as deep recurrent neural networks [ 20 ].

Limits of such technologies as Word2Vec [ 22 ], Glove [ 25 ], and FastText [ 8 ] fall into the lack of use of context of terms when such representation is built (contextfree). This means that each term has only a single wordembedding representation in the distribution space, and di erent concepts related to the same term are not represented. New strategies such as ELMO [ 26 ], GPT/GPT-2 [ 30 ], and BERT [ 14 ] overcome this limit by learning a language understanding model for a contextual and task-independent representation of terms. In particular, these models are trained to predict the totality or a span of the starting sentence. This allows obtaining a model able to predict, from a speci c context (often both previous and subsequent), the most probable word from its vocabulary. Recently, several articles have demonstrated the e ectiveness of this technique in almost all NLP tasks in the English language, and recently, some multilingual models have been distributed. This entails signi cant limitations related to the type of language learned (related to the document style) and the limit of vocabulary extracted. These reasons have led us to create the equivalent of the BERT model for the Italian language and speci cally on the language style used on social networks: alBERTo [ 29 ].

4 http://hatespeech.di.unito.it/resources.html

The classi er proposed in this work about HS is based on AlBERTo, demonstrating that its ne-tuned version is suitable for the task and it obtains better results than them presented ad HaSpeeDe 2018 evaluation campaign. 3

AlBERTo-HS classi cation model

The aim of this work is to create a classi cation model able to accurately classify HS contents written in the Italian Language on Social Network such as Facebook and Twitter. The analysis of state of the art shown that the main strategies for facing these challenges, on the English language, are currently based on a pre-trained language understanding models. Them, even in their multilingual version, are not suitable for an use with data completely in a single language and with a writing style di erent from that of books and encyclopedic descriptions. It is well known that the language used on social networks is di erent from the formal one as consequence of the presence of mentions, uncommon terms, links, and hashtags that are not present elsewhere. AlBERTo [ 29 ] wants to be the rst Italian language understanding model to represent a style of writing of social networks, Twitter in particular, written in Italian. The ne-tuned classication model proposed in this work is based AlBERTo derived by the software code distributed through GitHub by Devlin et al. [ 14 ] 5 under the concession of Google. It has been suitably modi ed to be learned without consequences on text spans containing typical social media characters including emojis.

The core deep learning structure of BERT and AlBERTo is a 12x Transformer Encoder, where for each input, a percentage of terms is hidden and then predicted for optimizing network weights in back-propagation. This strategy of learning is commonly named "masked learning". In AlBERTo we implement only the "masked learning" strategy, excluding the one based on "next following sentence". This is a crucial aspect to be aware of because, in the case of tweets, we do not have cognition of a ow of tweets as it happens in a dialog. For this reason, we are sure enough that our AlBERTo is not suitable for the task of question answering where this property is essential to have been learned by the model. On the contrary, it is good enough to be used in tasks of classi cation and predictions. In order to tailor the tweet text to BERT's input structure, it has been necessary to carry out pre-processing operations. More speci cally, using Python as the programming language, two libraries were mainly adopted: Ekphrasis [ 6 ] and SentencePiece 6 [ 18 ]. Ekphrasis is a famous tool for performing an NLP pipeline on text extracted from Twitter. It has been used for: { Normalizing URL, emails, mentions, percents, money, time, date, phone numbers, numbers, emoticons; { Tag and unpack hashtags.

The normalization phase consists in replacing the term with a xed one in the style of < [entity type] >. The tagging phase consists of annotating hashtags

5 https://github.com/google-research/bert/ 6 https://github.com/google/sentencepiece

by two tags < hashtag > ::: < =hashtag > representing its beginning and end in the sentence. Whenever possible, the hashtag has been unpacked into known words. For making the text clean and easily readable by the network, it has been returned to its lowercase form and all characters except emojis, !, ? and accented characters have been deleted.

SentencePiece is a segmentation algorithm used for learning in an unsupervised and language independent way the best strategy for splitting text into terms for language models. It can process till 50k sentences per seconds and to generate an extensive vocabulary. It includes in it the most common terms in the training set and the subwords which occur in the middle of words, annotating them with '##' in order to be able to encode also slang, incomplete or uncommon words. SentencePiece produced also a tokenizer used for generating a list of tokens for each tweet lately processed by the BERT "create pretraining data.py" module. The dataset used for the learning phase of AlBERTo is TWITA [ 4 ] a huge corpus of Tweets in the Italian language collected from February 2012 to September 2015 from Twitter o cial streaming API. In our con guration, we randomly selected 200.000.000 of Tweets removing re-tweets , and processing them with the pipeline of pre-processing previously described. The AlBERTo classi cation model is the basis for any single-label or multi-label classi cation task. For the speci c task of content classi cation of Hate speech, we will carry out a subsequent phase of ne-tuning and adaptation of the model to domainspeci c data. This allows us to obtain a classi er that exploits the language knowledge obtained during the learning phase on the generic data and the speci c domain characteristics learned during the ne-tuning phase. The ne-tuning phase is con gured as a new training of AlBERTo with a number of epochs sufciently small not to over t the model on the new data provided (usually from 3 to 15 epochs). This process allows us to vary the weights of the last layers of the model in order to predict correctly the content provided in the testing phase. We named the ne-tuned version of AlBERTo for Hate Speech as AlBERTo-HS. 4

Evaluation

In order to evaluate Alberto-HS with contents produced by real users on social networks, written in the Italian language, we decided to use the data released for the evaluation campaign HaSpeeDe [ 9 ] at EVALITA 2018. This choice was made considering that most of the available state of the art datasets are in English or focused only on data collected from a single social media site such as Facebook, Twitter, and others. The HaSpeeDe evaluation campaign was carried out by dividing the problem into four di erent tasks: { HaSpeeDe-FB: where the goal is to train the model and predict if the contents are HS on data extracted from Facebook; { HaSpeeDe-TW: where the goal is to train the model and predict if the contents are of HS on data extracted from Twitter; { Cross-HaSpeeDe FB: where the goal is to train the model on data collected from Facebook and predict if the contents are of HS on data extracted from Twitter; { Cross-HaSpeeDe TW: where the goal is to train the model on data collected from Twitter and predict if the contents are of HS on data extracted from Facebook; It is interesting to note that in the rst two tasks, the model must be able to classify data coming from the same information source as the training phase. Unlike the two "Cross" tasks, the data to be classi ed are di erent from those used for the test, making the task of the classi er more challenging due to the di erences in writing styles of the two platforms. In fact, not only are twitter data shorter, containing mentions, hashtags, and retweets, but overall, they are also less HS than Facebook data (only 32% compared to 68% for Facebook). 4.1

Dataset and Metrics Facebook dataset is collected from public pages on Facebook about newspapers, public gures, artists and groups on heterogeneous topics. More than 17,000 comments were collected from 99 posts and subsequently annotated by 5 bachelor students. The nal dataset released consists of 3000 training phrases (1618 not HS, 1382 HS) and 1000 test phrases (323 not HS, 677 HS). Twitter dataset is part of the Hate Speech Monitoring program, coordinated by the Computer Science Department of the University of Turin with the aim at detecting, analyzing and countering HS with an inter-disciplinary approach [ 10 ]. Data were collected using keywords related to the concepts of immigrants, Muslims and Rome. Data are annotated partly by experts and partly by Figure Eight contributors. Also for this dataset 3000 training tweets were released (2028 not HS and 972 HS) and 1000 test tweets (676 not HS and 324 HS).

The evaluation metrics used in HaSpeeDe campaign are the Precision, Recall and F1-measure classics. Since the two classes (HS and not HS) are unbalanced within the datasets, the F1 metric has been calculated separately on the two classes and then macro-averaged. 4.2

AlBERTo-HS

ne-tuning We ne-tuned AlBERTo two di erent times, in order to obtain one classi er for each di erent dataset available as a training set. In particular, we created one classi er for the HaSpeeDe-FB and the Cross-HaSpeeDe FB tasks using Facebook training data and one for the HaSpeeDe-TW and the Cross-HaSpeeDe TW using the Twitter training set. The ne-tuning learning phase has been run for 15 epochs, using a learning rate of 2e-5 with 1000 steps per loops on batches of 512 examples. The ne-tuning process was last 4 minutes every time.

Systems and baseline HaSpeeDe has received strong participation from the scienti c community and therefore a large number of solutions to the task have been proposed [ 9 ].

GRCP [ 24 ] The authors developed a Bi-LSTM Deep Neural Network with an Attention-based mechanism that allows to estimate the importance of each word; the weight vector is then used with another LSTM model to classify the text.

HanSEL [ 28 ] The system proposed is based on an ensemble of three classi cation strategies (Support Vector Machine with RBF kernel, Random Forest and Deep Multilayer Perceptron), mediated by a majority vote algorithm. The social media text is represented as a concatenation of word2vec vectors and a TF-IDF bag of words.

InriaFBK [ 21 ] The authors implemented three di erent classi er models: RNN, n-gram based and linear SVC.

ItaliaNLP [ 12 ] Participants used a newly-introduced model based on a 2layer BiLSTM which exploits multi-task learning with additional data from the 2016 SENTIPOLC task [ 2 ].

Perugia [ 32 ] The participants' system uses a document classi er based on a SVM algorithm. The features used by the system are a combination of FastText word embeddings and other 20 syntactical features extracted from the text.

RuG [ 1 ] The authors proposed two di erent classi ers: a SVM based on linear kernel and an ensemble system composed of an SVM and a CNN combined by a logistic regression meta-classi er.

sbMMMP The authors tested two di erent systems. The rst one is based on an ensemble of CNNs, whose outputs are then used as features by a metaclassi er for the nal prediction. The second system uses a combination of CNN and a GRU.

StopPropagHate [ 17 ] The authors use a classi er based on RNN with a binary cross-entropy as loss function. In their system, each input word is represented by a 10000-dimensional vector which is a one-hot encoding vector.

VulpeculaTeam [ 7 ] According to the description provided by participants, a neural network with three hidden layers was used, with word embeddings trained on a set of previously extracted Facebook comments.

For all tasks, the baseline score has been computed as the performance of a classi er based on the most frequent class. 4.4

Discussion of results The evaluation of the results obtained by the AlBERTo-HS classi er proposed in this work was carried out using the o cial evaluation script released at the end of the campaign 7. Consequently, all the results obtained are replicable and comparable with those present in the nal ranking of HaSpeeDe.

7 http://www.di.unito.it/ tutreeb/haspeede-evalita18/data.html

NOT HS HS Precision Recall F-score Precision Recall F-score Macro-Avg F-score 0.2441 0.8410 0.8288 0.8002 0.7841 0.7751 0.7738 0.7554 0.7428 0.7147 0.6532 0.2424

From the previous tables of results, it is possible to observe how AlBERToHS succeeds in obtaining a state of the art results for two tasks out of four. The di erences with other systems proposed in the evaluation campaign are about its simplicity to be applied. A simple ne-tuning phase of AlBERTo on domain data allows us to obtain very encouraging results. It is therefore interesting to note that the entire process of pre-processing and ne-tuning lasts a few minutes, and it can be used for obtaining excellent results for a wide variety of classi cation tasks. In particular, the model is able to adapt in an excellent way to annotated data (with the risk of over tting) producing excellent results if used in the same application domain of the tuning phase. This is the case with the results obtained for the HaSpeeDe-FB and HaSpeeDe-TW tasks.

Looking at the results obtained for the classi cation of data coming from Facebook (Tab. 1), it is possible to observe how the classi er is able to capture the characteristics of the social language through the ne-tuning phase. In particular, it is able to move its learned weights from them obtained parsing the original training language based on Twitter to the one used on Facebook. AlBERTo-HS obtains better performances than those of other participants in the evaluation campaign, with regard to the precision in identifying the posts not hate (0.8603), and the recall of those of hate (0.9453). The high value of recall for hate messages allows us to assume that, on Facebook, they are characterized by speci c thematics that make the classi cation task more inclusive at the cost of accuracy, especially when not explicit hate messages are faced. As an example, the message "Comunque caro Matteo se non si prendono provvedimenti siamo rovinati." is classi ed as a hate message even if the annotators have considered it to be not a hate message. In this example, it is clear that a basis of hate is present in the ideas of the writer, even if it is not complicated by what he writes. In other cases, words like "severe" have tricked the model into classifying clearly neutral messages like the following as hate messages: "Matteo sei la nostra voce!!! Noi donne non possiamo fare un cavolo! !! Leggi piu severe!". Nevertheless, the average F1 score higher than 0.8410, show us that, unlike in Twitter, the use of more characters available for writing allows people to be more verbose and, therefore, more comfortable to identify. Table 2 shows the results obtained for the classi cation of tweets. Here the values are not so di erent from the rst in the ranking during the evaluation campaign Haspeede even if the average value of F1 obtained of 0.8023 proves to be the best. This suggests that the presence in the tweets of particular characters and implicitly of hate, the brevity of the latter, and the increase in the number of ironic tweets make the task more complicated than the previous one.

As far as "Cross" classi cation problems are concerned, the results are not guaranteed. In Tab. 3 it can be observed that the model has not been able to correctly abstract from the domain data, obtaining not very good results for the classi cation in a di erent domain. In particular, the model trained on Facebook is able to obtain a score of 0.4750 of F1 on Twitter test data. A similar situation is repeated for the results in Tab. 4 where for the task Cross-HaSpeeDe TW the model is able to generalize slightly better than before but still gets the second place in the ranking. These results con rm the di culty of the Cross tasks and the drop in performance that is obtained through a transfer-learning strategy like the one adopted here. The great di erences in writing styles used on the two social networks do not allow the model to adapt properly to the domain of application if ne-tuned on di erent stylistic data. So that AlBERTo is not able to grasp those particularities of the language to be used in the classi cation phase.

In any case, we want to observe how it has been possible to obtain an excellent result of classi cation by merely carrying out a phase of ne-tuning on the model. To this end, we will consider as future works those of making a further comparison with other language understanding models such as GPT2, XLNet, RoBERTa trained on the Italian language with the aim of verifying if they can be more robust to the changes in the writing style of the text to be classi ed. 5

Conclusion

The problem of hate speech is strongly perceived in online communities because of its repercussions on the quality of life of hate victims. It is therefore of great interest to both public and private organisations to be able to quickly identify and remove hate messages. Numerous national and international initiatives have been carried out in recent years, especially for the English language, leaving the Italian language with few resources to address the problem. In this work we have proposed a simple model of classi cation obtainable through a quick ne-tuning phase of a wider language understanding model pre-trained on the Italian language (AlBERTo). This model was evaluated on the data released for the HaSpeeDe evaluation campaign held at the EVALITA 2018 workshop. Data containing phrases extracted from Facebbok and Twitter were classi ed according to four di erent tasks. The rst two involved training the model on data from the same domain as the test data. On the contrary, the last two "Cross" tasks involved a classi cation on data from a domain di erent from the training one. The results obtained showed excellent performances when the model is evaluated on data coming from the same distribution of training data. On the contrary, good performances in this transfer learning task are not guaranteed due to the great stylistic di erences of the language used on di erent online platforms such as Facebook and Twitter. Future work will focus on the possibility of learning a model that includes data from di erent online sources so as to make it more complete and robust to stylistic variations. 6

Acknowledgment

This work is funded by project "DECiSION" codice raggruppamento: BQS5153, under the Apulian INNONETWORK programme, Italy.

References

1. Bai , X. , Merenda , F. , Zaghi , C. , Caselli , T. , Nissim , M. : Rug@ evalita 2018 : Hate speech detection in italian social media . In: EVALITA@ CLiC-it ( 2018 )

2. Barbieri , F. , Basile , V. , Croce , D. , Nissim , M. , Novielli , N. , Patti , V. : Overview of the evalita 2016 sentiment polarity classi cation task . In: Proceedings of third Italian conference on computational linguistics (CLiC-it 2016 ) & fth evaluation campaign of natural language processing and speech tools for Italian . Final Workshop (EVALITA 2016 ) ( 2016 )

3. Basile , V. , Bosco , C. , Fersini , E. , Nozza , D. , Patti , V. , Pardo , F.M.R. , Rosso , P. , Sanguinetti , M. : Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter . In: Proceedings of the 13th International Workshop on Semantic Evaluation . pp. 54 { 63 . Association of Computational Linguistics ( 2019 )

4. Basile , V. , Lai , M. , Sanguinetti , M. : Long-term social media data collection at the university of turin . In: Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ). pp. 1 { 6 . CEUR-WS ( 2018 )

5. Bassignana , E. , Basile , V. , Patti , V. : Hurtlex: A multilingual lexicon of words to hurt . In: Proceedings of the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ), Torino, Italy, December 10-12 , 2018 . CEUR Workshop Proceedings , vol. 2253 . CEUR-WS.org ( 2018 ), http://ceur-ws. org/ Vol- 2253 /paper49. pdf

6. Baziotis , C. , Pelekis , N. , Doulkeridis , C. : Datastories at semeval -2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis . In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval2017) . pp. 747 { 754 . Association for Computational Linguistics, Vancouver, Canada ( August 2017 )

7. Bianchini , G. , Ferri , L. , Giorni , T. : Text analysis for hate speech detection in italian messages on twitter and facebook . In: Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018 ) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ), Turin, Italy, December 12-13 , 2018 . ( 2018 ), http://ceur-ws. org/ Vol- 2263 /paper043.pdf

8. Bojanowski , P. , Grave , E. , Joulin , A. , Mikolov , T. : Enriching word vectors with subword information . Transactions of the Association for Computational Linguistics 5 , 135 { 146 ( 2017 )

9. Bosco , C. , Felice , D. , Poletto , F. , Sanguinetti , M. , Maurizio , T. : Overview of the evalita 2018 hate speech detection task . In: Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018 ). vol. 2263 , pp. 1 { 9 . CEUR ( 2018 )

10. Bosco , C. , Viviana , P. , Bogetti , M. , Conoscenti , M. , Ru o, G., Schifanella , R. , Stranisci , M. : Tools and Resources for Detecting Hate and Prejudice Against Immigrants in Social Media . In: Proceedings of First Symposium on Social Interactions in Complex Intelligent Systems (SICIS) , AISB Convention 2017 , AI and Society ( 2017 )

11. Chen , Y. , Zhou , Y. , Zhu , S. , Xu , H. : Detecting o ensive language in social media to protect adolescent online safety . In: Privacy, Security, Risk and Trust (PASSAT) , 2012 International Conference on and 2012 International Confernece on Social Computing (SocialCom) . pp. 71 { 80 . IEEE ( 2012 )

12. Cimino , A. , De Mattei , L. , Dell'Orletta , F. : Multi-task learning in deep neural networks at evalita 2018 . In: EVALITA@ CLiC-it ( 2018 )

13. Del Vigna , F. , Cimino , A. , Dell'Orletta , F. , Petrocchi , M. , Tesconi , M. : Hate Me, Hate Me Not: Hate Speech Detection on Facebook . In: Proceedings of the First Italian Conference on Cybersecurity (ITASEC17) ( 2017 )

14. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : BERT: Pre-training of deep bidirectional transformers for language understanding . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 4171 { 4186 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 ), https://www.aclweb.org/anthology/N19-1423

15. Fersini , E. , Nozza , D. , Rosso , P. : Overview of the evalita 2018 task on automatic misogyny identi cation (ami) . In: Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA'18) , Turin, Italy. CEUR. org ( 2018 )

16. Fersini , E. , Nozza , D. , Rosso , P. : Overview of the evalita 2018 task on automatic misogyny identi cation (AMI) . In: Caselli, T. , Novielli , N. , Patti , V. , Rosso , P. (eds.) Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018 ) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ), Turin, Italy, December 12-13 , 2018 . CEUR Workshop Proceedings , vol. 2263 . CEURWS.org ( 2018 ), http://ceur-ws. org/ Vol- 2263 /paper009.pdf

17. Fortuna , P. , Bonavita , I. , Nunes , S. : Merging datasets for hate speech classi - cation in italian . In: Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018 ) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ), Turin, Italy, December 12-13 , 2018 . ( 2018 ), http://ceur-ws. org/ Vol- 2263 /paper037.pdf

18. Kudo , T. : Subword regularization: Improving neural network translation models with multiple subword candidates . arXiv preprint arXiv: 1804 . 10959 ( 2018 )

19. Le , Q. , Mikolov , T. : Distributed representations of sentences and documents . In: International Conference on Machine Learning . pp. 1188 { 1196 ( 2014 )

20. Mehdad , Y. , Tetreault , J.: Do characters abuse more than words? In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue . pp. 299 { 303 ( 2016 )

21. Michele , C. , Stefano , M. , Pinar , A. , Sprugnoli , R. , Elena , C. , Sara , T. , Serena , V. : Comparing di erent supervised approaches to hate speech detection . In: EVALITA 2018 . pp. 230 { 234 . aAccademia University Press ( 2018 )

22. Mikolov , T. , Sutskever , I. , Chen , K. , Corrado , G.S. , Dean , J. : Distributed representations of words and phrases and their compositionality . In: Advances in neural information processing systems . pp. 3111 { 3119 ( 2013 )

23. Musto , C. , Semeraro , G., de Gemmis, M. , Lops , P. : Modeling community behavior through semantic analysis of social data: The italian hate map experience . In: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization . pp. 307 { 308 . ACM ( 2016 )

24. De la Pen~a Sarracen, G.L. , Pons , R.G. , Mun~iz- Cuza , C.E. , Rosso , P. : Hate speech detection using attention-based lstm . In: EVALITA@ CLiC-it ( 2018 )

25. Pennington , J. , Socher , R. , Manning , C. : Glove: Global vectors for word representation . In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . pp. 1532 { 1543 ( 2014 )

26. Peters , M. , Neumann , M. , Iyyer , M. , Gardner , M. , Clark , C. , Lee , K. , Zettlemoyer , L. : Deep contextualized word representations pp. 2227 { 2237 (Jun 2018 ). https://doi.org/10.18653/v1/ N18 -1202, https://www.aclweb.org/ anthology/N18-1202

27. Poletto , F. , Stranisci , M. , Sanguinetti , M. , Patti , V. , Bosco , C. : Hate speech annotation: Analysis of an italian twitter corpus . In: Proceedings of the Fourth Italian Conference on Computational Linguistics (CLiC-it 2017 ), Rome, Italy, December 11-13 , 2017 . CEUR Workshop Proceedings , vol. 2006 . CEUR-WS.org ( 2017 ), http://ceur-ws. org/ Vol-2006/paper024.pdf

28. Polignano , M. , Basile , P. : Hansel: Italian hate speech detection through ensemble learning and deep neural networks . In: EVALITA@ CLiC-it ( 2018 )

29. Polignano , M. , Basile , P., de Gemmis, M. , Semeraro , G. , Basile , V.: AlBERTo: Italian BERT Language Understanding Model for NLP Challenging Tasks Based on Tweets . In: Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019 ). CEUR ( 2019 )

30. Radford , A. , Wu , J. , Child , R. , Luan , D. , Amodei , D. , Sutskever , I. : Language models are unsupervised multitask learners ( 2019 )

31. Sanguinetti , M. , Poletto , F. , Bosco , C. , Patti , V. , Stranisci , M.: An Italian Twitter Corpus of Hate Speech against Immigrants . In: Proceedings of the 11th Language Resources and Evaluation Conference 2018 ( 2018 )

32. Santucci , V. , Spina , S. , Milani , A. , Biondi , G. , Di Bari, G.: Detecting hate speech for italian language in social media . In: EVALITA 2018 , co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018 ). vol. 2263 ( 2018 )

33. Schmidt , A. , Wiegand , M.: A survey on hate speech detection using natural language processing . In: Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media . pp. 1 { 10 ( 2017 )

34. Sood , S. , Antin , J. , Churchill , E.: Profanity use in online communities . In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 1481 { 1490 . ACM ( 2012 )

35. Van Hee , C. , Lefever , E. , Verhoeven , B. , Mennes , J. , Desmet , B. , De Pauw , G. , Daelemans , W. , Hoste , V. : Detection and ne-grained classi cation of cyberbullying events . In: International Conference Recent Advances in Natural Language Processing (RANLP) . pp. 672 { 680 ( 2015 )

36. Warner , W. , Hirschberg , J.: Detecting hate speech on the world wide web . In: Proceedings of the Second Workshop on Language in Social Media . pp. 19 { 26 . Association for Computational Linguistics ( 2012 )

37. Xu , J.M. , Jun , K.S. , Zhu , X. , Bellmore , A. : Learning from bullying traces in social media . In: Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies . pp. 656 { 666 . Association for Computational Linguistics ( 2012 )