Profane or abusive speech with the intention of humiliating and targeting individuals, a specific community or groups of people is called Hate Speech (HS). Identifying and blocking HS contents is only a temporary solution. Instead, developing systems that are able to detect and profile the content polluters who share HS will be a better option. In this paper, we, team MUCIC, present the proposed Voting Classifier (VC) submitted to Hate Speech Spreader Detection shared task organized by PAN 2021. The task includes profiling HS spreaders for two languages, namely, English and Spanish from the text collected from Twitter. This task can be modeled as a binary text classification problem to classify an author (Twitter user) based on his/her tweets as 'Hate speech spreader' or 'Not'. The proposed models utilizes a combination of traditional char and word n-grams with syntactic ngrams as features extracted from the training set. These features are fed to a VC that employs three Machine Learning (ML) classifiers namely, Support Vector Machine (SVM), Logistic Regression (LR), and Random Forest (RF) with hard and soft voting. The proposed models with accuracies of 73% and 83% for English and Spanish languages respectively, obtained second rank in the shared task.
Rapid dissemination, low cost, ease of access, and more importantly anonymity are the
significant features of social media in current era [
Appropriate tools and benchmarked labeled corpora are required to address the challenges of
HS detection and profiling the spreaders [
LR is an impressive algorithm for binary and linear classification problems which models
the probability of a discrete outcome from an input variable. Ease of realize and exquisite
performance are the major features of this algorithm for binary classification [
Traditional n-grams are a set of co-occurring items or elements such as characters, words, Part-Of-Speech (POS) tags, etc. as they appear in a text. But, the idea of Syntactic n-grams (sn-grams) is to follow a path in the syntactic tree to construct n-grams, rather than taking them from surface representation. In other words, the sequence of words that appear in the path of a syntactic tree are considered as neighbors and the real neighbors of words based on syntactic relations [14, 15, 16] are extracted. To obtain the benefits of both the n-grams structures, the traditional char and word n-grams are extracted and combined with sn-grams as a feature set and transformed into vectors using CountVectorizer to feed the VC model. Rest of the paper is organized as follows: the related work and methodology are discussed in Section 2 and 3 respectively followed by results in Section 4. The paper eventually concludes with future work in Section 5.
Most of HS detection tasks are modeled as short TC and rarely has been explored as profiling task. Some of the recent works on HS detection and text profiling have been reviewed here. Zimmerman et al. [17] has proposed an ensemble of Deep Learning (DL) models for HS detection and also Sentiments Analysis (SA) from tweets. The authors ensembled 10 Convolution Neural Network (CNN) models by summing softmax results from the underlying models and then averaging it. Considering the average soft-max score of all models, the class with highest average is assigned to the given tweet. Utilizing the publicly available embedding models, this model was evaluated on two datasets, namely, abusive speech [18] and SemEval 2013 SA [19] and obtained average F1-scores of 77.83 and 70.36 respectively, with batch size and epochs of 10 each. HASOC 2020 [20] shared task organized by Forum for Information Retrieval Evaluation (FIRE) 2020 consists of two subtasks; i) a binary TC task where a given text should be categorized as HOF (containing HS contents) or NOT (Not Ofensive) and ii) texts identified as HOF should be further classified into one of three categories namely, Hate speech (HATE), OFFENSIVE and PROFANITY. Datasets for this task has been provided for three languages, namely, English, Hindi, and German as detailed in [18]. Overall results reported by HASOC shows very competitive performances among the teams and diferences between the F1-score of best performances and average ones are less than 0.04.
As a participant of HASOC 2020, Balouchzahi et al. [
PAN at CLEF have managed to go further in identifying the content polluters who share HS, fake news, etc. or identifying bots from human followed by gender detection and profiling. Some of them are PAN 2018: Multimodal Gender Identification In Twitter [ 22], PAN 2019: Bots and Gender Profiling in Twitter [ 23] and PAN 2020: Profiling Fake News Spreaders on Twitter [24]. The task of profiling fake news spreaders on Twitter in PAN 2020 consists of datasets
2https://scikit-learn.org/stable/
for Spanish and English languages which includes 100 tweets per user and totally 300 users
per language as training set and 100 tweets per user and totally 100 users per language as
test set. Shashirekha et al. [
The significance of ensembling ML models lies in improving the strength and covering the weakness of individual classifier models. Taking a note of this concept, a VC model of three ML estimators namely, SVM, LR, and RF is developed by exploiting hard and soft voting configuration for English and Spanish languages respectively.
ML models used in the proposed VC are chosen because of their eficient performances for binary classification as proved in the available literature and based on our experiments. While RF which is already a method of ensembling utilize 10,000 decision trees as estimators, SVM uses linear kernel. Rest of parameters for these two models and all parameters of LR estimator have been set to default. As a preprocessing step, texts are striped and hashtags such as USER, URL, and RT are removed and all words are converted to lower case for English. However, preprocessing is avoided for Spanish language texts as our experiments without preprocessing performed better.
A feature extraction module as shown in Figure 1 is used to extract char (2, 3, 4, 5) and word (2, 3) n-grams and sn-grams (2, 3). SNgramExtractor3 library has been used to extract sn-grams from English and Spanish texts. The extracted features are transformed to vectors using CountVectorizer. Figure 2 illustrates the structure of proposed VC model graphically.
Datasets provided by PAN consists of a training set of 200 XML files for each language and each XML file represents a user with 200 tweets. The test set consists of 100 XML files per language and the proposed models should identify whether a user (represented by an XML file) is a ‘HS
spreader’ or ‘Not’ based on the analysis of tweets given in the XML files. Details of the training data along with label distribution which is presented in Figure 3 illustrates that the dataset is completely balanced.
PAN uses TIRA Integrated Research Architecture submission system [25] that provides Virtual Machine (VM) for the shared task participants through which they can submit and evaluate their proposed models. As PAN encourages early bird submission of the models, the initial models of the proposed approach are submitted through TIRA and due to technical issues the ifnal model and predictions on test set (labels) are submitted through mail. Performances of the models are evaluated by the task organizer based on accuracy metric and the results in shared task website4 illustrate that the VC model obtained accuracies of 83% and 73% for Spanish and English texts respectively.
Performances of the best teams presented in Table 1 shows very competitive results and our proposed models (mentioned as MUCIC) obtained second rank in the shared task. The highest
Following the adventures in text processing tasks, PAN 2021 called for a shared task to detect Hate Speech Spreaders in English and Spanish language tweets. This challenge is tackled by team MUCIC by building a robust VC using ML classifiers and traditional char and word n-grams along with syntactic n-grams as features to train VC model. Our team (MUCIC) obtained second rank with an average accuracy of 78% in the shared task. As future work we would like to explore more feature sets with ML models and also experimenting DL and TL approaches.
Team MUCIC deeply appreciates the eforts, guidance and support of the shared task organizers and reviewers for the valuable comments and suggestions. for heterogeneous datasets, SMU Data Science Review 1 (2018) 9. [14] G. Sidorov, Continuous and noncontinuous syntactic n-grams, in: Syntactic n-grams in
Computational Linguistics, Springer, 2019, pp. 63–67. [15] G. Sidorov, Syntactic dependency based n-grams in rule based automatic english as second language grammar correction, International Journal of Computational Linguistics and Applications 4 (2013) 169–188. [16] G. Sidorov, F. Velasquez, E. Stamatatos, A. Gelbukh, L. Chanona-Hernández, Syntactic n-grams as machine learning features for natural language processing, Expert Systems with Applications 41 (2014) 853–860. [17] S. Zimmerman, U. Kruschwitz, C. Fox, Improving hate speech detection with deep learning ensembles, in: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018. [18] Z. Waseem, D. Hovy, Hateful symbols or hateful people? predictive features for hate speech detection on twitter, in: Proceedings of the NAACL student research workshop, 2016, pp. 88–93. [19] P. Nakov, S. Rosenthal, Z. Kozareva, V. Stoyanov, A. Ritter, T. Wilson, SemEval-2013 task 2: Sentiment analysis in Twitter, in: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Association for Computational Linguistics, Atlanta, Georgia, USA, 2013, pp. 312–320. URL: https://www.aclweb.org/anthology/S13-2052. [20] T. Mandl, S. Modha, A. Kumar M, B. R. Chakravarthi, Overview of the hasoc track at fire 2020: Hate speech and ofensive language identification in tamil, malayalam, hindi, english and german, in: Forum for Information Retrieval Evaluation, 2020, pp. 29–32. [21] M. D. Anusha, H. L. Shashirekha, An ensemble model for hate speech and ofensive content identification in indo-european languages, in: P. Mehta, T. Mandl, P. Majumder, M. Mitra (Eds.), Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826 of CEUR Workshop Proceedings, CEUR-WS.org, 2020, pp. 253–259. URL: http://ceur-ws.org/Vol-2826/T2-20.pdf. [22] F. Rangel, P. Rosso, M. Montes-y Gómez, M. Potthast, B. Stein, Overview of the 6th author profiling task at pan 2018: multimodal gender identification in twitter, Working Notes Papers of the CLEF (2018) 1–38. [23] F. Rangel, P. Rosso, Overview of the 7th author profiling task at pan 2019: bots and gender profiling in twitter, in: Working Notes Papers of the CLEF 2019 Evaluation Labs Volume 2380 of CEUR Workshop, 2019. [24] F. Rangel, A. Giachanou, B. Ghanem, P. Rosso, Overview of the 8th author profiling task at pan 2020: Profiling fake news spreaders on twitter, in: CLEF, 2020. [25] M. Potthast, T. Gollub, M. Wiegmann, B. Stein, Tira integrated research architecture, in: Information Retrieval Evaluation in a Changing World, Springer, 2019, pp. 123–160.