The detection of hate speech in online social media platforms is of great importance in text classification. There is a need to research languages other than English. In this paper, we describe our team Astralis' combined effort in the shared task HASOC. We analyzed various models such as Naive Bayes, SVM, ANN, CNN, and embeddings such as TF-IDF, Multilingual BERT, and OPENAI-GPT2. Our relative performance was better in Subtask B for all languages, with our best-performed system ranked in second position in German Subtask B.
● (HOF) Hate and Offensive - This division consists of Hate and offensive content.
Sub-task B is a fine-grained classification offered for English, German, Hindi. Hate-speech and offensive posts from the sub-task A are further classified into three categories. ● (HATE) Hate speech:- This contains a class of hate speech content. ● (OFFN) Offensive:- Posts under this class contain offensive content.
● (PRFN) Profane:- This subcategory contains profane content.
Hate speech detection is a vast field of research and attracts many. Here we briefly describe some of the works done in this area.
The GermEval presented a task that identifies offensive language. The performance was measured by
F1 score, precision, and recall.[
Work was done on text classification using CNN by Yoon kim[
Below we briefly describe the dataset used for HASOC 2020. Given below is the class-wise distribution of the dataset provided to us during this task.
HOF 1856 847
NOT 1852 2116
Here we describe the various methodologies we used in different steps of the experiment.
For preprocessing, we followed a few simple traditional steps. At first, all of the Twitter handles
were removed. After that, the links in the tweets were removed. All the retweets in the data had
“RT” at the start. So we removed that. Then we removed all the residual blanks and kept the emojis.
As the transformers[
Here we define the methodology that we used to analyze two different subtasks that were given to us. We have used various models to get the best possible results.
4.2.1. TF-IDF
It is short for the term frequency-inverse document frequency. One gram and bigrams were
used to create this vectorizer with minimum occurrences of 5 for English. The length of the vectors
derived was 1281
4.2.2. BERT[
BERT stands for Bidirectional Encoder Representation from Transformers. The hugging
face[
Similarly, shaped matrices as BERT were received from this transformer model for Hindi, English, and German.
We have used various Machine Learning algorithms and Deep Neural Networks, and here we
describe them in detail. We used tensorflow[18] keras[19] to make all the Neural Networks.
4.3.1. SVM[
SVM performs exceptionally well in specific NLP scenarios. To implement it, we used the scikit-learn[14] library. We used SVC with RBF kernel. To implement this with matrices that we got from transformers, we used the Continuous Bag Of Words method, in which the mean of embeddings of each word is taken. So the input to SVM was 768 length vectors.
4.3.2. KNN
This was also implemented using scikit-learn library with neighbours set to 3. Here again, the CBOW method was used.
4.3.3. Naive Bayes[15]
Naive Bayes also works well for NLP. Here, the CBOW method was used to get the embeddings.
4.3.4. ANN
The input to this ANN was the CBOW embeddings and tf-idf for English. It was a five-layer NN with 128, 128, 256, 512 neurons and the last layer had 1 or 4 neurons depending upon the subtask.
4.3.5. CNN[16]
Here we have used a similar approach as the work produced by Yoon Kim. The architecture of CNN has layers in the following order.
1. Input: For initializing the input tensors 2. BatchNormalization[20]: To normalize each batch that is being processed. 3. Dropout[17]: Dropout helps to reduce the possibility of overfitting 4. After this, it is divided into various branches. Each branch computes x-gram.
a. Conv1D: The kernel_size is set to x to compute x-gram. b. GlobalMaxPool1D: To get a vector of length of the number of filters c. BatchNormalization 5. The above three layers produce one x-gram. After this, all of the x-grams are merged into one Tensor. 6. The merged tensor is then passed into a Dense layer of 128 neurons. 7. And then, finally, a dense classifier layer with 1 or 4 neurons based on the subtask.
The architecture described above can be seen in Figure 1.
First, we experimented with CNN with Bigram, Trigram, and Four-gram. Then after with Bigram and Trigram. In most cases, the Bigram + Trigram model was working better than the Bigram + Trigram + Four-gram model. So, for further analysis, we considered only the Bigram+Trigram model.
For the above models, training Adam optimizer [21] was used. We also used mild kernel regularization[22] in English subtask 2. We have also used class weights[23] in subtask B for all languages because there was an imbalance of classes in the dataset.
Experimental results in the available test set show that CNN(bigram + trigram) outperforms all other models. CNN can exceed most of all baseline models, precisely because of the nature of tweets. For example, tweets can be indirect texts. (e.g., sarcasm), full of noise and may not follow proper grammatical structure.
CNN can identify many small and large patterns in a tweet; if some are impacted by the noise[
So from the above work, we concluded to submit the CNN with Bigram + Trigram model with bert-based-uncased transformer for English subtasks and bert-base-multilingual-cased transformer for Hindi and German subtasks. The label wise F1 scores for the submitted models are shown in Table 6. OFFN and HATE categories from subtask 2 have relatively lower F1 scores due to relatively lower occurrences in the dataset. The final results on the private dataset on which the final ranks were given are displayed in Table 7.
This paper describes offensive text identification into three Indo-European languages. We have shown our methodology for classifying tweets and posts from social media using multiple models in given three languages categorizing hate and offensive speech. After analyzing different models, we observed that the bert-based-uncased Bigram + Trigram model gives the best performance in English and bert-base-multilingual-cased transformer for Hindi and German subtasks. The results indicate that organizing profane and hate content is a strenuous task. In future work, we hope that better models and methods can be used to improve the effect of identifying hate speech.
2011. [15] [18] [16] [17]