Nowadays, social media sites like Twitter and Facebook emerge as user-friendly and accessible sources for people to express their voice. Everybody, irrespective of their age group, uses these sites to share every moment of their life, making these sites flooded with data. This has led to many positive outcomes. At the same time, it has brought risks and harms as these sites set no restrictions. The volume of hate speech is not manageable by humans. As part of the HASOC-2021 shared task on information retrieval, we, Team Ignite, address the problem of hate speech identification in the Hindi corpus. Subtask A aims to identify binary hate or non-hate speech. This work was further extended with subtask B to determine the result of subtask A into three categories: profane, offensive, and hate. Hence, this paper compares the performance of three feature engineering techniques and four machine learning algorithms to evaluate their performance on a publicly available dataset with two distinct classes. With these two classes of hate and non-hate, we create a baseline model and improve model performance scores using various optimization techniques. Moreover, the output of different comparisons can be used further for text classification techniques.
India is a very diverse country in terms of its culture and language variations. It has around 44% of the population speaking Hindi. In addition, the number of people using social media has increased extensively in the past decade. The exponential growth of social media has revolutionized communication and content publishing. Most youths are inclined towards it for news consumption and social interaction. However, the freedom to post any content on the internet has resulted in problems like offensive and inappropriate posts. Users quickly take advantage of this liberty to promote abuse and hatred through posts and comments.
Hate speech is a form of non-verbal or verbal communication expressing aggression or speech intended to provoke hatred or violence against a group. It is defined as an act of deprecating a person or community based on their gender, age, race, religion, nationality, etc. These hate speech posts on social media lead to a lot of violence. They face a lot of criticism for the increasingly offensive content on their sites.
Even though there has been significant research into the analysis of hate speech in numerous languages, the majority of the work has been done for English only. For many other languages, especially Indian languages, there is limited research on this recent and significant topic. Therefore, our study contributes to solving this problem by comparing three feature engineering and four machine learning classifiers on the given Hindi tweets dataset. We have created models which classify the content of a text as hateful or not. The vectors obtained from the different feature extraction techniques are fed to various classical machine learning models to predict the target class.
The different experiments with the various models yielded different results, which will be discussed. The highest Macro F1 score obtained for Task A was 69%, and that for Task B was 63%.
Classifying hate speech can become tedious considering the different ways a user can use a word. For example, the term "gay" can be used derogatorily and in the circumstances unrelated to hate speech.
To identify cyber hatred on Twitter, Peter Burnap et al. [
Few researchers have used machine learning to detect hate speech automatically in recent years. To construct the numeric feature vectors, they used unigram in conjunction with the TFIDF feature representation technique. The features were input into four machine learning classifiers: Naïve Bayes (NB), rule-based, J48, and Support Vector Machine (SVM). The rule-based classifier beat the NB, J48, and SVM classifiers in their experiments with 73% accuracy.
To classify hate speech communications, some researchers compared three feature engineering techniques and eight machine learning algorithms. The experimental results revealed that when bigram features were represented using TF-IDF, they performed better than word2Vec and Doc2Vec features engineering techniques.
Several teams submitted runs utilizing classical classifiers such as SVM, Logistic Regression, and
Random Forest employing word or character n-gram representations weighted by some TF-IDF score
in the first edition of the HASOC [
The format of the text in the training set shows the pattern as follows: _id, tweet_id, text, task_1, task_2
Where _id is a progressive number denoting the text within the dataset, tweet_id is the id of a particular tweet on Twitter, the text is the given post, and the task_1, task_2 columns contain the labels of type of speech. And the test set only includes _id, tweet_id, and the text columns. 3.1.
Subtask A offered for English, German, and Hindi datasets focus on hate speech and offensive language identification. It is a coarse-grained binary classification in which teams must categorize the given tweets into two classes, as follows:
Hate and Offensive (HOF) - This text contains hateful, offensive, or profane language.
Non-Hate and Offensive (NOT) - There is no hate speech or offensive content in this text. 3.2.
Subtask B involves fine-grained classification. Posts tagged as Hate and Offensive in Subtask A are further classified into three categories as follows:
Hate Speech (HATE) - Posts that make hateful remarks about someone based on their race, political beliefs, sexual orientation, gender, socioeconomic standing, health condition, or similar.
Example: अब ग ांडीव उठ ने क समय आ गय है प्रभुजी अधर्मिय ां क र्वन श क समय आगय है अभी नही त कभी नही Translation: Now the time has come to lift the Gandiva, Lord, the time has come for the destruction of the unrighteous, now or never. 2. Profanity (PRFN) - Posts that contain unacceptable language that isn't always insulting. This usually refers to the use of abusive language and cursing.
Example: मीर्डय द गल है देश र्वर धी है और अपने म र्लक के र्सि र् त लवें च टन जनत है । Translation: The media is crazy, it is anti-national, and people only lick the palate of their master. 3.
Offensive speech (OFFN) - This category includes posts that are humiliating, dehumanizing, insulting an individual, or threatening with violent acts.
Example: ग ांधी ख नद न क ग ांधी की पपड़ ती ने र्सद्ध ्रकय बस स त स ल जेल,यह ां क् ां बेल पर घूम रहे है कमीने? Translation: Gandhi's great-granddaughter proved the Gandhi family, just seven years in jail, why are bastards roaming here on bail? The posts that do not contain any Hate and Offensive (HOF) content are labeled as NONE. In the given dataset, most of the posts belong to the NONE category; some are HATE, whereas the other two classes are less frequently seen.
This section discusses the proposed system that we have implemented to classify the given Hindi tweets into two different classes for Subtask A and four classes for Subtask B. We have used various feature extraction techniques such as TF-IDF, Bag of words and Word2Vec in combination with four different classifier algorithms. Figure 1 shows the complete system overview of the implemented system implemented for the experiments.
The dataset used in this research study was provided by HASOC for the shared task of hate speech detection in Indo-Aryan Language tweets. This dataset comprised monolingual Hindi tweets, collected from Twitter, labeled into two categories, namely, 'Hate and Offensive (HOF)' and 'NotHate-Offensive (NOT).' The dataset consisted of a total of 4594 tweets, of which 3161, i.e., about 69% belonged to the class NOT, and 1433, which is about 31%, were of the class (HOF). Out of the tweets labeled as 'HOF', about 15% were of the class' Offensive (OFFN)', around 13% of class 'HATE', and only 5% were labeled 'Profane (PRFN)'. A pictorial representation of this is given in Figure 2. 4.2.
It has been proven in various studies that using text preprocessing makes natural language processing easier and provides better results. Therefore, we have implemented different preprocessing techniques step-by-step to filter out the unnecessary non-informative data. Since the dataset is obtained from Twitter, we have removed all the hashtags, URLs, and user mentions. Also, we have removed punctuations and stop-words, which are not very useful while analyzing a corpus. Since the dataset is entirely in Hindi, we have used regular expressions and pattern matching for most of the cleaning process. Besides cleaning, we have also performed stemming and tokenization of the individual tweets to get a clearer corpus. Stemming removes suffixes and converts individual words to their root forms. In contrast, tokenization breaks down the sentences into various tokens or words. Lastly, to ensure a smooth training process, null values have been removed from the dataset. 4.3.
Working with datasets containing hundreds (or even thousands) of characteristics is becoming increasingly prevalent. When the number of features in the dataset approaches (or exceeds!) the number of observations contained in the dataset, a Machine Learning model is likely to suffer from overfitting. It is vital to use either normalization or dimensionality reduction techniques to avoid this type of difficulty.
Feature Extraction is a technique for reducing the number of features in a dataset by generating new ones from existing ones (and then discarding the original features). The original set of features should then summarize the majority of the information in the new reduced set of features. From a combination of the original set, a summarized version of the original features can be generated. It can also help to reduce the amount of redundant data in a study. Furthermore, the reduction of data and the machine's efforts in constructing variable combinations (features) speed up the learning and generalization stages of the machine learning process.
For our experiments, we have vectorized the text column of the dataset using different techniques. Also, for Subtask B, we have given the output labels of Subtask A as an input feature to the classifier along with the featured text for better learning of the classification model. These two features, for Subtask B, have been concatenated and passed as a single input to the machine learning models. The three feature extracting approaches that we have used are discussed below.
Term Frequency-Inverse Document Frequency was the first feature extraction method we used. This strategy is based on frequency. When a corpus is considered, this strategy works by reducing the weight of words that frequently appear in all documents (these words are also known as stop words) and raising the weight of terms that frequently occur only in a subset of the documents. The weight of TF-IDF is determined by the two components, TF (Term Frequency) and IDF (Inverse Document Frequency). Term Frequency is a metric for determining how frequently a term appears in a document.
Take a look at a set of N documents. Define as the number of times the word or term i appears in the document j. The term frequency is thus defined as follows:
The term frequency of word I in document j is normalized by dividing it by the maximum number of occurrences of any term, excluding stop words, as shown in the above equation. The IDF for a given term is as follows: =
= ( ) (1) (2) Therefore, the TF-IDF score for term i in document j is calculated as: (3)
Bag of words is a text modeling technique. In technical terms, we can call it a method for extracting features from text data. This method of extracting features from documents is easy and adaptable. It is a text representation that describes the frequency of words appearing in a document. We only keep track of word counts and don't pay attention to grammatical subtleties or word arrangement because any information about the sequence or structure of words in the document is deleted. It is referred to as a "bag" of words. The model simply cares about whether or not recognized terms appear in the document, not where they appear.
One of the most significant issues with text is that it is messy and unstructured; machine learning algorithms prefer structured, well-defined fixed-length inputs. We can turn variable-length texts into fixed-length vectors using the Bag-of-Words technique.
4.3.3. Word2Vec
Word2Vec is a statistical method for learning a solitary word embedding from a text corpus quickly and efficiently. It was created by Tomas Mikolov et al. at Google in 2013 as a reaction to improving the efficiency of neural-network-based embedding training. It has since become the de facto standard for generating pre-trained word embedding. It's a hybrid of two models for grouping related words-: the Continuous Bag-of-Words and the Skip-gram models. A layer of multi-set of words, also known as a bag of words, is used as input to the CBOW model. In the concealed layer, these words are projected in a linear pattern. Because the hidden layer projects all of the words, the output of the hidden layer is the average of vectors.
For our experiments, we have custom-trained the Word2vec model for the given Hindi dataset. We have used the Skip-gram model with the hyperparameters such as vector length set to 200, context window set to 10, and the min_count to 2 to preserve the sentences' contextual meaning during the training process. This gave us a vocabulary of 7078 words which were then used to generate sentence embeddings based on the word embedding provided by the word2vec model and the total number of vocabulary words present in the tokenized sentence.
4.4.
For training and testing purposes for both tasks A and B, we have split the cleaned dataset into a 4:1 ratio, i.e. 80% for Train Data and 20% for Test Data. The classification model is trained to learn classification rules using the training data. In contrast, the test data is used to validate the learning of the classification model. Table I presents the class-wise distribution of the dataset before and after splitting it into train and test sets.
It is seen that there is no single classifier that performs best on different kinds of datasets. Hence, it is always suggested that one should apply several different classifier algorithms on different feature vectors obtained using various feature extraction techniques and observe which combination gives the best results. Therefore, we have selected four different classifiers, namely, Logistic Regression (LR), Naïve Bayes (NB), Support Vector Machine (SVM), and Random Forest (RF), to go with the three feature vectorization methods discussed previously.
Logistic Regression is a binary classification probabilistic model of machine learning. It predicts independent data variables by analyzing the relationship between one or more existing independent variables. It uses the sigmoid function with the Bernoulli distribution as follows: ( ) =
1 1+ −( 0+ 1 ) ( | ) = ( ) (( ) | )
It is a classification algorithm that uses the "Bayes theorem" to calculate probabilities to predict the class. It works on conditional independence among features. The conditional probability of the Bayes theorem is as follows:
It is a supervised learning algorithm used for classification and regression. It creates a hyperplane in multidimensional space by choosing the extreme points to separate different classes. These extreme points are called support vectors.
It is a type of ensemble supervised learning algorithm consisting of many decision trees. It is used for regression as well as classification. It constructs a decision tree and predicts the result, and performs a vote on each predicted result. The prediction with the most votes is the final prediction. ● ● ●
detect positive samples.
F1 Score weight to both precision and recall. −
A classifier's performance is calculated by evaluating four factors – True Positives (TP), False Positives (FP), True Negatives (TN), and, lastly, False Negatives (FN). A few commonly used metrics calculate the performance of a classifier based on these factors, namely Precision, Recall, Accuracy, and F1 score.
●
Precision Precision is the percentage of predicted positives that turn out to be true positives. It is also known as the positive predictive value.
= = Recall gives the percentage of all positive cases, which measures the classifier's ability to It's the total number of instances that have been correctly categorized. (6) (7) (8) (9) (10) (11) The F1 Score is the harmonic mean of precision and recall. The classical F1 score gives equal Since Task B is a multi-classification problem, we have used the Macro-averaged F1 Score for evaluation. For single-label multi-class situations, this is the default aggregation technique for the F1-score. In this report, both the subtasks, A and B, have the same evaluation metrics.
The results of the various analyses performed for this report are presented in Table 3 to Table 6. Most of the models performed well and gave similar results with very slight differences between them. For Task A, it is observed that the LR model performed almost similarly with both TFIDF and BOW features. Logistic Regression, by default, is a two-class classification method. Therefore it has been excluded for the Subtask B experiments, which consists of multiple classes. NB, however, shows a deficient performance with Word2Vec embeddings. On the other hand, the TFIDF vectorizer performs similarly with both SVM and RF classifiers for Task B. The high accuracy scores and low macro F1 scores result from the skewness in the given dataset, as seen in sections 4.1 and 4.4.
Naïve Bayes 0.61 0.63
Hate speech has become much more common on social media in recent years, which might significantly impact society in the coming years. Various studies have been done on automatic hate speech detection in social media text. However, most of the studies are limited to the English language. To monitor Indian social media feed, Hindi hate speech detection must be explored. Hence, to address this problem, we have conducted a study and explored the classification of Hindi text using some statistical models. We have used Feature Extraction techniques like TF-IDF, Word2Vec, and Bag-of-Words (BOW) and different classification models like Logistic Regression, Naïve Bayes, Support Vector Machine, and Random Forest for Subtask A, which was to classify tweets into NonHate-Offensive, Hate and Offensive tweets.
Then, using the labels obtained from Subtask A, we used a similar approach to do the further finegrained classification of Hate-speech and Offensive posts from Subtask A to three specific classes namely, Offensive, Profane, and Hate, as a part of Subtask B.
Our findings show a good performance in the scores of Subtask A. However, Subtask B requires some more study using more experiments based on specific classes using different machine learning approaches that can analyze text in more depth.