Hate Speech is described as any form of speech in which speakers attempt to ridicule, humiliate, or inculcate hatred in someone else's minds based on characteristics such as religion, the colour of skin, race, or sexual preference. In recent years, social networking sites have been a major source of excessive amounts of hate speech. If unaddressed, these might cause anxiety and despair in the afected individuals or groups. As a result, the above-mentioned social networks utilize an assortment of algorithms to identify such hate speech. Detecting Hate Speech in English texts has been one of the hottest topics in recent years, with multiple types of research being published. However, in regional and indigenous languages, hate speech detection is a recent area with not much research being conducted. It is dificult to perform hate speech detection using data in regional languages due to a lack of large enough training data and a lack of resources about that domain. The HASOC [1] 2021 Hate Speech Detection Task solves one of the problems. It provides a dataset containing Tweet data in English, Hindi [2] and Marathi [3] languages. There were two subtasks as part of the main task. The subtask was to classify the hate speech and ofensive texts in the Hindi and Marathi tweet dataset as Hate Speech (HATE), Ofensive (OFFN) or Profane (PRF). This work compares the performance of diferent models on both subtasks and provides a conclusion on the best performing model. The Random Forest Classifier reports the most remarkable accuracy on the first subtask with a macro F1 score of 75.19% and 73.12% on the Marathi and Hindi tweet datasets. The XGBoost algorithm is the best performing algorithm on the second subtask with a 46.5% macro F1 score. Overall any of these models can get satisfactory results when dealing with hate speech detection in regional language. This work has been submitted to the FIRE2021 shared task, Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages (HASOC-2021) by team DLRG.
Hate speech is defined as any communication, whether spoken, written, or physical, that criticises or uses discriminating, or disparaging terminology about a person ( or group) based on who they are, such as colour, religion, nationality, ethnicity, race, sexuality, descent, or another identifying feature. Nowadays, social media has evolved into a popular forum for ordinary people to share their feelings on any topic. It has its own set of advantages and disadvantages. Everyone understands their point of view and perceives the world in a new light when they express themselves. However, this might also imply that people have the right to share whatever they wish, even ofensive content that is harsh and upsetting to people. It impacts not only the feelings of that particular group but also their self-esteem and dignity. As a result, maintaining a courteous demeanour while portraying one’s viewpoint on any social media site is essential. Unfortunately, preventing people from posting or tweeting hate speech on social media is quite tricky. Thence, every social media company globally is attempting to build a revolutionary approach for detecting and preventing hate speech. It has been done to some level in English, as English is a widely known and spoken language worldwide. However, this is not the case in Indian regional languages such as Hindi, Marathi, Tamil, and others. Therefore, it is critical to have a solution in place.
Machine learning is an application of data analysis that facilitates the creation of analytical models. Artificial intelligence is predicated on the idea that computers can recognise patterns, learn from data, and make decisions without the need for humans. Machine learning has been utilised to solve problems in many science and technology domains, including social media. Many academics are now working on multiple machine learning models to detect highly accurate hate speech and an optimised model. However, there has not been much advancement in the subject of detecting hate speech in regional languages. Because it is challenging to train a model in a language that isn’t widely spoken and hence lacks the necessary dataset and procedures to train the model efectively. In this study, data embedding and multiple classifier models were employed to determine whether or not a tweet contains hate and ofensive content in two Indian regional languages, namely Hindi and Marathi. This research is split into two sections: subtask A and subtask B. Binary classification models were used to determine if the tweet contained hate speech in Hindi and Marathi. The multiclass classification was used for subtask B to classify hate speech further, and they are: hate speech, profane or ofensive content in Hindi. Our team DLRG participated in Subtask A-Marathi, Subtask-A and B for Hindi. In section 2 of this paper, the related works have been analysed and discussed. In section 3, the dataset utilised in both subtasks is discussed. Section 4 includes an explanation of the proposed methodology. In section 5, the findings are presented and discussed, and in section 6, the conclusion and future work are presented.
In earlier works, diferent term weighting methods have been applied for web page categorization
using TF-IDF based methods [
The dataset was obtained as part of the HASOC ’21 competition at FIRE 2021. These datasets consist of tweets sampled from the Twitter social media platform. The data set consisted of English, Hindi and Marathi tweets. For this work, the English tweets were disregarded. The hashtags, URLs and keywords prevalent in regular tweets are also present in this data. For Subtask A, the dataset was broadly divided into two categories, NOT tweets that contained non-ofensive content and HOF tweets that contained ofensive or profane content. The HOF tweets in the Hindi tweet data were further divided into Hate Speech (HATE), Ofensive (OFFN) and Profane (PRF) as part of Subtask B. The dataset was in a CSV file with each entry containing the ID, Tweet ID, Text and corresponding labels. The Hindi tweet data consisted of 4594 entries, while the Marathi data consisted of 1874 entries.
The proposed methodology for all the subtasks are detailed in the following sections.
In the preprocessing of the data, the first step performed was removing noise. The URL links and Twitter handle names were the primary sources of noise in the dataset. All the words are removed that followed @ symbol using regular expressions. After the above step, Twitter handle names are removed and to remove URL links, regular expressions were used to remove anything that followed HTTP/HTTPS. Then the stop words like a, an, the, is, etc., followed by punctuation marks and special characters, are removed. These stop words are not necessary to ifnd the statement’s sentiment as they do not carry any significant meaning. The punctuation and special characters cannot be used to identify the sentiment.
We removed these as they are unnecessary, and we can work solely on the essential characters and words for better performance. Further, Stemming is performed to reduce the inflection in words, and the words can be brought to their respective base forms. Stemming helps deal with spelt words and remove unwanted sufixes, which is very common in a social media text. We used a snowball stemmer imported from the NLTK package, and it is based on a programming language called "Snowball". Finally, we removed high-frequency words. Based on the term frequency method, we removed the most frequently occurred 1000 words.
The Scikit-Learn [
The process by which a group of data is divided into a set of categories is known as classification. This process can be performed on both unstructured and structured data. The most crucial step in this process is predicting the category by performing some calculations on the data points provided. This proposed model uses the following classifiers: Logistic Regression, Support Vector Machine, Stochastic Gradient Descent, Random Forest, Ensemble, and XGBoost.
Classifying the labels of a collection into two groups using a classification model is known as binary classification. In most binary classification problems, one class represents the normal state, and the other represents the aberrant state. The regular state class is assigned the class label 0, while the aberrant state class is labelled 1. Logistic Regression, k-Nearest Neighbours, Decision Trees, Support Vector Machines, and Naive Bayes are the most prominent binary classification techniques. In this research, subtask A consists of binary classification with the labels Hate and ofensive (HOF) speech or Not hate speech (NONE).
Classifying the labels of a collection into three or more groups using a classification algorithm is known as multiclass classification. In most multiclass classification problems, the diferent classes represent the various labels that are to be assigned to each set of input variables. Generally, when label encoding is done for multiclass problems, each class would be converted into a binary number. This method of having multiple output variables instead of multiple outputs simplifies the process of training. Logistic Regression, Support Vector Machines, K Nearest Neighbours, Naive Bayes and Decision Trees are some of the most prominent classification techniques that can also be extended to multiclass problems. In this research, subtask B consists of multiclass classification with the labels Hate Speech (HATE), Ofensive (OFFN) and Profane (PRF).
Gradient descent is an iterative procedure that begins at a random position on the slope of a function and gradually falls until it reaches its lowest point. Stochastic Gradient Descent is an optimisation approach that takes one data point at random from the entire data set at each iteration to minimise computations drastically. SGD is an easy algorithm to implement as well as an eficient method to find the minimum error point. Because of the randomness of SGD’s descent, it usually takes more iterations to reach the minima. Even though it takes more rounds to reach the minima than normal Gradient Descent, it is computationally considerably less expensive. Stochastic Gradient Descent has been used in both subtasks A and B. This model was chosen due to the fact that, it takes very little amount of time to train the model. That and the reason that this model is more likely to reach the local minima of the error function than other models, makes it an interesting prospect. Python’s Scikit-learn package loads the SGD model. After that, the model is trained on the training data, and the results are verified and documented.
This is a popular machine learning algorithm that has seen its fair share of usage as a classifier. The base for Logistic Regression can be traced to its conception from statistical mathematics. The model is trained on data where the dependent variables are categorical. It is derived from the Linear Regression model. The output obtained from the Linear Regression Model is fitted into a Logistic Function, which predicts the target variable. This model makes use of a decision boundary. It sets a threshold that diferentiates one class of variables from another. The linearity or the non-linearity of the decision boundaries depend on the input variables. The activation function used is a Sigmoid activation function. This constraint the output obtained between the numbers 0 and 1. Hence, the output becomes an estimated probability, which gives the final class prediction when subjected to the decision boundary.
This model was chosen due to the fact that it is one of the most simple and basic classifiers
that can be implemented and it would provide a good base for comparison against the other
approaches. The Scikit-Learn [
A support vector machine (SVM) is a type of machine learning technique that evaluates data for categorization and regression analysis. A supervised learning algorithm called a support vector machine sorts data into two groups. An SVM algorithm’s job is to figure out which category a new data point fits in. As a result, SVM is classified as a non-binary linear classifier. An SVM’s output is a map of the sorted data with a separation between both sides. The SVM method represents each piece of data as a position in n-dimensional space, where each feature is the value of the specified coordinate. Later, the classification process is then completed by selecting the hyper-plane that best diferentiates the 2 classes. In the SVM classifier, creating a linear hyper-plane between these two classes is easy. The SVM kernel is a function that transforms a low-dimensional input space into a higher-dimensional space to convert a non-separable problem into a separable one. SVM is eficient in high-dimensional spaces and works well with a clear separation margin. Here in this research, SVM is used in both subtasks. This model was chosen for the reason that it works extremely well in cases where there are a large number of features when compared to the training set. This comes in handy when dealing with data vectorized with TF-IDF where a large number of features are extracted. It also is highly memory eficient. Python’s Scikit-learn package is used to load the SVM model. After that, the model is trained on the training data, and the results are verified and documented.
When individual classifiers do not perform well on a given dataset, it is common practice to combine the models in a process called Ensembling. In most cases, this combined model would perform as well as or even better than the individual classifiers. For t One such Ensembling technique explored for this task was Majority Voting. It is an ensemble technique where the predictions from various weak classifiers for a given set of input variables are considered votes. These votes are tallied together, and the final outputs are obtained from the tallied votes. In the current Binary Classification problem, the same process occurs. The predictions of all the weak learners are considered as votes. The number of votes obtained by every output class are tallied. The output class that gather the highest number of votes will be considered the output for that particular set of input variables.
The combining classifiers used for this task are Support Vector Machine, Logistic Regression, Stochastic Gradient Descent, K Nearest neighbours, Naive Bayes, Random Forest and Decision Tree. Instances of each of the above models are initially trained on the Vectorized input data and their corresponding labels. Then the validation data is passed to all seven models, and the predictions are obtained. Majority voting is performed on these predicted labels, and the final predicted labels from Ensembling are obtained. These predicted labels are compared with the actual labels to obtain the performance metrics of the model. A similar process is also carried out for the test dataset.
XGBoost is a popular Machine Learning algorithm which works as decision tree-based ensemble using a gradient boosting framework. The XGBoost algorithm takes a text string as input and loads an XGBoost model trained to predict its label. Bagging and boosting are the most commonly used ensemble learning techniques. The XGBoost classifier is robust and even in distributed environments XGBoost yields eficient results.In the Scikit-learn framework,the algorithm provides a wrapper class by allowing models to be treated like classifiers or regressors . With this we can use the entire sci-kit-learn library with XGBoost models. The ensemble tree methods such as XGBoost and Gradient Boosting Machines use gradient descent architecture to boost weak learners principles.This model was chosen as a fact that it works well with large number of texts in training dataset. As an advantage, At each iteration the XGBoost algorithm has a built in method,which is cross validation and it is made easy by not specifying the exact count of boosting iterations required during a single run.
Random Forest is an advanced machine learning technique that may be applied to various tasks, including regression and classification. It is an ensemble method, which means a random forest resulting in a combination of many small decision trees called estimators, each of which makes its predictions. To provide a more accurate prediction, the random forest model integrates the estimators’ predictions. The problem of standard decision tree classifiers is that they are prone to overfitting the training set. The ensemble design of the random forest compensates for this and allows the random forest to generalise efectively to anonymous data, including data with missing values. Random forests can also handle enormous datasets with much dimensionality and several feature types. While expanding the trees, the random forest adds more randomness to the model. When splitting a node, it looks for the best feature from a random subset of features rather than the essential feature. As a result, there is much variety, leading to a better model. Random forest is used in both subtasks in this study. The Random forest model is loaded using Python’s Scikit-learn package. The model is then trained using the training data, with the results being validated and reported.
A comparison was performed on various models that were trained on the Marathi dataset. The diferent classifiers used after cleaning and vectorising the Marathi tweet dataset are, Logistic Regression, XGBoost, Support Vector Machine, Stochastic Gradient Descent, Majority Voting and Random Forest. A comparison of the performance metrics of all the diferent classifiers used can be viewed in Table 2. As the table explains, the highest performance was obtained with the Random Forest classifier model. The model attained an F1 score of 0.7519. Out of all the classification models implemented, the Logistic regression model achieved minor performance with an F1 score of 0.6924. It can be explained because the Logistic regression algorithm does not work well with non-linear data. It also does not bode well with outliers in the training data. Since the data considered here is mostly text, it is valid to assume that it contains multiple cases of outliers being present in it.
The Logistic regression algorithm also finds it challenging to accommodate the distributed data, which might explain its diminished performance. As expected, the Majority Voting technique performs better than its weak constituent learners due to reasons explained in section 4.7. However, it can be observed that the usage of Random Forests improves upon this technique. It can be explained by the fact that Random Forest is by itself an Ensembling algorithm. Tree type algorithms are generally better than probabilistic models in handling outliers and noise in the input data. They are also less impacted by the noise. A unique feature of the Tree type models and incredibly Random Forests is the innate ability to handle missing data independently. The embedded data is also non-linear. This negatively afects all probabilistic models, as that is the area in which they thrive. Random Forests are not impacted by non-linear data as much as the probabilistic models. The comparison between all the diferent models used and the F1 score procured by them is illustrated in Figure 1.
A comparison of several models trained on the Hindi dataset was carried out. After cleaning and vectorising the Hindi tweet dataset, the following classifiers were used: XGBoost, Logistic Regression, Stochastic Gradient Descent, Support Vector Machine, Majority Voting, and Random Forest. Table 2 contains a comparison of the performance metrics of all the various classifiers utilised. The Random Forest classifier model produced the best results, as seen in the table. The model achieved an F1 score of 0.7312, and the XGBoost model performed the most unsatisfactory out of all the classification models tested, with an F1 score of 0.6628. This low performance could be explained because the boosting algorithm does not work well with distributed data. It’s also bad news for anomalies in the training data. Given that the data in question is largely text, it’s reasonable to presume that it contains several instances of outliers. As predicted, the Majority Voting approach outperforms its weak learners, for reasons stated in section 4.7. However, it can be shown that using Random Forests improves on this method. It may be explained by the fact that Random Forest is an Ensembling algorithm in and of itself. In general, tree-type algorithms outperform probabilistic models in dealing with outliers and noise in input data. They are also less influenced by noise. The capacity to manage missing data on its own is a specific property of Tree type models, particularly Random Forests. The embedded data is also non-linear and harms all probabilistic models because it is the domain in which they thrive. Random Forests are less afected by non-linear data than probabilistic models. Figure 2 depicts a comparison of all the diferent models employed and the F1 score obtained while using those classifiers.
Several machine learning models were trained on the Hindi dataset and then compared. The input data was obtained after cleaning the tweets and vectorising them. The various machine learning algorithms used are XGBoost, Logistic Regression, Stochastic Gradient Descent, Support Vector Machine, Majority Voting and Random Forest. A comparison of performance metrics for all the above models is shown in Table 2. For the given subtask, XGBoost performed well with the highest F1 score of 0.4658, followed by the Ensemble model with an F1 score of 0.4425. The model with the lowest performance observed was the Random forest, with a macro F1 score of 0.405. This can be attributed to the fact that as the number of features increase, Random forest algorithm is more prone to overfitting. Following Random forest, the next least performance was showed by Support Vector machine with a macro F1 score of 0.4225. This performance achieved by Support Vector Machine can be explained because it does not perform well for large datasets or datasets with more noise. This is especially true with text data as it is prone to the presence of noise.
For multiclass classification, XGBoost performs well. At each iteration of the boosting process,the XGBoost algorithm runs cross validation, and in a single run the exact optimum number of boosting iterations are obtained. It leads to good results being obtained. The Majority Voting performed well after XGBoost due to the reasons explained in section 4.7. Therefore for the Hindi subtask B dataset, XGBoost produced the best performance and Random forest exhibited the least performance. Figure 3 illustrates the macro F1 scores obtained by diferent models in subtask B. Table 1 shows the various submissions made to the FIRE2021 shared task, Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages (HASOC2021), along with their subtask names, classifiers and macro F1-scores. We have extended the experiments by applying other classifiers and tried to improve the performance, the final results are tabulated in Table 2.
This work was submitted to the FIRE2021 shared task, Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages (HASOC-2021). The challenge of recognising conversational hate-speech in Indian regional languages has been empirically studied in this research utilising binary and multiclass classifiers for Marathi and Hindi tweets. This research has projected a complete method from analysing the tweets to comprehending the results. In subtask A, Random forest showed the best Macro F1-score of 75.19% and 73.12% respectively for both Marathi and Hindi tweets. With a Macro F1-score of 46.58% in subtask B for Hindi tweets, it is evident that XGBoost produced the most significant results. Overall, it can be inferred that Random Forest works well for binary classification, whereas XGBoost is a better classifier model for multiclass classification. In this work, only one type of feature extraction method has been implemented. In the future, multiple other types of embeddings can be implemented to get a better model by combining it with deep learning algorithms.