Social media platforms pave way for the public to express their opinions. These opinions are mostly on the events and happenings across the world. These comments are most often unbiased and cross the individual boundaries that cause hurt to the people involved. Some comments are intentionally delivered through these platforms with the purpose of ofending the party concerned. An automatic technique is needed to identify ofensive comments to prevent unwanted consequences. Our work is a part of Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages, where the ofensive public opinions on social media platforms are to identified. We have devised a system that uses both machine learning and deep learning techniques to detect the ofensive comments. Random Forest has obtained 78% macro F1 score, in Hindi Recurrent Neural Network performed well with 73% and Support vector machine with 75% macro F1 score.
With the development of technologies such as Artificial Intelligence, Machine learning, email, internet applications, came the social media and instant messaging applications. With the right to speech, people used these platforms to convey their thoughts and opinions. Most of the time these opinionated thoughts would cross their rightful boundaries and became ofensive. Comments made on the social media platform get viral and publicised to get more attention from the public which are often hurtful, and derogatory. The hateful messages are mostly on the premise of religion, one’s identity, gender, race, nationality, inclination towards one’s ideology and so on. Such comments may incite violence and imbalance to the peacefulness of the society in its worst case. For an individual, the insulting comments may lead to anxiety, depression and mental instabilities.
The hateful expressions can be handled in a few ways manually namely, calling out the inaccuracies in the comments or messages, tackling them politely, challenging them back and refuting them to show the truthful side or facts. Most of the time, handling social media harassment’s went out of hand and demands the need for automatic handling of such messages. On the other hand the issue can be taken up on a legal way and could be handled to counter the efects caused by the hurtful messages.
Necessity towards building a computational model to handle the ofensive social media comments has been increased in recent times. Social media organizations are working towards automatic handling of such ofensive contents in order to preserve the sanity of their platforms. A single generalized model to identify the ofensive comments irrespective of the languages they are based on, is the need of the day.
In this paper we investigated the performance of diferent machine learning and deep learning techniques in classifying the social media comments in English and Indo-Aryan languages into hate and ofensive. This work is a part of Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages (HASOC, 2021) under Forum for Information Retrieval Evaluation (FIRE, 2021). The section 2 provides the survey of existing work, section 3 discusses about the proposed system, experimental results and related discussion are given in section 4 and conclusion with direction for future enhancement in section 5.
Machine learning and deep learning algorithms [
MOLD, the Marathi Ofensive Language Dataset has been created to classify the ofensive
comments in Marathi language [
A corpus is constructed with linguistic taboos and euphemisms in Nepali language [
An USAD (Urdu Slang and Abusive words Detection) automatic lexicon-based system was
designed [
Indo-Aryan and Dravidian languages are investigated [
The proposed system of detecting ofensive content from the HASOC2021 Subtask 1 data is described in the following sections. The steps involved in the proposed system are as follows: 1. Data preprocessing 2. Feature extraction 3. Model training 4. Testing
Pre-processing is done to prepare the input data for further processing namely by removing the words that do not give meaning of the tweet, removing the special characters and getting the root word from the derived word, etc. The following preprocessing steps are carried out for the training data of all the three languages: 1. Stop word removal 2. Remove numbers 3. Remove special characters 4. Lemmatization through wordnet lemmatizer 5. Stemming through port stemmer
Term Frequency Inverse Document Frequency vectors are extracted from the text. The char TF-IDF is calculated as explained in equation 1.
For each char in a document from the document set , TF-IDF is calculated as: N is the total number of documents. where (, , ) = (, ). (, )
(, ) = (1 + (, )) (, ) = (
( : ) ) (1) (2) (3)
The extracted features (TF-IDF) are used for training the machine learning algorithms such as Random forest, Linear regression, Support vector machine and Recurrent neural network. For Recurrent neural network, an embedding layer is included to get the embeddings for the text that uses default embedding in Keras. The model parameters used for these algorithms are given in Section 4. From the given dataset, 80% of the data used for training and remaining 20% of the data used for model tuning. The machine learning model implementations and the metrics of comparison is used from Scikit-learn 1.
The four trained models using Random forest, Linear regression, SVM and LSTM are tested with the test data for evaluating their performance of classification. The metrics observed during the testing are discussed in the following section.
In this proposed work, Subtask 1A data of HASOC 2021 is used. The data sets are given in three
languages namely English, Hindi and Marathi. The datasets are sampled from Twitter. More
details about the dataset are given in [
Sub-task A focus on Hate speech and Ofensive language identification ofered for English, Hindi, and Marathi. The objective of this task is to classify the tweets into two classes, namely: Hate and Ofensive (HOF) and Non- Hate, and ofensive (NOT). In the proposed approach, the preprocessed text is trained using the machine learning algorithms such as Random forest, Linear regression, Support vector machine, and Recurrent neural network for all three languages. For a Random forest classifier, a number of jobs are assigned as 2 and the random state is initialized as 0. For the Support vector machine, the linear kernel is used. For logistic regression classifier, the random state is initialized as 0. For Recurrent neural network, the embedding layer is included to get the embeddings of preprocessed text. The number of LSTM units is 40, adam optimizer is used.
The cross-validation accuracy for English is given in Table 2. From Table 2, it has been noted that Random forest classifier produces the F1-score of 0.68 and accuracy of 80.2%.
The cross-validation accuracy for Hindi is given in Table 3. From Table 3, it has been noted that Random forest classifier produces the F1-score of 0.86 and accuracy of 79.2%.
The cross-validation accuracy for Marathi is given in Table 4.
From Table 4, it has been inferred that the performance of support vector machine is better than other models.
The performance of the proposed system using test data is shown in Table 5.
The need for automatic identification of ofensive comments across social media platforms motivated us to devise a system that uses both machine learning and deep learning techniques. This work has been submitted as a part of Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages (HASOC, 2021). One generalized approach cannot be designed to perform this classification in English and Indo-Aryan Languages. In the proposed automatic system for ofensive comment identification, Random Forest, SVM and RNN have outperformed with 78%, 75% and 73% of macro F1 score in Marathi, English and Hindli respectively. This can be further explored using multilingual word embeddings using transfer learning deep neural networks and by exploring the various parameters involved in those networks.