With the massive growth and widespread usage of social media platforms, the rates of its misuse and its corresponding impact on society have seen an exponential rise in numbers. The comfort of anonymity and wide reach ofered by social media has led to the convenient spread of hatred and incitement to threats, that are often targeted against particular users and communities. Thus identifying hate speech, threats and intense emotions in the digital arena has gained attention recently. This is also the aim of the EmoThreat: Emotions and Threat Detection in Urdu 2022 Challenge. In this paper, we describe a few traditional machine learning models and deep neural networks submitted by our team Aces for Task A: Multi-label emotion classification in Urdu. The models tested include Classifier Chains, MLKNN, RNN and LSTM Networks implemented with a combination of feature extraction methods such as the Count Vectorizer and TF-IDF, as well as embedding models like Word2Vec and FastText. Each model has been discussed in detail in Section 4 after a brief overview of the dataset adopted, in Section 3. Out of these tested permutations, the Classifier Chains model with TF-IDF vectorization proved to give the most promising results, which has been detailed in Section 5.
The right to free speech and expression on global platforms has inadvertently led to the
generation of numerous digital posts containing hateful, sensitive and abusive content. These
vulgar narratives, often-times targeted against certain individuals and communities worsen
users’ experience from communication via such media, while other posts contain actual threats
that put users in danger. Today major social-media platforms such as Google, Meta, YouTube
and Twitter are taking significant eforts to resolve this issue by detecting and censoring
objectionable content before it can instigate disruptions and chaos in society. With the Urdu
language having more than 230 million speakers worldwide[
With social media as a major platform for blogging and sharing information impacting millions of users’ lives every day, detecting and identifying intense emotions and objectionable content becomes a very important task. A lot of research has been conducted over the years on various aspects of the said issue keeping in mind the significance it holds on today’s digital society. Over the years, individuals and organizations have come together and proposed various diferent models.
For instance, in [
On a much recent scale, the researchers of [
The dataset used is adapted from Task A of EmoThreat: Emotions and Threat Detection in Urdu, FIRE 2022. The Urdu text in question has been generated using the Nastalīq Urdu script dataset and consists of tweets made by various users on Twitter. The labels adopted for the text classification are based on Ekman’s six basic emotions and neutrality. These six basic emotions apart from Neutral are: Anger, Disgust, Fear, Sadness, Surprise and Happiness. The data set consists of 8 columns, representing the 7 labels of emotion followed by the Urdu tweet in the last column. There is a total of 7,800 rows, each of which corresponds to an independent tweet and its corresponding classification label - marked as ’1’ in the appropriate column if the emotion is detected, or ’0’ otherwise. Table 1 shows the distribution of the data set.
Classifier chains is a machine learning model used for problem transformation in multi-label
classification scenarios. It uses |L| (number of labels) binary classifications where each
classification is linked along a chain and deals with the binary relevance problem associated with its
corresponding label L_j ∈ L[
RNN stands for Recurrent Neural Networks, which is a class of neural networks with loops in them, allowing information to persist for a period of time. This neural architecture thus allows inputs to be taken from previous outputs, enabling patterns to be dependent on previously extracted patterns instead of treating each data point as an individual entity. The RNN model employed we have built here uses a sigmoid activation layer along with a binary cross-entropy loss function and Adam optimizer.
Long Short-Term Memory (LSTM) is a special recurrent neural network (RNN) architecture that was built to hold an edge over standard feed-forward neural networks and RNN because of their trait of selectively remembering patterns for a longer duration of time. LSTMs use feedback connections which distinguishes them from traditional feed forward neural networks. This property enables LSTM models to process sequences of data without treating each sequence point as an independent entity, but rather, retaining useful metadata about the previous sequence points to help with the computation and processing of new data points. We used LSTM-based neural network classifiers built with the Keras toolkit, along with fastText Urdu word embedding to classify our Urdu text. We used word tokens, Embedding layer (300 dimensions), input length (10001 words) as inputs to the LSTM (128 units) layer and sigmoid activation function for the output layer. In this pipeline, we used categorical-cross entropy as the loss function and the Adam optimizer for parameter optimization.
CountVectorizer is a tool provided by the scikit-learn library in Python, used to convert text into a vector based on the frequency (count) of each word that occurs in the entire text. This is used as a feature extraction method for text classification problems. Making use of frequencies, it converts a group of text documents to a matrix of token counts. Here, the MLKNN model uses CountVectorizer and takes max_features as 1000 with max_df value of 0.85. 4.2.2. TF-IDF TF-IDF, short for Term Frequency-Inverse Document Frequency is also a scikit-learn library feature provided by Python for feature extraction in text processing. It converts a collection of raw documents to a matrix of TF-IDF features. Term frequency(TF) refers to the frequency of a word that occurs in a document while Inverse Document Frequency(IDF) implies the number of times the document contains the word in the corpus. TF-IDF is nothing, but the multiplication of TF and IDF. It works similarly to CountVectorizer but gives more importance to the words and their relevance. This enables us to remove the words that are less important for our analysis, thereby simplifying the model building by reducing the input dimensions. 4.2.3. Word2Vec Word2Vec is a technique to eficiently create word embedding which makes use of shallow neural networks. The efectiveness of Word2Vec is due to its ability to gather and group together vectors of similar words. The embedding matrix was generated using the Word2Vec model, having a model vocabulary’s size of 85868. The embedding layer was done using Word2Vec embedding matrix and used a vocabulary size of 5000.
FastText is an open-source, free library used for eficient learning of word embedding and text classifications. It is a very fast NLP library created by Facebook’s AI Research (FAIR) lab. It allows the training of both supervised and unsupervised representations of sentences. The implemented model utilizes Urdu word embeddings which are represented as vectors having a dimension of 300.
The performance parameters of the various models tested are presented in the tables below.
Table 2 presents the model performance results shown while training and Table 3 displays the
results shown by test data. Considering the training data set, a performance accuracy of around
60% was achieved using the Classifier chains model and the simple RNN model. Overall, in the
test data set, the Classifier Chains model gave the best results, with a performance accuracy of
around 43%. This can possibly be attributed to its property of combining the computational
eficiency of the widely-known Binary Relevance method for multi-label classification, while
still being able to take the label dependencies into account for classification. It also retains the
advantages of the binary method including low memory and run-time complexity. Moreover, in
the recent decade, studies exploring the underlying theory and working behind classifier chains
have made many improvements to the training procedures, such that this method remains
among the state-of-the-art options for multi-label learning of text data. [
The least performing model here is MLKNN, though there has been a visible improvement in metrics when diferent feature extraction algorithms and parameter tuning were applied. The reasons behind its underperformance appear to be inconclusive. Yet we believe that the pre-processing of text and the size of training data used always play a much bigger role than expected. MLKNN is moreover known to be computationally expensive because the algorithm stores all of the training data and is sensitive to irrelevant features. This could potentially lead to errors during classification, which hypothetically explains its high performance with the training data and poor performance with the test data.
In this paper, we have presented solutions to Task A of EmoThreat: Emotions and Threat Detection in Urdu, at FIRE 2022. In this work, 4 diferent ML models along with 4 feature extraction methods were adopted and tested for identifying the seven categories of emotions (six basic emotions along with, neutral), and thereby classify various kinds of texts or tweets. Prediction accuracy, F1 macro and micro scores were used as evaluation metrics with the F1 score as the key parameter. Although the results presented are adequately good, we believe that they could potentially be improved with better training and cross-validation methods. It can also be enhanced further by exploring the linguistic features of the Urdu language for better feature extraction. The possibility of other ML models outperforming the proposed systems remains open, and a higher performance accuracy can be fore-looked by employing fine-tuned parameters. For future works, transformer-based models and other deep learning approaches can be applied with fine-tuned parameters to improve emotion detection and classification. Thanks to the Department of CSE, SSN College of Engineering, Kalavakkam, India, for providing us with the opportunity, awareness, and guidance for this task. Special thanks to the organizers of FIRE 2022 for providing us with the necessary data set and for bringing together individuals nationwide to work on this project.