Emotion and targeted abuse detection i.e threat, are problems that have been studied in many rich resource languages. However, when it comes to low-resource languages such as Urdu, we find a dearth of resources and methodologies. Our paper presents the findings of the shared task "EmoThreat: Emotions and Threat detection in Urdu", where we focused on presenting resources for multi-label emotion classification (Task A) and binary threat detection (Task B) in Urdu. Task B was further divided into group and individual threat detection, making it a multi-class problem. The paper presents a summary of the methodologies and findings of the ten diferent participating teams. Each team also presented a thorough error analysis for the best model. The best performing system in Task A achieved a macroF1 score of 0.687, whereas, Task B subtask 1 and subtask 2 achieved 0.716 and 0.539 macro-F1 scores respectively.
Every language has diferent expressions, syntax, lexicon and vocabulary. Apart from that,
languages go through a continuous process of evolution. This stands true for Urdu more than
any other language. We know from the history of Urdu, that its foundation is manifested
in evolution. Urdu is an amalgamation of many languages i.e. Turkish, Sanskrit, Arabic and
Persian and continues to absorb words from languages that influence the demographics i.e.
English. Although Urdu is written in a diferent script (Nastalíq) than Hindi (Devanagari), they
have similar grammar and phonology. Urdu’s structural similarity makes it resourceful for
South Asian languages that share the same structure [
Although Urdu has more than 230 million speakers worldwide [
The paper attempts to spread awareness and encourage the community to propose more eficient methods for automated detection of multi-label emotion detection in short texts in Urdu. We highlight the collection and annotation of the first and largest datasets for detecting emotions and targeted threat in the Urdu language described in Section 2. Section 5 provides us with the baseline results for each task. Lastly, an overview and discussion of the submitted solutions for emotion and threat detection in Urdu are given in Sections 6 and 7.
We created a Nastalíq Urdu script dataset for multi-label emotion classification consisting of
tweets and reviews using Ekman’s six basic emotions [
• Anger: also includes annoyance and rage and can be categorized as a response to a deliberate attempt of anticipated danger, hurt or incitement. • Disgust: in the text is an innate response of dis-likeness, loathing or rejection to contagiousness. • Fear: also including anxiety, panic and horror is an emotion in a text which can be seen triggered through a potentially cumbersome situation or danger. • Sadness: also including pensiveness and grief is triggered through hardship, anguish, feeling of loss, and helplessness. • Surprise: also including distraction and amazement is an emotion which is prompted by an unexpected occurrence. • Happiness: also includes contentment, pride, gratitude and joy is an emotion which is seen as a response to well-being, a sense of achievement, satisfaction, and pleasure. • Neutral: void of emotional afect.
The dataset for Task A was taken from two separate sources. The Ekman’s emotions were
published and publicly presented in [
This task1 was aimed to detect threat speech using Twitter tweets in the Urdu language without human intervention. The task was divided into two subtasks. Subtask 1 is a binary-class classification task in which participating systems are required to classify tweets into two classes, namely: (i) Threatening, and (ii) Non-Threatening.
• Threatening - this Twitter post contains any threatening content.
• Non-Threatening - this Twitter post does not contain any threatening or profane content.
Once the tweet is classified as “Threatening”, then subtask 2 requires further classification of the threat into two classes: (i) Group, and (ii) Individual.
• Group - This Twitter post contains threatening content for targeting a group (s). • Individual - This Twitter post contains threatening or profane content for threatening an individual.
To collect the dataset for Task B, we created a dictionary of the most commonly used threatening words in Urdu. Then, we used those words as keywords on Twitter to extract tweets containing more threatening words in Urdu, which we manually added to our dictionary of threatening words. This dictionary is publicly available for research purposes.2 Then, we used these seed words to further crawl tweets through the Twitter Developer Application Programming Interface (API)3 using Tweepy library. We collected tweets containing any of these keywords from our dictionary for a 20-month period from January 1st, 2018 to August 31st, 2022. At this time the general elections were being held in Pakistan in July 2018. Typically, during the election season, people tend to be more expressive when supporting as well as opposing political parties. In total, we crawled 70,000 tweets containing the seed words.
Since Urdu shared many common words in Persian, Turkish and Arabic, so when we crawled tweets using our initially collected words, the Twitter API also crawled many non-Urdu tweets. 1https://sites.google.com/view/multi-label-emotionsfire-task/home/task-b 2https://github.com/MaazAmjad/Threatening_Dataset 3https://developer.twitter.com/en/docs/twitter-api/v1/tweets/search/api-reference/get-search-tweets Since this research was primarily focused on the Urdu language, we discarded all the non-Urdu tweets manually. The threatening dataset 4 contains 9,950 tweets, 1,782 threatening tweets and the remaining tweets are non-threatening. In this shared task, to make the dataset balanced, we only randomly chose 1,782 non-threatening tweets from the remaining non-threatening tweets collected in this dataset.
It is important to mention that we created a new test set, that contains 935 tweets in total: 628 non-threatening tweets, and 307 threatening tweets. The threatening tweets are further classified into 56 tweets that target individuals, and 251 tweets that target group(s).
We defined guidelines to annotate abusive and threatening tweets. To annotate the dataset annotators were recruited. All of them satisfied the following criteria: (i) country of originPakistan; (ii) native speakers of Urdu; (iii) were familiar with Twitter; (iv) aged 20–35 years; (v) detached from any political party or organization; (vi) had prior experience of annotating data; (vii) educational level was a masters degree or above. We computed Inter-Annotator Agreement (IAA) using Cohen’s Kappa coeficient ( 71%) as it is a statistical measure to check the reliability between two annotators. We provided instructions with task definitions. Hierarchical annotation schema was used and the main dataset was divided into two diferent datasets to distinguish between whether the language is threatening or non-threatening, abusive or non-abusive. We followed Twitter’s definition to describe abusive and threatening 5 comments towards an individual or group to harass, intimidate, or silence someone else’s voice.
Similarly, the task ofered a dataset of tweets in Urdu annotated as threatening or nonthreatening split into the training and testing parts, with the annotations of the testing part hidden from the participants. The annotation procedure for the sub-task 2 dataset followed Twitter’s definition of threatening tweets 6 as those that are against an individual or group meant to threaten with violent acts, to kill, inflict serious physical harm, to intimidate, or to use violent language. The task and the evaluation procedure were identical to sub-task 2.
Table 1 and 2 show the dataset distribution of both tasks. The dataset for both tasks is publicly available on the EmoThreat website 7.
Fine-grained emotions have been extensively studied in rich resource languages such as
English [
Among the notable works of sentiment classification in Urdu, we see [
Threatening language detection in Urdu was first introduced in [
Task A used multi-label accuracy, micro-averaged F1, weighted F1 and macro-averaged F1. The ground truth annotations were used to compare the labels predicted by the participants’ classifiers. All the participating teams were allowed to submit up to 3 diferent runs, i.e labels for the testing set generated by their proposed classifiers. The rankings were made on the basis of macro-F1 scores. We also gave the Hamming loss scores that computes the average of incorrect labels of an instance. Lower the value, the higher the performance of the classifier as this is a loss function. For Task B, we presented accuracy, macro-F1 score and ROC-AUC score. The ROC-AUC score gives an estimate of the overall quality of the model at the various level of predicted confidence thresholds and serves as a more holistic evaluator. The rankings were made on F1 scores.
To get our baseline results, we used four diferent types of features, including character ngrams, word n-grams, stylometric features and pre-trained word embeddings. Count-based features are character n-grams and word n-grams and we used uni-, bi-, and trigrams for word n-grams while trigrams to nine grams were selected for character n-grams. Moreover, we applied Term frequency-inverse document frequency (TF-IDF) technique to choose the best count-based features. Stylometric based features contains 47 character-based, 11 word-based and 6 vocabulary based features. For the pre-trained word embeddings, we used fastText 8 library to extract 300 dimensional vectors since it provides embeddings for the Urdu language.
We approached this problem as a supervised classification task and our aim was to predict
multiple emotions from the six primary emotions. To test the performance of our dataset, we
utilized various machine- and deep-learning algorithms such as RF, J48, DT, SMO, AdaBoostM1,
Bagging, 1D CNN, and LSTM. MEKA 9 software was used to calculate the baseline results for the
machine learning algorithms while Keras 10 library was used to implement deep learning models.
We used multi-label accuracy, micro-averaged F1, macro-averaged F1 and Hamming Loss (HL)
for our model evaluation. Uni-gram features yield the best results on RF with the combination
of a Binary Relevance (BR) transformation method, achieving 51.20% accuracy, 19.40% hamming
loss, 60.20% micro-F1 and 56.10% of macro-F1 scores. The 2nd best results were achieved
using 1-dimensional convolutional neural network (1D CNN) obtaining 45.00% accuracy, 36.00%
8https://fasttext.cc/docs/en/crawl-vectors.html
9http://meka.sourceforge.net
10https://keras.io/
hamming loss, 35.00% micro-F1 and 54.00% of macro-F1 scores. In the EmoThreat Task A, we
have added neutral emotional tweets to make Task A unique and challenging. These neutral
tweets are collected from [
For Task B, we used Logistic Regression, Support Vector Machine, Adaboost, Decision Tree, Random Forest, and Naive Bayes with word n-grams (1-3) as features. In Task B subtask 1 we achieved the best results (0.5117 macro-F1, 0.6760 ROC-AUC) with Random Forest, and in subtask B we got the best results (0.42966 macro-F1, 0.5 ROC-AUC) with Logistic Regression using word unigrams.
This section gives a brief summary of the methods applied in the competition by the participating teams. In total 35 teams registered for the competition. Teams were allowed to participate in both or just one of the given tasks. A total of 10 diferent teams submitted the experimental results with the distribution of 7 teams in task A and 7 teams in Task B. We report the findings of 9 teams who submitted their methodologies in the form of technical report papers. 1. FOSU NLP team: The best-performing model for Task A was presented by team FOSU which used transformers. They used XLM-RoBERTa along with adamW optimizer and “ReduceLROnPlateau”, a Keras module that reduces the learning rate when a metric has stopped improving. 2. UMUTeam: UMUteam presented a model for Task A that ranked second in the competition. The team used language-independent linguistic features, non-contextual sentence embeddings from fastText, and multilingual contextual embeddings from BERT and RoBERTa. They conducted multiple experiments leveraging Knowledge Integration (KI) and Ensemble Learning (EL). The best-performing method used a multi-input deep-learning model with Knowledge Integration (KI) of all the above-mentioned features. 3. ERTIM: Team ERTIM presented three distinct methods. The first method comprised transformer models (BERT-large, MuRIL, ALBERT) taken from HuggingFace 11. In Task A, the participants analyzed the misclassified tweets and created an external dataset with similar tweets to enhance the model. The second technique comprised of Unsupervised multiword expressions (MWE), where the team first extracted autonomous chunks of text using unsupervised text segmentation. Those tokens (character and word) were then used as features for logistic regression to create a seven versus one multi-output design. Lastly, a linguistic approach was used in Task B using lexical and syntactic cues fed to machine learning models (Logistic Regression, LinearSVC, Stochastic Gradient Descent). MWE approach gave the best results in Task A, while, transformers and linguistic approach proved to be the best in Task B subtask 1 and 2 respectively. 4. Hate-alert: Team Hate-alert participated in both tasks and worked with many transformer methods, however, the most successful methods proved to be mBERT (multilingual BERT) and MuRIL (Multilingual Representations for Indian Languages). For task A and task B subtask 2, mBERT produced the best results. On the other hand, MuRIL achieved the best results in Task B subtask 1. 5. MUCS: MUCS participated in both tasks and proposed a transfer learning model with mDistilBERT (Multilingual Distilled version BERT) and Classifier-chain model with SVM (linear Support Vector Classifier). The classifier-chain model is used for multi-label classification and used word n-grams (1-3) features to achieve the 4th rank in Task A. A ifne-tuned mDistilBERT with LSTM as the final layer was used to achieve the best results for Task B. 6. Aces: The participants in team Aces experimented with four models for Task A including Recurrent Neural Network (RNN), Classifier Chains, Long Short-Term Memory (LSTM) and Multi-label K-nearest neighbours. For the deep learning models, they used fastText and Word2Vec, while for the machine learning models they used simple TF-IDF features. The best performing model on the test set was Classifier Chains with TF-IDF features giving the macro-F1 of 0.381. This approach stood last in the competition for multi-label emotion classification task. 7. SakshiEmo2022: Team Sakshi participated in both tasks of the challenge. The team ranked first in Task B subtask one and third in subtask two by using multiple fine-tuned transformer models. To handle the imbalanced classes in the task, the team used oversampling and stratified sampling. The authors presented the results with MuRIL, BERTbase, Multilingual-BERT (mBERT), Distil-BERT and UrduHack. It was seen that MuRIL outperformed other transformers using the over-sampling method, whereas mBERT remained the next best method. In task A, team Sakshi experimented with transformers (UrduHack and mBERT) and machine learning models (Naive-Bayes, LinearSVC, Logistic Regression). The best results were achieved with LinearSVC using TF-IDF features and data decomposition (One vs Rest). This method ranked 7th in the oficial rankings. 8. Ttdmnx: The authors utilised pre-trained BERT models available on HuggingFace and achieved 4th and 5th positions in subtasks 1 and 2 of Task B. They simplified subtask 2 into a binary classification problem after separating the non-threatening tweets from the threatening tweets and applied the model to detect group vs individual threat tweets.
Table 3, 4 and 5 present the best results of the submitted systems in each task. The results of the participating team showed a significant increase in the F1 scores compared to the baseline values of Task A. The best performing team achieved 0.687 macro-F1 with four out of eight teams having more than 0.6 macro-F1 score. Weighted F1 scores were more competitive with the top 7 teams getting more than 0.6. The best-performing teams used transformer models which was a trend most of the teams carried with slight variations.
Task B comparatively had higher scores in the binary classification challenge, however, we saw that all teams score less than 0.6 F1 in the multi-class setup. In Task B, it was noticed that models failed to diferentiate “hate” from “threat” frequently. Examples of such cases are highlighted in the paper which was the third-best model presented in the shared task. The paper also highlighted confusion in task A between the labels “sadness” and “surprise” which was a frequent occurrence.
Similarly, team ERTIM documented ambiguities among the classes “anger” and “disgust” in their best model. they divided the errors in Task B into three diferent categories: lexical, phrasal and deictic. The authors noticed that the subjunctive mode for jussive phrases, second-person address, and future tense to express consequence was more employed in threat tweets. They observed the connection of abusive language with both classes of threat and were not able to identify MWE categories and phrase expressions of “non-threat” class. Team hate-alert made similar observations and reported that for threatening tweet detection, sometimes the presence of words such as “killing” made confusion.
Emotion and targeted abuse detection-based NLP tasks are overlooked frequently, especially in low-resource languages. This overview paper presents the findings of the shared task "EmoThreat: Emotions and Threat detection in Urdu". In this shared task, thirty-five diferent teams registered and ten teams submitted their proposed systems (runs). These teams used various techniques ranging from feature engineering to ML and DL algorithms. The approaches used include ensemble methods, deep learning methods and transformers. Mostly, teams used non-Urdu specialized transformers such as BERT and RoBERTa as well as Urdu-specialized transformers such as MuRIL, RoBERTa-urdu-small to achieve better results. FOSUNlpTeam outperformed all the proposed systems by using XLM-RoBERTa for Task A. Team SakshiEmo2022 and ERTIM produced the best results for Task B subtasks 1 and 2 respectively. These tasks aim to attract and encourage researchers working in diferent NLP domains to address multi-label emotion detection and threatening language detection problem in Urdu. It also helps to mitigate the abusive and threatening content on social media platforms.
The work was done with partial support from the Mexican Government through the grant A1S-47854 of CONACYT, Mexico, grants 20220852 and 20220859 of the Secretaría de Investigación y Posgrado of the Instituto Politécnico Nacional, Mexico. The authors thank the CONACYT for the computing resources brought to them through the Plataforma de Aprendizaje Profundo para Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE, Mexico and acknowledge the support of Microsoft through the Microsoft Latin America PhD Award.