Our participation in the EmoThreat 2022 challenge relied on 3 distinct methods. We used algorithms based on BERT, a traditional statistic method and unsupervised Multi Word Expression (MWE) extraction to resolve the two proposed tasks to classify Urdu tweets. Task A consisted in classifying tweets into 6 classes of emotions. Task B was subdivided into 2 sub-tasks: B1) detecting threat and B2) specifying if the threat targets a single person or a group. For task A, the unsupervised MWE extraction method obtained the best results. For task B, the BERT based approach was more eficient to identify threats and the ML traditional method turned out to be the best among challenge participants to detect individual or collective threat. After discussing the types of errors, we suggest some guidelines for further improvement of those models.
Several approaches have been experimented on very similar tasks during the last years. In
Zameen and Sayeed [
Organizers of EmoThreat 2022 provided train and test datasets for each task. For task A, the
train set consisted of 7,801 tweets and the test set of 1,951 tweets. For task B, there were 3,564
tweets in the train set and 635 in the test one. As commonly seen in classification tasks, datasets
are unbalanced, especially for task B. Further details are given in Amjad et al. [
We also used an extra set of tweets from Batra et al. [
In addition to those datasets, we also conducted a small annotation of the test dataset. Our idea was to have an idea of the annotation agreement between our Urdu speaker and EmoThreat organizers and of the required human efort to conduct the considered task. Those annotations were not used to learn models for our run submissions. This resulted in 60 tweets annotated, among which 14 of them did not receive the same class than those provided by organizers. The agreement rate is 0.78 and Cohen’s Kappa 0.67, which reveals a strong agreement.
Three approaches have been implemented. For both tasks, we used models provided by the HuggingFace repository. In addition, we used and unsupervised MWE extraction for task A and a machine learning and a linguistic-based approach for task B.
In order to classify tweets in relevant categories, we experimented the famous HuggingFace
transformers, by searching available models able to process Urdu (including multilingual ones).
Among proposed models, we conducted a preliminary evaluation for roberta-urdu-small1,
muril-large-cased' [
AutoModelForSequenceClassification2.
Considering task B, we directly implemented the B2 subtask, which allows to submit both B1
and B2 runs. We also extended our dataset with an external corpus of tweets[
Multiword expressions (MWE) are very likely to carry emotions and to be more reliable clues
than single-token words when used as features for text classification. One main issue consists
in being able to spot such MWE when they appear in spontaneous text such as tweets. It
can be very dificult, costly and time-consuming to keep a lexicon up-to-date with the most
recent usage on social media. In a previous work [
We took the opportunity ofered by this challenge to apply the algorithm for Urdu language
(a very diferent script) and to evaluate it in a slightly diferent set-up (3 classes versus 7 classes).
Unlike the previous work on Chinese, we strictly limited ourselves to the provided training
data and did not extract any emotion lexicon from other sources. The algorithm consists in two
steps. Firstly, it extracts autonomous chunks of text based on unsupervised text segmentation
[
As the Perso-Arabic alphabet difers significantly from the Chinese script, we combined two diferent segmentation strategies for the first step. One is character-based, it performs segmentation of a sentence into tokens from the sequence of characters (one can think of this procedure as removing spaces from the input and finding them back). The other one processes input as a sequence of tokens and extracts autonomous multitoken expressions. It does not re-segment the input into larger units, but rather yields all MWE candidates.
One benefit from the logistic regression is that it is very straightforward to inspect. It computes weights for the whole vocabulary of features and allows us to check which words and MWE contribute the most to the decision. 2The AutoModelForSequenceClassification library handles multiclass and task A is multilabel. Due to lack of time, we just selected the latest sentiment in the order of the columns in the dataset to learn models.
The linguistic method addresses challenges such as EmoThreat 2022 by thinking about linguistic
cues that may be used to solve the issue. For emotion classification, linguistic cues are mainly
lexical and afect lexicons, for instance thesauri such as SentiWordNet [
The methodology behind this take on Task B consists in using linguistic features as machine
learning features. To do so, we first annotated both train and test datasets using Stanza’s
Universal Dependencies (UD) model4 for Urdu [
Each word is thus represented as follows: token, token_lemma, token_POS. This method allows us to disambiguate homographs using their lemma or their POS. For instance, ےگ ںیرم ےگ ںیرم رورض They will die, for sure is presented to our model as a set of 15 elements: (1) ںیرم, (2) ےگ, (3) رورض, (4) ںیرم, (5) ےگ, (6) ںیرم_رم, (7) ےگ_ےگ, (8) رورض_رورض, (9) ںیرم_رم, (10) ےگ_ےگ, (11) ںیرم_VM, (12) ےگ_VAUX, (13) رورض_JJ, (14) ںیرم_VM, (15) ےگ_VAUX.5
It is important to note that as any automatic system, Stanza is prone to making errors. Those errors definitely restrain the positive impact our annotation could make, but they are not necessarily a waste. The first reason is that the model only uses the feature it finds useful, and the second reason is linked to the systematic nature of those errors. As long as an error is the same both in the train and the test datasets, it can be used rightfully by the model.
HuggingFace models We report performances of HuggingFace models and number of epochs
in Figures 1 (task A) and 2 (task B2). Comparing models clearly shows that roberta, muril
obtain better scores than albert and bert, while comparison between the former models does
not show significant diferences. The number of epochs doesn’t impact much results. We also
3It is noteworthy that SentiWordNets for other South Asian languages, namely Hindi, Bengali and Telugu, have
been developed [
0.5 0.4 0.3 0.2 0.8 0.75 0.7 note that precision is better than recall on task A, while they are balanced for task B. Our final runs use the roberta model with 3 epochs6.
Results obtained on the train set with cross validation (5 folds) and train / test split are reported in Table 1 and 2. Unsurprisingly, there is a diference between cross-validation and train/test split, which is more important for task B. We also see a great diference between macro (F1M) and micro (F1m) evaluations, which is due to the unbalancing of dataset in terms of categories for both tasks. The oficial scoring scheme relied on macro scores. For task A, HuggingFace models have a precision much higher than recall, this is most probably due to the fact that we naively convert multilabel dataset to a multiclass one.
4 4 4 4 F1 micro Recall F1 micro Recall 6 6 6 6 unsup MWE+logit roberta muril
F1M 0.57±0.02 0.49±0.01 0.52±0.01
Results for task A comparing models in the train set with cross validation (5 folds) and train / test split 1 reveal a significant diference between F1 macro (F1M) and F1 micro (F1m). We note that roberta model outperforms muril on F1M scores. Interestingly, the traditional approach based on a logistic regression outperforms transformer-based models. This may be due to our method being trained specifically and only on the task data, creating a model better suited to this dataset. Our team ranked 5th for task A with our MWE method.
Model muril muril+ext muril+test roberta roberta+ext roberta+test LogReg LogReg+ling LinearSVC LinearSVC+ling HingeSGD HingeSGD+ling 5 Folds Train Cross Validation
Train / Test
We conducted deeper analysis for task B depicted in Table 2, by comparing models and augmented training datasets, with external data (+ext) and with our manually annotated test set (+test). Globally, the complementary annotations do not provide clear improvements, at least for F1 macro (F1M), where it even lowers scores in some cases. We will examine those results in details to understand how those complementary data are used by models.
Linguistic Approach This approach described in 4.3 was combined with traditional machine learning algorithms using Scikit-Learn 1.1.2. We first used the benchmarking of classifiers proposed by scikit-learn7. This allowed us to narrow down the testing algorithms to three: Logistic Regression, LinearSVC and Stochastic Gradient Descent, which results are reported in 7https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html.
Results for both tasks have been reviewed by an Urdu speaker in order to identify error types and try to explain them. 6.1. Task A In order to review this task’s errors, we have analysed the trends of a table displaying the most significant tokens for each of the 6 classes (anger, disgust, fear, sadness, surprise and happiness). Results having a score above 0.5 only have been considered as relevant. Examples are given by descending order of relevance. anger: Unsurprisingly, we find here occurrences belonging to the lexical field of “hatred” ( زیگنارتنف: hateful, ت رنف: hatred, ت ربنفاقل: which deserves hate), adjectives or nouns pejoratively connoted ( ت ریغےب: dishonour, ت نعل: curse, دیدش: serious, ٹھیاگ: mediocre, لہاج: ignorantر,و چ : thief, ٹپرک: corrupt, م اعلتق: massacre), then comes the field of justice ( ف اصنا: justice, ت لادع: court), followed by names of political persons or movements ( نارمع: Imran, سیسایارآ: RSS, ف یرشزانو: Nawaz Sharif ), country names and professional or religious categories ( ی رٹاپو: notary, ت فحصا: journalism, ینیاداق: Ahmadi religious community, ناملسم: Muslim). These named entities are subject of criticism in a restrained political and time context, they cannot be used to build a strong model that could be used for other corpora. A handful of words denoting threat like ت شابدرbear, support or ی رباturn are exclusively found in this category. Regarding utterance, the non-polite second personal pronoun combined with ergative case منےت is very specific of this category. disgust: a great part of the lexicon stressed in “anger” is also found here: ت رنفhatred, ٹھیاگ mediocre, ینیاداق Ahmadi, etc. Some entities and communities are overrepresented here: army )جوف(, military (یجوف), whole world ( نیدایروپ), Muslims )ںنواملسم(, politicians ( نادتسیاس) or country names like ناتسوہدنIndia. fear: We find here occurrences of terms related to “fear”, mixed with religion : ف وخکاادخ( fear of God, ے سرڈےک : in fear of, ےہرڈ : fear is, ف وخ: fright, یلاعہتللا: God Almighty. Other themes are: COVID ( ے سناورک: from Corona), Islam ( ںنواملسم: Muslims), children ( وکںوبچ: to the children), countries (India, China). These occurrences are once again related to specific contexts. Regarding phraseology, several forms of the verb “look like” are recurrent ( ےہگالنگےل: began to look like, ےہہگرال/ےہاگتل: it looks like). The postposition ے س, which indicates the origin and is required to build arguments related to fear, is also very present. sadness : Related to the lexical field of “sadness” ( یسادا: sadness, س ادا: sad, س وسفا: sorrow,م غ : afliction, جمربو: obliged,: س ویام disappointed), “scorned love” ( ٹ:واbroken, ے حسمب:ت by love, broken heart emoji, ےکیلا: lonely, مجنا ا : consequence), and “hatred” ( ت رنف) Some neutral words are typically linked to this category: دیاش(maybe), میلعت (education), children )ںوبچ(.
And more surprisingly: یہنس(laughter), ت اعقاو (event). surprise : Terms related to surprise ( ت ریح: surprise, ہجو ےب: without any reason), sometimes denoting some sarcasm ( ںیمشوخسک: in which honor, ہاو : interjection of admiration, دینقت : criticism) On the phraseology level, the forms of “know” are overrepresented, especially for the plural present verb ( ہںیےتناج), question word ںویک why, On the level of utterance, impersonal subjects like یمدآ man are quite numerous in order to depersonalize the phrase. happiness : The most salient tokens here are not shared with any other category and make this label quite apart. Emojis are ranked at the top. Then comes the field of “joy” ( یشوخ: happiness, راوگشوخ: happy,ا ھ ا ھ ا ھ: ha ha ha, ) ; some religious formulas and expressions of gratitude are found only here ( ہلحلدلما : God be praised, ہکیرش : thanks, ہللا: ّ God, نیمآ: amen, ہللاءاشنا: if God wants) ; positive values and adjectives ( ریاپ: love, ت روصبوخ: beautiful), Surprisingly, words such as : ت وم: death, or ض ارنا: angry are also linked to this category.
Regarding the potential ambiguities, “anger” and “disgust” categories are the closest. Most of significant tokens present in ”anger” are also included in ”disgust”, which is understandable from the context of utterance: tweeters express there grievances against political persons, foreign powers or communities.
Regarding ambiguities in the other categories, the word یشوخ happy is present in 4 out of 6 classes. Adversely, it is most significant in the “anger” category. “sadness” shares quite a number of tokens with “fear” and to a lesser degree with “surprise”. “Surprise” has more shared words with “sadness” and “happiness” appears to be the most singular category as it has the least number of tokens shared with any other category. 6.2. Task B 6.2.1. Overall error reports Among the 3 runs, Run-1 (HuggingFace baselines approach) made less errors. Most errors are about the existence or not of a threat: in 124 tweets, a threat was detected whereas there was none. On the opposite, 106 individual threats were not detected.
The confusion between “no threat” and “collective threat” is less (32 tweets) and there are 25 errors of number.
In Run-2 (HuggingFace baselines approach), there are 27% more errors than for Run-1. The most frequent errors are between “collective” or “individual” threat (167), followed by the confusion between “no threat” and “individual” threat (140). However results are far better here for detecting threat from “individual threat” (only 24 errors)
Run-3 (Linguistic based approach) produced exactly as many errors as Run-2 (HuggingFace baselines approach). It outperformed the former regarding errors between “individual threat” and “no threat” but tagged more often tweets without threat as “individual threats”. It was better at distinguishing between “collective threat” and ”no threat”. 6.2.2. Typology of errors For this task, errors have been identified on 3 levels: lexical, phrasal and deictic. Some elements were at first considered as core features to identify threatening tweets, but turned out not to be useful as they were present in both classes. Regarding the verbal system, we noticed that second person address, subjunctive mode for jussive phrases and future tense to express consequence were more employed in threat tweets. On a lexical level, moderate abusive language and terms related to threat ( نہباشنا to target, یناد ازسto punish, وھکر دیاremember! ) are present in both classes, even if they are more numerous in the “threat” one. Besides, no MWE nor phrase expression seems to characterize the “non threat” class.
Conversely, the errors review revealed some elements belonging exclusively to the “threat” category: 1) multiple clause sentences in which a first clause exposes an order and a second a future consequence, both clauses being linked by نرہو otherwise; 2) MWE using ت شابد رbear co-occurring with a negative element ( نہںیش ابدرdon’t bear, ت شابدرِ لباقنا unbearable); 3) other significant MWE as ینتگ ٹیلا countdown, ہکےہ یاآ ت قو time has come to, ےٹکا رسcut heads.
A comparison of the 3 train sub-corpora (“no threat”, “individual threat” and “collective
threat”) with the textometry software TXM [
Our participation in the EmoThreat 2022 challenge about sentiment and threats in Urdu tweets was the opportunity to test three diferent approaches to address classification tasks: HuggingFace models based on BERT, unsupervised MWE extraction and linguistics-based approach. It turned out that unsupervised MWE extraction models obtain better scores for task A (6 classes of emotions), with much higher precision than recall. Still, HuggingFace models dedicated to Urdu language (roberta, muril) obtained satisfying results. For task B, the HuggingFace obtain better results for sub-task B1 (binary detection of threats) while the linguistics-based approached appeared to be the best one in the sub-task B2 (detection of individual/collective threat). Considering results for both tasks leads to the conclusion that HuggingFace models do not outperform other methods. Our error analysis identified numerous lines of research that we intend to explore in the future.