This paper describes the system submitted by WSSC Team to the EmoEvalEs@IberLEF 2021 emotions detection competition. We propose a novel model for Emotion Detection that combines transformers embeddings with topic information and o ense features. The system classi es social media text emotions leveraging its context representations. Our results show that, for this kind of task, our model outperforms baselines and state-of-the-art text classi cation methods. As for the leader-board, our classi cation model achieved a macro weighted averaged F1 score of 0.661427, and a overall accuracy of 0.675725, reaching the 9th and 10th place respectively.
Emotion Detection is a branch of sentiment analysis that seeks to extract
negrained emotions from either speech/voice, image, or text data. Detecting
emotions from texts has proven to be quite a challenging task, regardless of the
quantity of available data [
Besides that, the need for disambiguating emotion-conveying words in order
to verify classi ed emotions as real emotions still represents a signi cant hitch,
since texts often contain expressions that could refer to di erent emotions. For
example, a phrase like \I can't stand it" could convey anger and disgust
depending on the context. Nonetheless, recently, state-of-the-art results were obtained
by using pre-trained transformer-based models. Needless to say, in the past three
years, pre-trained language models such as BERT [
The existing state-of-the-art emotion detection models usually only extract
context features from texts and pay less attention to external features like the
kind of event these messages were posted for. In our work, we tried to ll this gap
by including additional context information and by also considering the presence
of o enses inside these messages. LASER [
The main contribution of this study is an approach based on a combination of
contextualized work embeddings, topic information, and o ense features
specifically tailored for improving the emotion detection process. We evaluated our
methodology on the EmoEvalEs@IberLEF 2021 [
The remainder of this paper is organized as follows. We rst present the related work, then we introduce our approach, and nally we show the experiment results and the error analysis. 2
There are ve classes of approaches for recognizing emotions in texts:
keywordbased approaches, rule-based approaches, classical learning-based approaches,
hybrid approaches, and deep learning approaches[
Deep learning is a branch of machine learning in which deep neural network
architectures learn from experience and understand the world in terms of a
hierarchy of concepts, where each concept is de ned in terms of its relation to
simpler concepts. This approach allows a model to incrementally learn complex
concepts putting together simpler ones [
Here's a list of the main steps to take when using LSTM for emotion recognition in texts: 1. text preprocessing, that is tokenization, stopwords removal, and lemmatization; 2. encode texts through an embedding layer and then use these embeddings to feed one or more LSTM layers. 3. delivering outputs into a dense neural network (DNN) with units equal to the number of emotion labels and a sigmoid activation function to perform classi cation. 2.2
The encoder block of transformers, initially designed for machine translation,
has become the de-facto standard pre-trained language modeling architecture
for solving most NLP tasks such as text classi cation, text generation, document
summarization, question answering just to name a few [
One of the ways to improve the performance of emotion classi cation is the
extension of BERT model by a linear transformation layer with sigmoid
activation. The proposed model was evaluated using the EmoBank data and obtained
a micro F1 score of 0.688 and 0.695 when ne-tuned on the ISEAR and SemEval
datasets, respectively [
3.1
During the pre-processing step, we only detected and replaced all emojis with their respective short-codes using a freely available Python library emoji. Since data provided by organizers of the competition were polarized, they replaced all the hashtags with the keyword \HASHTAG" in order to prevent the automatic classi er from relying on hashtags to categorize the emotion associated with a tweet. Moreover, the user mentions were replaced by \@USER". 3.2
For representing the input data, we used embeddings generated by two
pretrained transformer-based models: DistilBERT and Language-Agnostic
SEntence Representations (LASER) [
Given an input sentence, LASER provides sentence embeddings which are
obtained by applying max-pooling operation over the output of a Bidirectional
LSTM (BiLSTM) encoder. BiLSTM output is constructed by concatenating
outputs of two individual LSTMs working in opposite directions (forward and
backward). This way more contextual information is included in the output with
respect to a single LSTM reading text from left to right. In our experiments, we
used LASER to embed all tweet sentences into 1024-dimension xed-size vectors.
As additional features, we detected o enses and we extracted topics of tweets.
The both types of features were provided in the EmoEvent corpus. LASER
embeddings are passed as input to a Long Short Term Memory Network model
to encode the social media texts. Finally, we combined all features through the
architecture originally proposed for the detection of fake news articles [
We use the dataset released for the EmoEvalEs@IberLEF 2021 competition [
The dataset is based on events that took place in April 2019 related to
di erent domains: entertainment, catastrophe, political, global commemoration,
and global strike. There are messages in a total of 8 di erent topics. For the task
dataset was split into dev, training, and testing partitions. The distribution of
EmoEvalEs@IberLEF 2021 dataset is shown in Table 1.
The proposed model computes the feature vectors separately and then combines
these with the help of an MLP layer. We use categorical cross-entropy as the
loss function to optimize our architecture with a soft-max layer that tries to
classify any given social media text into one of seven emotion classes. The
hyperparameter setting is shown in Table 2. The full code is provided in the project
repository https://github.com/vitiugin/ComboLASER.
In the current work, we also used schemes with a combination of DistilBERT
embeddings. The concept of distillation in neural networks aims at speeding up
models. The key idea is to replace massive architectures with countless
parameters with a lightweight version of the same architecture that possesses fewer
parameters [
According to recent surveys SVM is the most popular machine learning
scheme for emotion detection from text [
To prove the need of using additional topic and o ense feature vectors we also used only transformer embeddings as input to a LSTM model. 4.4
As evaluation measures we used two multi-class classi cation metrics: accuracy and macro weighted averaged F1 score. The full results on development and test splits are shown in Table 3.
We can observe that the SVM-based models with concatenated feature vectors have high performance even compared with LSTM-based networks trained only on transformers embeddings. Further LASER embeddings demonstrate a higher performance compared with DistilBERT embeddings. The proposed Combo LASER model shows the highest performance, which is perhaps due to the fact that the model takes into consideration the sentence-level context encoded in the LASER embeddings. In terms of performance, the proposed solution is worse than the solution that took the rst place by 4.5%.
Analysing our model mistakes, we found that our model often (more than 50% compared to the volume of class in tested data) mis-classi ed Disgust as Anger and Fear as Sadness. On the other hand, the best results were achieved for Sadness, Surprise, and Others (less than 25% of mistakes).
We also found out that two pairs of emotions detected with mistakes on the both sides: of Anger {Disgust and Joy {Other. While the similarity of the rst pair could be explained by close nature of this emotions, the second pair could be explained only by the size of trained and test data. Joy and Others classes are over-represented in the train and test data. 5
In this paper, we explored the bene t of incorporating transformers, topic
information, and o ense features to deep neural networks on the task of
multiclass emotions detection. We also presented our model based on extracted
pretrained LASER embeddings. Experiments on the dataset released during
EmoEvalEs@IberLEF 2021 competition demonstrate that our Combo LASER model
performs better than several baselines and additional features improves the
model performance compared with models based only on transformer
embeddings[