Emotion detection in text plays a crucial role in various applications, such as customer feedback analysis, social media monitoring, or for the analysis of the verbal part of human communication. Deep learning techniques have shown promising results in accurately recognizing and classifying emotions in textual data. This paper describes the approach to categorical emotion detection of the Emotion Hunters team. After a preprocessing phase, a model fine-tuned from AlBERTo together with the ChatGPT APIs was used to address the challenge. The results show that on the out-of-domain test set our approach performed better than on the in-domain one thus showing a good generalization capability.
for other less-resourced Indo-European languages, such as Italian.
Emotion detection in texts has gained significant impor- The EMit (Emotions in Italian) Subtask A [
As far as emotion detection for the Italian language surprise, trust, neutral [
to the topic. This is for us an interesting result because we want to apply the model in different domains like the one of conversational experiences with intelligent agents.
We did not have the time within the challenge deadline to train a model based only of LLMs like ChatGPT and this is part of our future work.
A pipeline of preliminary operations is performed in
order to rfist clean and standardize the input messages and
then prepare them for a format suitable to the selected
learning algorithms [
To augment the number of sentences of the fear class, rationale behind these operations is to make tokens and
we integrated the training dataset with sentences taken symbols expressing emotions explicit and jointly to
augby the dataset MultiEmotions-It [
ants), which have achieved state-of-the-art results in text classification tasks. BERT utilizes special tokens [CLS] and [SEP] to indicate the beginning of the input sequence and the separation between sentences.
In this specific case, the contextualized embedding associated with the [CLS] token is used as the embedding for the entire sentence. Thanks to the multi-head attention mechanism, it can capture the semantics of the entire sentence effectively.
Several instances of pre-trained BERT models that
contain at least some Italian text in their training corpus were
considered. For each model, a fully connected layer was
added to perform multi-label classification and fine-tune
the models on the specific dataset. The models considered
are:
• dbmdz/bert-base-italian-xxl-cased [
The best trials were achieved with the BERT AlBERTo model, with learning rates ranging from 2e-05 to 3e-05, a patience value of 3 allowing for training for 4 epochs, a batch size of 16, and employing the "transform" preprocessing strategy. The best trial, in particular, was achieved with the following hyperparameters:
ing code to reproduce their execution. The first baseline uses count vectors to represent the documents based on token frequency, while the second baseline uses TF-IDF vectors. Both implementations limit the vector dimen- AlBERTo and dbmdz had a similar performance, howsions to 5000. In the emotion recognition task, the TF- ever, we selected AlBERTo with an average F1 score of IDF baseline performs better, achieving an F1 score of 0.562 also because it had a better performance in classi41.48%. fying fear, which was one of the most problematic due to
For both the baselines and the experiments conducted the limited number of examples (see Figure 3). in this work, a seed was fixed to ensure reproducibility.
Taking into consideration the work presented in
GoEmotions [
Various preliminary experiments were conducted by number of neutral examples, and in some cases, the model manually modifying the model’s hyperparameters, such was unable to determine the emotion, leaving the result as the number of epochs, batch size, learning rate, etc., to ifeld empty. We noticed this problem only on the test set understand which was the better approach for this task. To and not on the validation set, therefore, to address this improve the results, the following decisions were made: issue, in addition to using our trained model, we integrated The Optuna library was adopted to systematically test the ChatGPT APIs to obtain additional results to fill in different combinations of hyperparameters. The MLFlow the case of neutral or undetermined sentences. Then, library was used to track intermediate (F1) and final results each sentence in the test set is pre-processed and given (F1 and metrics for each class). as input into the fine-tuned model, with a threshold set at
The hyperparameter search space was defined as fol- 0.5. At the end of the prediction phase, every sentence lows: that didn’t receive any label or was classified as neutral is passed to a Python program that utilizes the ChatGPT APIs 3.5. Below, we show the prompt used and some of the examples provided to ChatGPT. Due to the limitations of the free APIs, the number of tokens and the amount of input examples are limited. • Learning rate: between 2e-05 and 5e-05 • Epsilon (AdamW optimizer): between 1e-8 and 1e-6 • Hidden dropout probability: between 0.1 and 0.3 • Patience for early stopping: a discrete interval between
1 and 5 • Batch size: 16, 32, and 64
prompt = "Your are an emotion recognition tool for
The numerous trials conducted using Optuna allowed us ˓→ tweets and your task is to " \ to observe that the models pre-trained consistently on an ˓"→analeymzoetitohnesm, asnedpagriavteedabsyincgolmemaemtohtaitonyoourmaiglhitstthoifnk Italian text corpus performed better than those pre-trained ˓→ are expressed in the current tweet and you should on a multi-lingual corpus, including Italian. ˓"→thisusleisotnl[y'atnhgeere'm,ot'iaonntsicfirpoamti"on\', 'disgust', 'fear',
In general, it is observed that training that exceed the ˓→ 'joy', 'love', 'neutral', 'sadness', 'surprise',
fourth epoch often result in a degradation of performance ˓→ 'trust']"
in terms of F1 score, and the ideal batch size was found to examples = """ \
be 16. On average, the performance of all models benefits IHnepruets"oImoe aenxcaomrpalenso:n ho capito se la voce mentre
from the preprocessing step, as this strategy likely retains ˓→ cantano sia modificata o meno
more informative content and includes typical social me- ˓I→npu#tIl"CRaTnt@aunsteerM:asTcahretraartuog"a Oiustptuhte:nneeuwtrzaolccola enorme
dia expressions in standard Italian [
The results in the following Tables 1 and 2 suggest that our approach, even if it is not the best model of the challenge, at least generalizes quite well to a domain different from the training one. This is for us a good result since we aim at applying the model in contexts different from social media analysis. In particular, we are working on the multimodal analysis of human communication with conversational agents, in which the analysis of the textual part of verbal communication can be very important to fully understand the emotional state of the user. We are actually exploring the performances of models based on