Lithuanian language is a complex and rich language with a unique grammar structure, making it an interesting choice for natural language processing (NLP) tasks such as emotion detection. This study provides helpful insights into the emotional nuances of Lithuanian texts by utilizing a translated and augmented emotion dataset. We present a methodology that leverages translated datasets for emotion recognition and augmentation approaches to improve the performance of emotion identification models in low-resource languages. We compared the outcomes of transformer-based language models, such as RoBERTa, LaBSE, and LitLat BERT on the translated and augmented data. Our results demonstrated that LitLat BERT, which is primarily trained on Lithuanian texts, showed the most significant improvement in performance when data augmentation was applied. We conclude that LitLat BERT could be the preferred choice for emotion recognition tasks for Lithuanian language due to its specialized training and enhanced adaptability when provided with diverse and augmented data. This study provides valuable insights into the challenges and potential solutions for emotion identification tasks in morphology-rich languages, like Lithuanian language. NLP, emotion detection, transformer-based models, BERT, translation, augmentation, 28th International Conference Information Society and University Studies - IVUS 2023 Proceedings
Emotion recognition in texts is an important natural language processing (NLP) task and has wideranging applications, including sentiment analysis, customer support, mental health monitoring and others. With the rapid development of deep learning techniques, significant advancements have been made in emotion recognition for high-resource languages, such as English, Chinese and Spanish. However, research in low-resource languages, like Lithuanian language, remains limited due to the scarcity of annotated datasets and the lack of pre-trained models.
In this paper, we propose a methodology for improving emotion recognition performance in lowresource and morphology-rich languages, using translated datasets and augmentation approaches for the Lithuanian language.
In order to reach our goal, we perform the following tasks: related work analysis (Section 2), dataset analysis (Section 3), methodology for translating and augmenting datasets, as well as fine-tuning transformer-based language models (Section 4), experimental evaluation of different fine-tuned models (Section 5), conclusions and future plans (Section 6).
2023 Copyright for this paper by its authors. CEUR
ceur-ws.org
Research in emotion recognition for low-resource languages has been an area of growing interest in
recent years [
Also there have been studies, where emotion detection datasets across 19 languages were trained on
a multilingual emotion prediction model, XLM-EMO, for social media data, which showed competitive
performance in a zero-shot setting and is particularly useful for low-resource languages [
In another study, researchers addressed the shortage of annotated gold standard resources for
emotion mining by presenting a multilingual emotion dataset of tweets in English and Spanish, labeled
with one of seven emotions and proposed a machine learning approach for automatically detecting
emotions in tweets for both languages [
Also, researchers conducted experiments on sentiment and emotion recognition for English and
Polish texts in the context of a chatbot, creating a parallel corpus named CORTEX and employing
various classifiers such as Support Vector Machines, fastText, and BERT, with BERT-based models
yielding the highest accuracy and F1-scores, although results for Polish were slightly inferior to those
for English [
However, these methods are often problematic due to the need for parallel data, noise adding, or relying on uncontrolled training goals that may not match the target task.
We selected the CARER (Contextualized Affect Representations for Emotion Recognition) [
Type Training Validation
Testing Total
This dataset is a collection of 20,000 English social media text samples, with each sample annotated with one of the six emotions. The dataset is split in the following way: 16,000 samples for training, 2,000 samples for validation, and 2,000 samples for testing. The data is diverse, containing a variety of sentence structures and extensive vocabulary, allowing the models to learn the nuances of different emotions in textual data. Label distribution is presented in Figure 1 and text length distribution is presented in Figure 2.
1. High-quality annotations: accurate and well-annotated labels for model training and evaluation. 2. Diversity of emotions: wide range of emotions for learning linguistic patterns and improved emotion recognition. 3. Transfer learning suitability: English dataset enables transfer learning for low-resource
Lithuanian tasks using translations. 4. Compatibility with models/frameworks: Widely used dataset, compatible with pre-trained models, simplifies incorporation and focuses on emotion recognition challenges.
Summarizing, the dataset was chosen due to its high-quality annotations, diversity of emotions, suitability for transfer learning, and compatibility with existing models and frameworks. By leveraging this dataset in conjunction with language-specific models and translation techniques, we can effectively address the challenges associated with emotion identification in low-resource languages such as Lithuanian.
The methodology part covers the methods applied in this research: (1) translation of the dataset using neural machine translation models; (2) augmentation process for enriching and balancing the dataset; (3) fine-tuning transformer-based models for emotion classification task.
For the emotion recognition task, the choice of translation service plays a crucial role in generating
high-quality translated datasets. To increase the linguistic diversity of our dataset, we translated these
datasets using two state-of-the-art machine translation models:
1. Opus-MT [
However, depending on specific requirements and constraints of certain projects, DeepL [
By combining and translating these datasets, we aimed to create a rich, diverse dataset that covers a wide range of emotions, contexts and social texts.
When choosing a translation service, it is important to think about the quality of the translation, the number of domains it covers, and how well it fits the task at hand. In some situations, it might be helpful to improve the quality of the translated information by using a combined approach, merging translations from more than one service.
After combining the translations, generated by both models, we obtained a dataset with the following
distribution of emotion labels, as displayed on Figure 3.
It was observed that certain emotion classes, specifically love and surprise, have a significantly lower
number of samples in the dataset. To address this imbalance and improve the performance of our
models, we decided to augment the dataset by applying synonym replacement to the underrepresented
classes. Synonym replacement has showed good results for dataset augmentation [
In our study, we employed data augmentation techniques to enhance the performance of the emotion recognition models and improve their robustness to different translations and imbalanced emotion distributions. For this purpose, we used synonym replacement as our data augmentation method. In this section, we describe the process of synonym replacement and its importance in the context of the Lithuanian language.
Synonym replacement is accomplished by replacing words in a given text with their synonyms in order to create new and slightly different variations of the original text while preserving its meaning and sentiment. The process of synonym replacement is shown in Figure 4.
We used the following algorithm to implement synonym replacement in our study:
1. Tokenization: Tokenize the Lithuanian text samples using a spaCy [
After analyzing the dataset, we decided to add additional augmented texts for the labels love and surprise, which had the fewest occurrences. The updated distribution of emotion labels in the dataset is presented in Figure 5. This balanced distribution of emotion labels enables the models to learn better representations of different emotions in the text, and potentially improves their overall performance in the emotion identification task. Final version of the dataset is shown in Table 2.
We chose three transformer-based models for our experiments for investigating the impact of
translation quality and data augmentation on emotion recognition tasks in Lithuanian:
1. RoBERTa [
By comparing these diverse models, we aimed to identify the most efficient approach for emotion recognition for the Lithuanian language, and evaluate the impact of translation quality and data augmentation techniques on their performance.
Experiments were carried out in a series of tasks: 1. Firstly, the quality of translations was assessed. 2. Secondly, base model for English language was fine-tuned using original English dataset. 3. Thirdly, the obtained word embedding models were evaluated using the adapted intrinsic evaluation benchmarks. 4. Finally, the resulting data was examined in order to determine the effect of different hyperparameters on benchmark evaluation results.
In our study, we used two translation models to generate Lithuanian versions of the dataset. Due to the morphological richness of Lithuanian, achieving consistent translations can be challenging. Translation models may struggle to generate accurate translations, which can impact emotion identification performance.
However, as long as the main sentiment remains consistent, this impact is expected to be minimal.
Training models on diverse translation styles can improve their robustness and performance in
realworld scenarios [
To assess translation quality, we calculated the BLEU (bilingual evaluation understudy) 0.2337 and WER (word error rate) 0.3349 scores for the translations. These scores indicate significant differences between the two models, attributable to the complex nature of the Lithuanian language. Despite the differences, the main idea is generally preserved, enabling emotion identification models to learn from the dataset. This highlights the importance of translation quality in low-resource language tasks and the need for more robust models to handle translation inconsistencies.
In order to establish a baseline for comparison, we trained a base model on the original English dataset. This experiment allowed us to evaluate the performance of the emotion identification models when trained on a high-resource language, like the English language, and compare the results with those obtained for the Lithuanian translations.
We selected the xlm-roberta-base model for this experiment, as it has demonstrated strong
performance in various NLP tasks [
As expected, the xlm-roberta-base model achieved a high level of performance when trained on the English dataset, with an accuracy of 0.9305 and an F1 score of 0.9313. These results serve as a benchmark for assessing the performance of the emotion identification models trained on the translated Lithuanian datasets, enabling us to better understand the challenges and potential improvements associated with semantically rich and low-resource languages.
In the second training experiment, we aimed to evaluate the performance of the emotion identification models when exposed to a diverse range of translation styles and potential errors. For this purpose, we combined the datasets generated using both translation models. This approach provided the models with a more varied and potentially challenging dataset, allowing us to assess the model's adaptability and robustness in the face of translation inconsistencies. Results are provided in Table 4.
Comparing the results of the experiments, we observed that when trained on the original English dataset, the xlm-roberta-base model achieved the highest accuracy, precision, recall, and F1 score. However, when using translated dataset, the performance of xlm-roberta-base decreased but remained competitive. In this scenario, LaBSE showed slightly better accuracy and F1 score compared to lmroberta-base, while LitLat BERT (litlat-bert) had marginally lower results.
This comparison demonstrates that translation quality and the choice of translation model can impact the performance of emotion identification models in low-resource languages like Lithuanian. Moreover, they emphasize the need for developing more robust models that can effectively handle translation inconsistencies and variations in the text.
In the third training experiment, we aimed to evaluate the performance of the emotion identification models when exposed to a diverse range of translation styles, potential errors, and augmented data. For this purpose, we used the translated dataset, and, additionally, applied data augmentation techniques. This approach provided models with a more varied and potentially challenging dataset, allowing us to assess the models' adaptability and robustness in the face of translation inconsistencies and the additional variations introduced by data augmentation. Table 5 shows the results.
When comparing the results obtained from the models trained with and without data augmentation, we can observe significant differences. After data augmentation was applied, the performance of LitLat BERT reached a highest accuracy with 0.813 and F1 score of 8 of 0.81.
The improvement in LitLat BERT's performance can be attributed to the fact that it is primarily trained on Lithuanian texts, unlike the other multi-language models. This specialized training allows LitLat BERT to be more sensitive to the nuances and linguistic patterns specific to the Lithuanian language. When the dataset is augmented, the model can effectively leverage its knowledge of Lithuanian text to adapt to a broader range of linguistic variations, improving its overall performance. Consequently, LitLat BERT could be the preferred choice for emotion identification tasks in Lithuanian due to its targeted training and enhanced adaptability when provided with diverse and augmented data.
This study marks the initial effort to translate and adapt a training dataset for downstream NLP tasks in the Lithuanian language. While we did not achieve the same performance level as the original English dataset, our findings demonstrate that models can effectively be trained on translated and augmented data for the Lithuanian context.
In summary, data augmentation, using synonym replacement, is an effective method for enhancing the performance and robustness of emotion identification models in low-resource languages like Lithuanian, by addressing the challenges posed by the morphological richness of the language and generating diverse and balanced training data.
In the future, we intend to accomplish a more in-depth study of how well these services translate in the context of detecting emotions. We also intend to look into ways of improving the translation quality by fine-tuning the translation models. 7. References
Available: