This paper presents the approaches proposed by the I2C Group to address MentalRiskES: Early Detection of Mental Disorder Risks in Spanish, as part of IberLEF 2023. Our proposal involves developing distinct transformer-based classifiers to tackle three specific tasks: i) Task1a: Binary classification for the detection of eating disorders, ii) Task1b: Simple regression for the detection of eating disorders, and iii) Task2c: Multiclass classification for the detection of depression. The main approach consisted of fine-tuning pre-trained transformer-based models. For the binary tasks, diverse methodologies were employed to predict users based on the predictions obtained from their individual messages. For the multiclass task, data augmentation approaches were used to balance the minority classes messages. The final submitted predictions achieved a Macro-F1 score of 0.641 for Task1a, ranking 19th out of 22 participants; an RMSE of 0.24 for Task1b, ranking 4th out of 17 participants; and a Macro-F1 score of 0.232 for Task2c, ranking 4th out of 10 participants.
Additionally, data augmentation [
The subsequent sections of this paper are structured as follows: Section 2 provides an exploration of related works, examining previous research carried out in the respective field. Section 3 outlines the dataset and tasks. Section 4 details the different approaches used to achieve each task. Section 5 presents the results and analysis obtained from the conducted experiments. Lastly, Section 6 offers conclusions derived from the study.
The significant increase in the number of mental disorder cases due to the COVID-19 pandemic has attracted a growing number of researchers to this field of study, resulting in various proposals and methodologies.
These proposals and methodologies are closely intertwined with the exponential growth of social media worldwide. The information derived from this vast amount of social media data is crucial for building these systems and conducting experiments.
A survey was conducted to examine case studies involving experiments [
The dataset supplied by the organizers comprises a collection of labelled messages transmitted to group channels on the widely used Telegram platform. The corpus was split into three distinct subsets, each associated with a distinct disorder. Each dataset encompasses several hundreds of users, with an average message count of 50 per user.
• •
Corpus eating disorders (Task 1): 175 users for training, and 150 for testing.
Corpus depression (Task 2): 175 users for training, and 150 for testing.
For each user, a json file was provided. Additionally, a distinct file containing the labels of each user was provided as well. By utilizing both files, a training dataset was created. The labelling scheme applied to each user's messages was consistent with the user's label. For instance, if a user was assigned label 1, all their messages were individually labelled as 1.
Two distinct training datasets were created, one for Task 1 and another for Task 2. The dataset for Task 1 consists of five columns: user, message, date, label_1a (label for Task 1a), and label_1b (label for Task 1b). Table 1 shows the structure of the training dataset for Task 1.
On the other hand, the dataset for Task 2 is composed of 4 columns as it can be seen in Table 2: user, message, date, label_2c (label for Task 2c).
After constructing the datasets, they were split into three sets: train, test, and valid. The train and valid sets were allocated 80% of the data, while the test set received the remaining 20%. Within the train dataset, 10% was further allocated for validation. Table 3 shows the distribution of examples for each dataset.
The distribution of the classes in each created dataset can be observed in Table 4 and Table 5. Furthermore, Table 6 provides an overview of the distribution specifically for Task 2c.
As it was described before, this paper is focused on Task 1 (Task 1a and Task 1b) and Task 2 (Task 2c). Task 1a is a binary classification in which it must be detected if the user suffers from anorexia or bulimia. Labels will be 0 for “control” (negative, the user does not suffer from eating disorder) or 1 for “suffer” (positive). Task 1b is a simple regression in which a probability for the user to suffer anorexia or bulimia must be predict. A value of 0 means 100% negative and a value of 1 would be 100% positive. Finally, Task 2c is a multiclass classification. The system must predict one of the four classes (“suffer+against”, “suffer+in favour”, “suffer+other”, “control”).
In this section, we provide an in-depth explanation of the various methodologies employed to address the tasks. We will delve into the specific experiments conducted for Task 1 and Task 2, outlining the steps taken and the techniques employed. Additionally, we present the results obtained from these experiments, which will be thoroughly analyzed and discussed in the subsequent sections. In this section, we aim to provide a clear understanding of the experimental framework employed in our study.
To perform Task 1a and Task 1b, different transformer-based models, available in the Huggingface
[
In order to prepare the data for analysis, several preprocessing methods were applied. This involved removing links, usernames, hashtags, and emojis from the text. By eliminating these elements, we aimed to focus solely on the textual content and minimize any potential noise or distractions.
Furthermore, to ensure optimal model training, a study of hyperparameters was conducted. This consisted in exploring different combinations of hyperparameters values to identify the most effective settings. To streamline this process, we leveraged the WandB tool, which provided valuable insights and facilitated efficient experimentation. Our objective was to optimize the performance and accuracy of our models by fine-tuning the hyperparameters, ultimately resulting in outcomes that are more robust and reliable.
After completing the training process and generating predictions for the individual messages, a user prediction methodology was employed for each task. In the case of Task 1a, once labels were predicted for each message of a user, a percentage-based approach was applied. The user's label will be determined by considering the majority label among the predictions generated for their individual messages. In other words, if a significant proportion of the user's messages are consistently predicted to belong to a particular class, that class will be assigned as the user's label. This approach ensures that the user's label reflects the prevailing classification consensus derived from their message predictions. For example, if a user has 20 messages and 14 of them were classified as 1, the user was automatically labelled as 1.
The user's label was derived by computing the average value of the predictions assigned to their individual messages. By aggregating the predicted values in this manner, we obtained a representative label that captures the overall tendency of the user's messages. This averaging process allowed us to capture the collective sentiment or characteristic exhibited by the user's messages and assign a label that reflects their overall classification.
Figure 1 shows a summary of the methodology used for Task 1.
Once this first study was done, a new study of hyperparameters was carried out with the models that obtained the best results in the previous experiment. The hyperparameter search was done using the WandB tool with strategy grid. Table 8 shows the hyperparameters space used for this study. Each model was trained with these combinations of hyperparameters, and the best four combination values are shown in Table 9.
Table 10 provides a comprehensive overview of the results obtained for Task 1a using the optimal hyperparameters. By carefully selecting and fine-tuning the hyperparameters, we aimed to maximize the performance and achieve the most accurate predictions for this specific task.
In order to gain a deeper understanding of the failures, we conducted an analysis of the errors made by the models. Specifically, we delved into the results obtained from the best model. Figure 2 depicts the confusion matrix for H2-roberta-base-bne model when making user predictions. It is worth noting that the model exhibits superior performance in classifying users who do not have an eating disorder. A noteworthy observation is that 91% of users with an eating disorder were correctly classified, while users without an eating disorder were classified with 100% accuracy.
Upon examining the two users incorrectly labelled as 0 by the model, it becomes apparent that both individuals made comments pertaining to topics such as the “gym” and “food”, including references to “supplements”, “proteins”, “BCAAs”, and similar content. Such messages have the potential to confuse the model since not all individuals making such comments necessarily have an eating disorder. Consequently, correctly solving this task can prove challenging due to the nuanced nature of distinguishing between individuals who genuinely require assistance and those who do not solely based on such comments.
Taking into consideration the best model results in Task 1a experiments, the two models with highest results were trained to perform Task 1b. Results are shown in Table 11.
To perform Task 2c, the model with best results in Task 1a was trained. The experiments done in this task are focused on roberta-base-bne model as it was the one that obtained best scores. The first phase of the study consisted of training the model mentioned before with the hyperparameter combination H2.
However, the results were not as expected, and another experiment was done. In addition to applying the above hyperparameters, a data augmentation method was applied to the train dataset to improve the results. Moreover, the training dataset has been restricted using only the three most representative classes: "control", "suffer+against" and "suffer+in favour". This technique is justified on the grounds that the "suffer+other" class could generate ambiguity during the model learning process, due to the similarity of the messages with the other two classes, "suffer+against" and "suffer+in favour".
The technique of back translation [14] was used with the minority classes, translating messages from Spanish to English, English to French and French to Spanish. Table 12 shows the distribution of the train dataset before and after the data augmentation method was applied.
Once the individual messages of each user were predicted, a method was employed to determine the class of each user. In this case, the classification of the user was determined by identifying the class that received the highest frequency of predictions. By selecting the class that was most commonly predicted across the user's messages, we aimed to assign a definitive label to each user that reflects the prevailing classification consensus. Figure 2 shows the methodology used in this task.
Results of both experiments are shown in Table 13, where Class 0 refers to “Control”, class 1 “Suffer+against”, class 2 “Suffer+in favour” and class 3 “Suffer+other”. The scores did not meet our expectations and it could be improved in the future. The model H2-roberta-base-bne refers to the first experiment. The second technique used is represented by the model H2-DA-roberta-base-bne, where data augmentation was applied and the class “suffer+other” was ignored.
For the evaluation phase, model H2-roberta-base-bne was used for Task 1a and Task 1b and H2DA-roberta-base-bne model was used to overcome Task 2c.
Our results [15] in the competition for Task 1a among the participants are shown in Table 14 and Table 15. The results were not as expected, and it will be improved in future works.
Table 16 and Table 17 show the results of the competition of Task 1b among the participants. In this case, position 4 was achieved, so the strategies used to overcome Task 1a were more useful to carry out this part of the task.
The observed promising results can likely be attributed to the specific training technique employed. By focusing exclusively on the three majority classes ("control," "suffer+against," and "suffer+in favour") during the training phase, while bypassing the "suffer+other" class, it is plausible that the model achieved heightened effectiveness, leading to favorable outcomes. This selective approach to training allowed the model to concentrate its learning on distinguishing between the most prevalent classes, potentially contributing to the attainment of highly satisfactory results.
In this paper, we presented our proposal for MentalRiskES (Early detection of mental disorders risk in Spanish) and the results obtained in the shared Task for IberLEF 2023. Our approach consisted of fine-tuning transformer-based models using models from Hugging Face library. Moreover, a study of hyperparameters was carried out. Different approaches have been applied to each classifier to achieve the proper results. Our final model for Task 1a achieved a 0.641 Macro-F1 and got the nineteenth position in the ranking. For Task 1b, our model achieved a 0.240 RMSE and got the fourth position in the ranking. Finally, for Task 2c our model achieved a 0.232 Macro-F1 and got the fourth position in the ranking. It is worth mentioning the first place achieved in ERDE Multiclass Classification of Task 1c.
In future works we plan to explore other techniques of creating ensemble as well as do more exhaustive hyperparameters search for each classifier and apply this technique to make a real investigation of the early detection of mental disorder on Telegram.
[8] Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805. [9] Pérez, Juan Manuel, Juan Carlos Giudici, and Franco Luque. 2021. Pysentimiento: A python toolkit for sentiment analysis and socialnlp tasks. arXiv Preprint arXiv:2106.09462. [10] Wu, Shijie, and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. arXiv Preprint arXiv:1904.09077. [11] He, Pengcheng, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decodingenhanced bert with disentangled attention. arXiv Preprint arXiv:2006.03654. [12] Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv Preprint arXiv:1907.11692. [13] Yang, Zhilin, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V.
Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32. [14] Sugiyama, Amane, and Naoki Yoshinaga. 2019. Data augmentation using back-translation for context-aware neural machine translation. Paper presented at Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019). [15] Mármol-Romero, Alba María and Moreno-Muñoz, Adrián and Plaza-del-Arco, Flor Miriam and Molina-González, María Dolores and Martín-Valdivia, María Teresa and Ureña-López, Luis Alfonso, and Montejo-Ráez Arturo. 2023. Overview of MentalRiskES at IberLEF 2023: Early detection of mental disorders risk in spanish. Procesamiento Del Lenguaje Natural 71.