This work is an overview of the CIMAT-NLP submission at the shared tasks of MentalRiskES at IberLEF 2023, which consisted of three diferent tasks related to the detection of mental illness on social media text in Spanish: detection of eating disorders, depression, and an unknown disorder. In this work, we proposed using models based on RoBERTuito using two ideas for their training and evaluation. The main idea of our work is to make packages of messages text to create new instances with more information, as the messages provided in the training data are small in length words. Our proposed models were first on precision metrics in Task 1b and fifth in RSME on same subtask. In depression Task 3a our model placen iffth in Macro-F1 and second on ERDE30.
A mental illness (or mental disorder) is a disease that involves diferent alterations in thinking,
emotions, or behavior, the mental disorders are usually related to stressful events or problems
functioning in social, work, or family activities [
One in eight persons sufered from a mental disease in 2019, where the most common
disorders were depression and anxiety, the number of persons afected by these disorders
incremented during the COVID-19 pandemic [
Automatic detection of mental illness is a complex problem because of diferent factors; the complexity of the mental illness to detect, the data available for the training of the models, the tools developed for this task and other factors.
The problem of automatic detection of mental disorders worsens in Spanish speaking countries. The predomination of the tools, programs, and models for the task is for the English language. In response to the absence of automatic systems to detect mental disorders in Spanish, MentalRiskES is a novel task on early risk identification of mental disorders in Spanish comments. With this novel task, the teams intended to do an online detecting duty, where the point was to detect a potential risk as early as possible, by simulating that the systems received messages like in a normal conversation.
Our proposed work has two main ideas: (1) Packaging of text for training and evaluation
and (2) data augmentation. We used models based on transformers pre-trained in Spanish
social media corpus, which is RoBERTuito [
The rest of this paper is organized as follow: In Section 2 we introduce history about the detection of mental illness on social media. Section 3 describes the competition and the distribution of the datasets. In Section 4 we talk about our approaches for the diferent tasks and in Section 5 we decide the models to use for each task with their results on the competition.In Section 6, we aboard the ethical issues of the competition and the environmental impact. Finally, Section 7 contains the conclusion of this work.
In mental illness detection, one of the essential workshops is the CLEF eRisk: Early risk prediction on the Internet. The main objective is to develop automatic systems that alert depending on the task. Because of the importance of this workshop, most of the world had been devoted to the English language, but now MentalRiskEs tackles Spanish for the first time.
The majority of the proposals in eRisk since 2017 are Bag of Word-based or Neural Network
based. In more recent editions, such as in 2020, the transformers started being used; one example
in [
Other works using transformers models are had been at the top of the Shared task on Detecting
Signs of Depression from Social Media Text (LT-EDI) contest in 2022. The LT-EDI contest focuses
on depression detection without the early detection factor, the works of [
In MentalRiskEs, the messages have diferent lengths; the majority of these messages could be considered short messages. The problem is that these messages, as instances, could not be enough to make decisions about the user; context is essential for model languages as they help to make a decision based on data. The messages may not contain relevant information and could mislead the users’ emotional state. Because of the previous ideas, we hypothesize that the union of several messages to be treated as one would give the necessary context for a better decision.
An approach to the problem of short messages is in the work of [
The datasets of MentalRiskES are data consisting of messages from telegram users. The eating disorder and depression sub-datasets have the same number of users: 10 for trial, 175 for training, and 150 for testing. The third task only has 150 users for testing, as the organizers intend that the teams use their subsystems developed in the previous tasks.
As the primary purpose of the competition is to predict the risk of mental illness the earliest possible, they have a server where each team communicates to get the test data. Because they wanted to simulate a conversation in real-time, we have one message per-round and predict the possibility of each user having a mental illness. In each round, we got a list of users with the id_messages, names (for example, "subject1"), messages, and dates of the messages.
The first task intention was to detect if users sufer from bulimia or anorexia. This problem is sub-divided into two sub-tasks, a The first subtask was a binary classification: only predicting one of the two labels ("sufer" or "control"). b The second subtask was simple regression: the probability of sufering the mental illness.
For the distribution of sufer and control users see Table 1, the same datasets were used for both subtasks.
Training Total users
Sufer label Control label
This problem is sub-divided into four sub-tasks, a Binary classification: only predicting one of the two labels ("sufer" or "control"). b Simple regression: the probability of sufering the mental illness. c Multiclass classification, only predicting one of the four classes (“sufer+against”, “sufer+in favor”, “sufer+other”, “control”). d Multi-output regression: as simple regression but the probability corresponding to each class.
Similar to Task 1, the subtasks uses the same dataset for training and test.In Table 2 we only show the distribution respect if a user has or not the mental illness, because we did not participate in subtask 2c and 2d.
Training
Trial Test Total users
Sufer label Control label
The last task was to detect an unknowing disorder (anxiety). Because we did not have a training corpus and previous knowledge about the disorder to be detected, we used the training corpus from Task 1 and Task 2 to make a model that detects general mental illness. This task has identical subtasks to Task 1. In this task, the number of users with the "sufer" label is 93, and the "control" label is 57.
Because the dataset is a Spanish corpus, we also use transformers pre-trained in this language.
RoBERTuito is a pre-trained model for content in Spanish [
As RoBERTuito is based on RoBERTa, the range of text length input is a maximum of 768 tokens. The training dataset from MentalRiskEs has messages with small lengths. If we join all the messages from each user, most users’ complete history has a mean of 500 tokens. Almost all users’ complete history can be given to RoBERTuito without truncation. The main problem with this approach is that all the records have diferent lengths. The diference between lengths could lead to the model specializing in learning a fixed number of the tokens (the more repeated lengths), and the user’s objective could not be the best approach, as we have small information in each round.
In our approach, we decided to make packages from joined messages for the training and
validation dataset as in [
In the training and validation data, a fixed size of messages is joined into packages because most messages in the competition corpus have a small length. When we merge the messages into packages, we add more information to the instance. Consequently, the model would be better for predicting the label for each package in the validation data. The order of the joining is date order, from the earliest to the latest, as we hypothesize that the temporal information is helpful as they simulate the user’s mood during their history of messages.
For the prediction of the users of validation data, even though the predictions are individual, the user’s decision is made by a voting scheme, which means that if the model predicted more often "sufer" labels, then the user will be labelled as "sufer," or in another case is labelled as "control."
In Fig.1, we have illustrated the packaging strategy described before for one user. First, the user has instances, and then taking subsequent messages, we join them in one package, so the number of packages will be /.
Instead of predicting all the messages, the model predicts the / instances created and ifnally makes one prediction for the user using the voting scheme and the new instances, the process is represented in Fig. 2.
For the experiment, we tried diferent learning rates and train batch sizes, where the evaluation
dataset is the new validation dataset mentioned before. The size of the packages tried were 3, 5,
and 10 for the training dataset following the same ideas in [
One problem that surges with the packaging strategy is the reduction of instances for training. The number of instances is essential because the transformer models usually perform better when they have big datasets instead of small ones. To solve this problem, we developed a technique for augmentation data.
For the augmentation data technique, we still used the process of making packages as in the previous subsection. The new part of this process is to make the packages with random messages from the same users to make more instances. As in Subsection 4.1, the packages are made from messages from the same user intending to preserve the user’s writing style, and the new instances are correctly labelled.
First, we make the packages in the process mentioned in the Subsection 4.1. Then, the packages for the augmentation data have messages selected randomly without the date as an essential factor for the joining to create new instances that are not duplicates and could contain diferent contexts. The final number of instances per user would be / + (/) * , where is the total of messages of the user, the size of the package and the number of times we repeat the process of making "random" packages.
Our main reason for making augmentation data is to give our model instances which contain messages that are from diferent moments in the life of the user and help to understand the patterns of their mood. Another benefit of this technique is that this solves the problem of working with small data, as the new datasets are more extensive and improve the model’s performance. The same values used in Subsection 4.1 were used to make the augmentation datasets for training; the number of instances for these datasets triplicates the original number of packages shown in the Subsection 4.1; the validation datasets remained the same. Table 4 shows the distribution of the augmented datasets.
This Section contains two subsections. In the Subsection 5.1, we talk about the results obtained in the validation datasets created. These results helped us to choose the best hyperparameters for the models of each task. The Subsection 5.2 shows the results obtained from our proposed models on the competition.
The F1-score binary metric for the classification of the users was used to select the best models for the competition.
Since the size of the trial data is only ten examples, the train and trial data merged into one
and split into validation data with 20% of the data. For the experiments, we used the uncased
version of RoBERTuito [
Packing size for validation datasets
3 packages 13228
16 5 packages 128 32 32 10 packages 32 16 present the best models trained with our approach, and ranking respect F1-score metric. Later, we present the best models trained with our approach adding the augmentation technique and raking respect F1-score on validation dataset.
In Table 5, all the models are RoBERTuito fine tuned in the training datasets mentioned in Subsection 4.1. The first row present a model which is the only one that was trained with learning rate 1− 5, the others model are trained with learning rate 5− 5, and the hyper-parameter that changes is the batch size for training.
In Table 6, the results of the augmentation data technique are shown. The models’ performance with this augmentation data is better than the models’ performance only trained with the package strategy.
The models chosen were, • RoBERTuito-T5-B16: with learning rate 5− 5 and batch size 16 trained with packages of 5. • RoBERTuito-T5-B16-DA: with learning rate 1− 5 and batch size 16, this model was trained with augmented data and in packages of 5.
Because this were the models that had better F1-score in the majority of their validation datasets, we decide to use packages of 3 for the test dataset in the rounds, as the performance increase with the use of more information.
For the Task 1b the logits of the output prediction from the models were used as the prediction for the subtask. 5.1.2. Task 2 The results of the depression detection task are shown in Tables 7 and 8, as in Task 1 the F1-score binary was intended to be maximize.
In the case of depression, most of the learning rate used was 1− 6, contrary to the eating disorder task. This learning rate could be because the model needed more steps to learn lr
5− 5 3 packages 1− 5 1− 5 1− 5 5 packages 5− 5 5− 5 1− 5 10 packages 1− 5 5− 5 1 1 1 • RoBERTuito-T5-B128-DA: with learning rate 1− 6 and batch size 128, this model was trained with augmented data and in packages of 5. • RoBERTuito-T5-B32: with learning rate 1− 6 and batch size 32 trained with packages of 5.
The reason for choosing the previous models is because the ERDE5 and other metrics that measure early detection, as the other models, have better performance with packages of information with more messages, and model the model performs better with packages of 2. Consequently, lr
1− 6 3 packages 1− 6 1− 6 1− 5 5 packages 1− 6 1− 6 1− 6 10 packages 1− 6 1− 6 we decide to use packages of 3 for the test dataset in the rounds, as the performance increases with more information.
For Task 2b, the logits of the output prediction from the models were used as the prediction for the subtask. 5.1.3. Task 3 Because Task 3 uses the subsystem of Task 1 and Task 2, the run A is an ensemble of two models: • RoBERTuito-T3-B16: with learning rate 5− 5 and batch size 16, this model was trained with augmented data from eating disorders and in packages of 3. • RoBERTuito-T5-B32 with learning rate 1− 6 and batch size 32, this model was trained with data from depression in packages of 5.
In this subsection, we show our team’s results in the competition in the diferent subtasks. The metrics we use for comparison are F1-macro and ERDE30 for binary subtasks and RMSE, p@a, p@10, and p@20 for regression subtasks. 5.2.1. Task 1 In Task 1 we focused to increment the Macro-F1 for the eating disorders detection.
The results from model RoBERTuito-T5-B16 are reported; the model RoBERTuito-T5-B16-DA did not make it top. Table 9 compares with other teams from the oficial overview [ 23]. Team CIMAT-NLP-GTO and UMUTeam are included because they have the best performance at Macro-F1 score. Three BaseLine models from the committee organization are included for comparison.
For Task 1b, we provided the probability of the user sufering an eating disorder, and then the Root Mean Squared Error (RMSE) was calculated respecting the truth golden. The rest of the metrics are the precision of the prediction at 5, 10 and 20 messages, respectively.
RoBERTuito-T5-B16 1st place 8th place 2nd place BaseLine-Deberta BaseLine-Roberta Large BaseLine-Roberta Base
Task 1a
Task 1b
Even though our model was not the best in the Macro-F1 metric, it has competitive metrics for Task 1b, surpassing the models proposed as Baseline in almost all the metrics as they obtained ifrst place on precision metrics. In the RMSE our model placed fifth, lower RMSE means better performance. Considering the hard baselines our team placed second and fourth in Task 1b. 5.2.2. Task 2 In the depression detection task, we focused too on maximizing the Macro-F1.
The results from model RoBERTuito-T5-B128-DA were in the top 20 of the results for Task 2, but the model RoBERTuito-T5-B32 was not at this top. Table 10 is shown the comparison with other teams from the oficial overview [ 23]. Team UMUTeam and UNSL are included because they have the best performance at the Macro-F1 score. Three BaseLine models from the committee organization are included for comparison. SINAI-SELA is added because the ERDE30 metric, CIMAT-NLP-GTO, was added by their performance in Task 2b.
As we can see, the Macro-F1 score maximum ranges are lower for eating disorders results. This is because depression detection usually is more dificult for a variety of themes that users with depression talk about.
In this case, the model using data augmentation was better that the model without it, we theorized that this technique is better with depression detection tasks, as they need more instances to classify correctly and we can note that our model had better performance in the ERDE30 than the best model in Macro-F1 or competitive results in this metrics in comparision to models with better Macro-F1. 5.2.3. Task 3 The results from model RoBERTuito-T3-B16 was in the top 15 of the results for Task 3. In Table 11 is shown the comparison with other teams from the oficial overview [ 23]. Team RoBERTuito-T5-B128-DA 1st place 2nd place 5th place 15th place BaseLine-Deberta BaseLine-Roberta Large BaseLine-Roberta Base Macro-F1 0.645 0.737 0.733 0.720 0.621 0.813 0.813 0.694
CIMAT-NLP-GTO, NLP-UNED, and UPM are included because they have the best performance at Macro-F1 score and RSME. Three BaseLine models from committee organization are included for comparison.
RoBERTuito-T3-B16 3rd place 11th place 1st place BaseLine-Deberta BaseLine-Roberta Large BaseLine-Roberta Base
Task 3a
Task 3b
The model proposed surpasses models with better Macro-F1 scores in task 3b, even though Macro-F1 is not the best; the model has competitive confidence in its predictions. Considering hard baselines, our team placed third and fifth in Task 3a, for the Task 3b, we placed on second place.
The data used to develop systems for the automatic detection of mental illness is vital to emphasize the anonymity of the users’ identities for whose text is recorded. Even though the identities are unknown, the labelling process is usually crowd-sourced; this process can not guarantee the correct labelling for the instances as it depends on the subjective judgment of the annotators. Furthermore, the data obtained did not consider the users’ permission; these users could feel as if their privacy is being violated because it is crucial to use the data only to develop the systems.
Because of the critical climate change that is afecting the world is vital to make eforts to reduce the impacts of carbon emissions. The MentalRiskEs workshop implemented the tracking of the carbon emissions made by the models of the competition. They provided the code carbon package implemented on Python to track carbon emissions. The principal problem of this package with our models is that the functions did not track the emissions correctly, as they count all the GPU in the cluster instead of the only GPU used for the run of the models, and therefore calculate an incorrect amount of energy and carbon emissions produced. Another problem is the incompatibility with the version of Python greater than Python 3.8.
The MentalRiskES competition has been a challenging task concerning the early detection of mental health issues based on sequences of social media texts in spanish. Our models have balanced performance along all the metrics; their best performance is in the eating disorder dataset, which could be because of the nature of this dataset, as they tend to write more about their food-related issues. The depression dataset is more dificult by the complex writing of their authors and the need for more information on an excellent classification. We noted that for eating disorders, the model trained in the augmentation data did not have better performance than the model trained in the non-augmentation data; this could be because the model could be overfitting and making more false predictions of "sufering" class.
However, the depression detection models presented another pattern, the model trained with augmentation data was better for the task; we think this is because the model needs more information for depression users as they tend to write about an extended range of topics (not necessarily talk about their illness) like in eating disorder users.
For both tasks of linear regression, the models presented the best performances on the metrics about precision, which means that in the early stages of classification, the probability that the model assigned to each user is correct.
In future work, we plan to explore the size of the packages for the training and test datasets for eating disorder detection; for example, in each round, we could try to mix diferent packages for the test as we obtain more information or make diferent voting schemes. For the depression task, the augmentation data worked better than the other model. We plan to use the augmentation data we proposed to mix diferent package sizes to make new instances or clustering techniques to ensure that the messages joined have the same topic.
This research was funded by Consejo Nacional de Humanidades Ciencia y Tecnología (CONAHCyT) master’s degree grant #1141296. The authors thank to CONACyT, CIMAT and Instituto Nacional de Astrofísica, Óptica y Electrónica (INAOE) for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies (Plataforma de aprendizaje profundo para tecnologías del lenguaje) and CIMAT Bajio Supercomputing Laboratory (#300832). Sanchez-Vega acknowledges CONACyT for its support through the Program “Investigadoras e Investigadores por México” by the project “Desarrollo de Inteligencia Artificial aplicada a la prevención de violencia y salud mental.” (ID.11989, No. 1311). abs/2107.02027 (2021). URL: https://arxiv.org/abs/2107.02027. arXiv:2107.02027. [22] J. M. Pérez, D. A. Furman, L. A. Alemany, F. Luque, Robertuito: a pre-trained language model for social media text in spanish, CoRR abs/2111.09453 (2021). URL: https://arxiv. org/abs/2111.09453. arXiv:2111.09453. [23] A. M. Mármol-Romero, A. Moreno-Muñoz, F. M. Plaza-del-Arco, M. D. Molina-González, M. T. Martín-Valdivia, L. A. Ureña-López, A. Montejo-Ráez, Overview of MentalriskES at IberLEF 2023: Early Detection of Mental Disorders Risk in Spanish, Procesamiento del Lenguaje Natural 71 (2023).