This paper describes our participation in the MentalRiskES task at IberLEF 2023. The task involved predicting the likelihood of an individual experiencing depression based on their social media activity. The dataset consisted of conversations from 175 Telegram users, each labeled according to their evidence of sufering from the disorder. We used a combination of traditional machine learning and deep learning techniques to solve four predictive subtasks: binary classification, simple regression, multiclass classification, and multi-output regression. We approached this by training a model to solve the multi-output regression case and then transforming the predictions to work for the other three subtasks. We compare the performance of two modeling approaches: fine-tuning a BERT-based model directly for the task or using its embeddings as inputs to a linear regressor, with the latter yielding better results. The code to reproduce our results can be found at: https://github.com/simonsanvil/EarlyDepression-MentalRiskES Mental health is a growing concern in our society. According to the World Health Organization (WHO), 1 in 4 people will be afected by mental disorders at some point in their lives [ 1]. In addition, the COVID-19 pandemic has had a negative impact on the mental health of the general population, with an increase in the number of people sufering from mental disorders [ 2]. Thus, it is becoming increasingly important to evaluate the use of new technologies to assess the risk of mental illness and the healthcare needs of the population [3]. At the same time, social media platforms such as Telegram have become a popular way for people to express their feeling and emotions. Telegram is a free, end-to-end encrypted messaging service that allows users to send and receive messages and media files in private chats or groups that can be focused on particular topics and allow any user to observe or actively participate. These characteristics make Telegram a suitable source for text-mining [4].
With this context, an interesting approach is to use Natural Language Processing (NLP)
techniques to analyze the language used by people who sufer from mental illness and discover
patterns that can be used to identify them and provide the necessary support. The MentalRiskES
task at IberLEF 2023 [
In this work, we present our proposed solution to Task 2 of the 2023 edition of MentalRiskES.
This task involves evaluating the likelihood of a Telegram user experiencing depression based
on their comments within mental-health-focused groups. The task is split into four predictive
subtasks (2a, 2b, 2c, 2d) according to the type of output required. Our main contributions and
ifndings can be then summarized threefold:
1. We conducted experiments using various language models based on BERT [
The rest of the paper is organized as follows: In the next section, we analyze the dataset used for the task (Section 2). Then, we describe in detail our methodology for training and evaluating the models (Section 3). Finally, we discuss the results obtained (Section 4) and present our conclusions and future lines of work (Section 5).
The dataset given for the task consisted of a total of 6,248 individual messages from 175 Telegram users, each with a variable number of messages (see figure 1). The annotation process consisted of labeling each user based on the evidence from their conversation history of sufering from depression. Thus, a total of 10 annotators were used for the tasks. Each was asked to assign one of the following four labels to each user: • sufer+in favour : Indicates evidence (from text messages) of the user sufering from depression but is also receptive/willing to help and overcome it. • sufer+against : Indicates evidence of the user sufering from depression but is against receiving or providing help to overcome it. • sufer+other : Indicates evidence of the user sufering from depression, but there’s not enough information to assign them to any further category (against or in favour) • control: Indicates no evidence of the user sufering from depression.
Furthermore, these labels were represented diferently to support each of the four subtasks of MentalRiskES: simple classification ( task 2a), binary regression (task 2b), multiclass classification (task 2c), and multi-output regression (task 2d).
In the classification tasks (2a, 2c), the label assigned to each user was the class that obtained
the majority vote from the annotators, with the labels being "1" for the "sufer" classes and "0"
for the control in the case of task 2a. For the regression tasks (2b, 2d), the values of the labels
were presented as numeric probabilities in [
The following figure displays the label distribution for each task in the training set. We can see how over 94 (∼ 54%) of users were classified as having depression. Furthermore, there is an imbalance in the labels for the classification tasks due to the "sufer" label being divided into diferent categories (leading to an over-representation of the "control" label). Additionally, the "sufer+other" category is underrepresented when compared to the other three.
We proceeded to evaluate diferent techniques to solve each of the four subtasks. Two main predictive-modeling approaches were explored: The first one involved fine-tuning a pre-trained language model on each subtask and the second was about training a standard ML regressor using sentence embeddings encoded from the user’s messages as features. The following section describes the steps taken for each approach, first describing how the data was pre-processed and later explaining the training and evaluation process done for each subtask.
Independent of the approach taken to train the models, the data was pre-processed and augmented in the same way. The first thing we did was group all the messages by the user they belonged to and concatenated them into a single string, obtaining a total of 175 messages (one per user). This was done to obtain a single representation of each user’s conversation history (from which the labels were assigned) to be able to use it as input for the models.
To prepare for training, the data was split into training and validation sets, leaving a random
26 (15%) users in the latter for stratified cross-validation, where each set receives the same
proportion of samples of each class [
To increase the amount of data available for training and, at the same time, attempt to model early detection (obtaining predictions early on in the lifetime of the message history), we augmented the training set by adding observations that only contained half of their messages. This was done by first sorting the messages of each user in the training set by its date and then only taking the first half, the resulting dataset was then appended to the original training set to obtain a new one with twice the number of observations to be used for training.
By the discussion in section 2, it should be clear to see that not all labels of the subtasks give the same amount of information about the condition of the subject and the likelihood of predicting it based on the available data. Indeed, it’s clear that the probability values of task 2b give more information about confidence in predicting depression than the simple binary labels of task 2a. For the same reasons, the labels of task 2d are more informative than those of task 2c as they give the full probability distribution across the four classes.
Furthermore, we can show that it’s possible to use the multi-output regression labels (2d) to recover the labels of the other three subtasks. To illustrate, the multiclass classification labels of task 2c can be recovered by selecting the class in the distribution that has the highest probability. Moreover, we can obtain the labels of task 2a by simply converting these classes into binary (1 for the "sufer" classes and 0 for all others). Lastly, the labels of task 2b can be obtained by summing the probabilities of the "sufer" classes in the distribution. We have confirmed this by applying these modifications to the labels of the training set for task 2d and comparing them to the original labels of the other three tasks.
This observation led us to consider using models that solve for more than one subtask by
only training it with the labels of task 2b or 2d. This allowed us to reduce the number of models
that had to be trained and focus on solving for a single data modality (regression on [
We approached simple regression in a standard way training models, training models to
minimize the Mean Squared Error between the output values and the real ones. Additionally,
we included the post-processing step of clipping the output predictions of models of this type
to the [
Multi-output regression using standard machine learning regression, on the other hand, wasn’t as trivial as in the simple regression case. The models we worked on didn’t support multi-output regression out of the box. The approach we did involve training four regressors for each model, one for each class, and then combining the predictions. We explored two methods for this: training independent regressors or training them in a chain as explained by figure 3. The full details of the process are described in appendix D.
Finally, similar to the simple regression case, the predictions of the multi-output models were post-processed by dividing each of the four values by their sum to obtain a vector whose ^ values add up to one. That is, ^ = ∑︀ ^ for each -th class. This was done to ensure that the predictions were valid probability distributions over the classes.
A sentence embedding is a semantically meaningful real-valued vector representation of a
sentence, obtained from the outputs of the hidden layers of a language model. The properties of
this representation are so that sentences that express similar meanings are mapped (encoded)
closer to each other in the vector space [
In this way, the process of encoding text as numeric vectors can be used directly to extract features for a classifier or regressor, which will try to learn from the semantic information of these encodings to predict the label of their corresponding messages. Note, however, that this approach requires the need to have a pre-trained model to perform this encoding. Furthermore, it assumes that the model will be good enough at capturing the semantic information of the texts given as input, enough for the classifier/regressor to learn from it.
Assuming that this is the case, this approach has the advantage that it is much faster to train these kinds of regressors with regular CPUs, with the most time-consuming part being obtaining the embeddings of the training/evaluation messages, which only has to be done once. However, it is necessary to evaluate diferent encoding models and diferent classifiers/regressors (prediction models) to find the best combination for the task at hand.
As such, we conducted experiments using diferent language models to find the best encoding
model. Particularly, we tested three diferent versions of BERT [
The process of training and evaluating these models proceeded then as follows: First, the training set was encoded using the language model and the resulting embeddings were used as features for a regressor. The regressor was then trained using the labels of task 2d (the most informative ones) and the resulting model was used to predict the labels of the validation set. The predictions were then evaluated with the root mean squared error (RMSE). This process was repeated for each combination of language model and regressor.
Appendix B contains the results of this experiment. Based on that, roberta-suicide-es was deemed to be the best model for encoding the texts. Additionally, appendix C shows a detailed report of the evaluation of the best regression model with these embeddings.
Description
RoBERTa-base-bne [
BETO [
RoBERTa model [
RoBERTa-base-bne fine-tuned for suicide detection.
Variant of BERT [
Apart from the approach mentioned above, we also experimented with the pure Deep Learning
(DL) approach of taking a language model and fine-tuning it with the labels of the corresponding
subtask. The model we fine-tuned was a version of RoBERTa pre-trained for detecting suicidal
behavior from texts in Spanish [
The HuggingFace Transformers [
Hyperparameters
Optimizer Learning rate Max Tokens Num Epochs Batch Size
Value AdamW 1− 5 1024 30 1
We used an Adam Optimizer with Mean-Squared Error (MSE) for the simple regression models and a Cross-Entropy loss function for multi-regression (since the labels consisted of numeric probabilities). Furthermore, since the output for task 2d consisted of a probability distribution over the four classes, we experimented with a custom loss function that adds a term to the standard cross-entropy loss to penalize outputs whose sum is diferent from one. However, this did not improve the results empirically as compared with simply normalizing the outputs of the predictions after inference. The formula of this loss is shown in equation 1. Other hyperparameters are shown in table 2.
ℒcustom = ℒ− + (1 −
∑︁ ^)2
∈[
In the equation above, ^ is the output of the model, is the target label, is a hyperparameter that controls the weight of the penalty term, and is the -th element of the target label.
Using the approaches mentioned in the prior section, we came up with diferent models to solve the four subtasks of Task 2 of MentalRiskES. The results in this section are obtained from selecting the best-performing models after evaluating the diferent approaches and hyperparameters on the validation set. The final predictions were obtained from a test set of messages from 149 subjects never observed during the training process and evaluated against the task’s true labels.
In the tables below, we report the relevant metrics obtained for each subtask and compare them
against the ones obtained from baseline models provided by the organizers of the competition.
In particular, we report both absolute metrics, obtained after observing all the messages of each
subject, and early detection metrics, obtained after incrementally observing the messages across
several rounds. Additionally, table 11 displays the inference-time CO2 emissions and energy
consumption of each model, based on computing their absolute predictions on the test set. These
values were estimated using the codecarbon python library [
For the absolute metrics, we show the accuracy, precision, recall, and F1 scores for the
classification tasks (2a and 2c) and the root mean squared error (RMSE) and coeficient of
determination (2) for the regression tasks (2b and 2d). The early detection metrics include the
early-risk detection metric (erde) computed after observing diferent rounds of messages as well
as other metrics (more details are provided in the competition guidelines [
The metrics are shown along with the name of the model used to obtain them. The models are named as follows: [task name]_[model name]_[approach]. For example, task2b_robertasuicide-es_fine-tuning refers to the model trained with the task 2b (binary classification) labels by fine-tuning the Roberta model pre-trained for suicide detection. The " approach" can be either embeddings or fine-tuning for the two approaches described in section 3.
Furthermore, all ML regressors trained with embeddings as features were Ridge regressors, and all embeddings were obtained using roberta-suicide-es encodings as this combination yielded the best results in the evaluation set. The embeddings approaches for task 2d also include the multi-regression method used (ind indicating that independent regressors were used and chain for chained regressors).
BaseLine - Roberta Large
BaseLine - Deberta
BaseLine - Roberta Base
BaseLine - Roberta Base BaseLine - Deberta BaseLine - Roberta Large 2b_roberta-suicide-es_fine-tuning 2b_roberta_embeddings 2d_roberta_embeddings_ind 2d_roberta_embeddings_chain 2d_roberta-suicide-es_fine-tuning 3.000 3.000 2.000 3.000 3.000 2.000 4.000 4.000 RMSE
r2
BaseLine - Deberta
BaseLine - Roberta Large
BaseLine - Roberta Base 2d_roberta-suicide-es_fine-tuning
5. Conclusions duration (secs) emissions (kgCO2eq) cpu_energy ram_energy The results show that the approaches considered in this work were successful at modeling each of the predictive subtasks, with at least one of our models outperforming the baselines in most cases. We can make the following observations: • The best-performing approach across all tasks seems to be the one that uses the embeddings of the messages as input to a multi-output regression model (task 2d). At least one model trained with this approach reached the top ranking for tasks 2a, 2b, and 2d absolute ranking metrics and outperformed the baseline absolute metrics across all tasks. • Most notably, the regression method that uses multi-output chained regressors obtained the best metrics for task 2d across all models, outperforming the fine-tuning approach by over 20% in the absolute metrics and reaching the second highest spot in the early-risk metrics for this task. • Models trained for multi-output regression perform very well for binary classification and simple regression tasks, even outperforming the models trained for simple regression targets in their own subtask. This suggests that using one model to solve for multiple targets was indeed a good approach to this problem. • The models obtained with a pure DL approach from fine-tuning a RoBERTa model are estimated to produce over 3-4x less emissions at inference time than the hybrid approach from training linear regressors on sentence embeddings. This gap is likely because the fine-tuning approach requires less computation at inference time than the hybrid approach, which requires the computation of the sentence embeddings before feeding them to multiple regressors, while the fine-tuning approach is made in one forward pass.
Another finding we can conclude from these insights is that while our models achieve great results in the absolute ranking metrics, they do not perform as well for the metrics that assess early-risk performance. In our work, we did not model explicitly for an early detection scenario; we only added information about prior messages through data augmentation. This limitation means our models may not perform as well in real-world situations where we aim to detect signs of depression in a conversation early on.
Thus, it may be important to explore diferent training approaches to improve the
performance of early-risk detection. This might include directly employing online learning to predict
and update the model as new messages come in or incorporating an ensemble of models to
make independent decisions about a message’s risk level and combining them for a final
decision (as seen in [
The data was given in JSON format after requesting the server. The following examples are meant to show the structure of how the data was given and later parsed.
"id_message": "1", "message": "Me parece que es una buena idea, pero no estoy seguro", "date": "2020-07-27 01:27:31" "id_message": 2, "message": "Buen dia a todos", "date": "2020-07-27 02:03:28" Example of the raw data describing the messages of one user. The original training set (later split into training+validation) constituted 175 JSON files like this.
To complement the files described in the example above, the labels of each subject were given in CSV format, where each row corresponded to one subject. Four of these files were given, one for each task. Table 12 below shows some examples of this.
a_label b_label c_label
d_sufer_in_favour d_sufer_against d_sufer_other d_control subject_id subject101 1 subject104 1 subject106 1 subject108 1 subject109 0
The estimators mentioned in the table above are implementations of common regressors
from Python’s Scikit-Learn library [
For task 2d, we were required to obtain four values corresponding to a probability distribution over the four classes (sufer+in favour, sufer+against, sufer+other, control). In section 3.2, we explained how this multi-output regression problem can be solved for the sentence embeddings approach by training four regressors and then combining their predictions using either the Independent Regressors or Regressor Chain methods.
Here we explain how these methods work and were implemented. First of all, the two methods can be summarized as follows, depending on how the regressors are trained to obtain the probability distributions. Figure 3 shows a graphical representation of the two methods.
To the second method, we can additionally add the option of applying Principal Component
Analysis (PCA) to reduce the dimensionality of the input embeddings before training the models
in the chain. Because the embeddings might have a large dimensionality, this is done to make
these models more likely to use the information of the previous predictions. Both methods
were implemented with the Scikit-Learn library [