Introduction

Mental health is an essential component of human well-being, yet its significance is often overlooked or underestimated. However, mental disorders affect millions of people worldwide, with consequences ranging from decreased quality of life to the risk of suicide. According to the World Health Organization (WHO), approximately one in eight people globally suffers from some form of mental disorder, and this figure could be increasing due to factors such as the COVID-19 pandemic [1].

The rise in anxiety, depression, and other mental disorders during the pandemic has underscored the urgent need to identify and address these issues early and effectively. Early detection of mental disorders can facilitate more effective interventions, improve treatment outcomes, and ultimately save lives.

In response to this need, the MentalRiskES competition [2,3] has become a focal point for research in early detection of mental disorder risks in the Spanish-speaking context. This competition, now in its 2024 edition, provides a platform for researchers worldwide to develop and evaluate methods for detecting mental disorders using social media data in Spanish.

In this edition of the competition, we have focused on Task 1: disorder detection, which involves multi-class classification to determine whether a user suffers from depression, anxiety, or shows no detected disorder. This challenge opens a path for identifying language patterns associated with these disorders in online environments, which can provide early risk signals and intervention opportunities.

This paper describes our approach to address Task 1 of the competition, including our methodology, results, and discussion of key findings. Through this work, we hope to contribute to the development of more effective tools and techniques for early detection of mental disorders in the Spanish-speaking community.

Materials and methods

In order to address Task 1 of the MentalRiskES competition, data collected from a collection of labeled users along with the messages they had posted over time were utilized [4]. The dataset for training consisted of a total of 485 users, with the following label distributions: 223 users with no detected disorders, 169 users with depression, and 93 users with anxiety, as shown in Figure 1 Originally, the class is assigned per user, however, there is no trace of which message or messages motivated the user label. Needless to say, a user classified as Depression or Anxiety might, very well, address in several messages similar topics as other users classified as None. Furthermore, not all messages within positive users show, necessarily, traces of depression or anxiety. However, the aim is to turn to message classifiers in an attempt to detect as early as possible positive users without the need of having processed all the messages. In other words, the approach would require to get messages annotated while we count on user-level classes. With this rationale, we did not find it consistent to label all messages with the corresponding user-label. For all this, we opted for a message-level heuristic re-labeling approach in an attempt to gain consistency at message-level labeling to leverage supervised classification approaches. Our re-labeling approach focused on the cosine similarity of the embeddings vectors of the message. That is, with the similarity of messages 𝑚 𝐴 and 𝑚 𝐵 computed as in (1) with A and B being the embedding vectors of messages 𝑚 𝐴 and 𝑚 𝐵 respectively.

𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(A, B) = A • B ‖A‖‖B‖(1)

This allowed to identify messages that did not show signs of mental health issues and, accordingly, getting them labeled with the "none" class. Increased consistency in supervised train has an impact in supervised inference approaches. Relabeling impact is shown by means of Figure 2.

Regarding the supervised learning approach, we involve language models to cope with textual information and a simple neural network to classify the information. In fact, two different language models were evaluated: SBERT (paraphrase-multilingual-mpnet-base-v2) [5] and BETO: Spanish BERT [6]. In an attempt to process the messages, a mean pooling layer was used to obtain a vector representation of each message. Finally, a neural network [7] model to assign a specific class to a user, we opted for a accumulated message class-threshold above which the user class was decided.

Results

Three different models were implemented to address the task. Each model was configured differently, using different language models and thresholds to make the decision on user classification. Table 1 summarizes the configurations for each run. The first model uses SBERT as the language model and a threshold of 4 for classification. The second model uses BETO as the language model and a threshold of 5. The third model uses SBERT as the language model and a threshold of 3. To evaluate the quality of model predictions, a variety of metrics have been used. These metrics include macro-precision (Macro_P), macro-recall (Macro_R), macro F1-score (Macro_F1) and accuracy. Accuracy, as defined in Equation ( 2), measures the fraction of correctly predicted instances out of the total number of instances. Macro-precision, calculated according to Equation (3), represents the average precision across all classes, while macro-recall, as defined in Equation ( 4), computes the average recall across all classes. Finally, the macro F1-score, as described by Equation ( 5), provides a balanced measure of the model's performance by taking into account both precision and recall.

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =

𝑇 𝑃 + 𝑇 𝑁 𝑇 𝑃 + 𝑇 𝑁 + 𝐹 𝑃 + 𝐹 𝑁

(2)

𝑀 𝑎𝑐𝑟𝑜_𝑃 = 1 𝑛 𝑛 ∑︁ 𝑖=1 𝑇 𝑃 𝑖 𝑇 𝑃 𝑖 + 𝐹 𝑃 𝑖 (3) 𝑀 𝑎𝑐𝑟𝑜_𝑅 = 1 𝑛 𝑛 ∑︁ 𝑖=1 𝑇 𝑃 𝑖 𝑇 𝑃 𝑖 + 𝐹 𝑁 𝑖 (4) 𝑀 𝑎𝑐𝑟𝑜_𝐹 1 = 2 × 𝑀 𝑎𝑐𝑟𝑜_𝑃 × 𝑀 𝑎𝑐𝑟𝑜_𝑅 𝑀 𝑎𝑐𝑟𝑜_𝑃 + 𝑀 𝑎𝑐𝑟𝑜_𝑅(5)

Our team, denoted as Ixa-Med, ranked seventh among the presented models, as can be seen in Table 2. Despite strong competition, our model achieved an accuracy score of 0.749, demonstrating its competitive performance. Additionally, our model achieved competitive results in several metrics, including a macro-precision of 0.796 and a macro-recall of 0.747. Notably, our team was one of only four teams to surpass a macro F1 score of 0.7, highlighting the difficulty of the task and the effectiveness of our approach in addressing it. However, it is worth mentioning that our score still fell below the baseline proposed by the organization. In Table 3, a comparison of the average emissions produced by the top four teams in the competition is presented. The Ixa-Med team positioned notably well in terms of emissions, achieving an average of 8.20E-05, surpassing most of the participating teams. Specifically, Ixa-Med generated significantly fewer emissions than ELiRF-UPV and UNED-GELP, which presented average emissions of 4.79E-04 and 1.84E-04, respectively. Although UnibucAI had the lowest average emissions, the performance of Ixa-Med is very close and competitive, reflecting an effective balance between efficiency and sustainability.

In summary, although the results obtained are encouraging, there is room for improvement and refinement of our approach in future iterations to remain competitive in the field of online mental disorder detection in the Spanish-speaking context.

Conclusions

In this MentalRiskES challenge, various approaches have been presented to address the early detection of mental disorders in Spanish using social media data. The results obtained reflect the growing interest and importance of this research field in identifying and preventing online mental health issues.

Our team, Ixa-Med, has demonstrated its ability to develop a robust and competitive model in the task of mental disorder detection. It is important to note that while we have achieved promising results, there is still room for improvement in our approach. Remaining challenges include optimizing the accuracy and efficiency of the model, as well as exploring new techniques and approaches to address mental disorder detection in a diverse and dynamic online environment.

Besides we found the MentalRiskES competition an invaluable opportunity to advance research in online mental disorder detection in the Spanish-speaking context.