This paper presents the approaches presented on the eRisk 2025 initiative, aimed at the early detection of mental health risks through the analysis of users' social media. On the one hand, we explore the first task, in which a user ranking has to be obtained to seize relevance of depression symptoms in user's writings. The symptoms are based on the second revision of the Beck Depression Inventory questionnaire, BDI-II. We deal with a transformer-based approach for ranking using a multilingual BERT model with multiple heads, and a two novel data reduction techniques to optimize training and inference time. Our results show the efectiveness of these selection strategies, as reduced inference time was obtained while achieving higher performance. On the other hand, we explore the Pilot Task. This consists of developing conversational agents that can interact with Large Language Model (LLM) personas and detect whether the latter have depression, also based on depression symptoms. We employed various LLMs for this task, including GPT and Falcon, evaluating their conversational assessments of depression. Our results indicate that LLMs, even without fine-tuning, can perform comparably to manual approaches in symptom identification and severity estimation, with GPT-based systems showing the most promise in balancing brevity and informativeness.
Even though the awareness for mental health issues has increased in the last years, there is still stigma related to mental illnesses and their treatment [1]. This is a concern, as symptom management of these issues is influenced by the mental health literacy, that is, the knowledge and acceptance around mental health of the person with the sickness or those around them. Taking into account that almost all people will come in contact with someone impaired by a mental illness [2], it is crucial to improve public awareness and knowledge regarding this topic. In general, it is clear that some work needs to be done in order to better integrate patient needs in this particular healthcare field.
The widespread use of social media presents a unique opportunity in this regard. Social platforms generate large volumes of user-generated content that can be analysed to identify early signals of mental health deterioration, including depression and suicidal ideation [3]. Moreover, the growing volume of online research studies, even if it supports scientific advancement and promotes information sharing, also presents challenges for mental health professionals who must navigate and synthesize an overwhelming amount of data, which can lead to stress and burnout and even reduce decision-making abilities [4]. Therefore, methods based on informatics, and, nowadays, Artificial Intelligence (AI) have arisen as an opportunity to reduce the administrative burden on these workers, and to act as a support to access information, among others [5].
In this context, the eRisk shared task, organized within the CLEF (Conference and Labs of the Evaluation Forum) workshop, provides a standardized benchmark for the early risk detection of mental health issues using social media data [6, 7]. The eRisk shared task is one of the initiatives in which current advances in informatics can be applied to the mental healthcare field. Our team participated in two of the subtasks: • Task 1: Early Risk Detection of Depression. This task involves ranking user-written sentences according to their relevance to depression symptoms according to the BDI-II questionnaire. This ranking is evaluated by comparing it to human annotation, which has two possibilities: label assignment by majority and by unanimity. • Pilot Task: Conversational Depression Detection via LLMs. This task, new for this edition, introduces a challenge of interacting with Large Language Model (LLM) personas. The challenge lies in determining the signs and symptoms of depression of the persona, and determining their possible score in the BDI-II questionnaire.
The main NLP-related challenges of the eRisk tasks include: 1. Detection of implicit expressions of mental states, and, particularly, depression symptoms. 2. Dealing with imbalanced and noisy data, including instances in languages other than English. 3. In Task 1 specifically, consensus versus majority label disagreements add another layer of complexity, where reliability of annotations varies across examples.
Our group, ixa_ave, explored methods informed by prior work in the eRisk framework, tailoring them to the current task and challenges with our own ideas. The following section reviews the current landscape of NLP in mental health detection and in the eRisk tasks, and lays the foundation for our methodology.
Historically, the way to assess a person’s mental state has been through a thorough psychological analysis performed by mental healthcare experts. However, as the number of people needing access to this kind of service increases, rule-based (RB) methods based on questionnaire assessment arose to be able to assess patients individually in a quicker manner. These methods normally consist of sets of questions that infer the most relevant aspects for a certain mental health issue, including the detection of suicidal intent [8]. This method, although more eficient, is not without its limitations, mostly regarding the potential of patients to lie more easily and a reliance on predefined questions that may not capture the complexity of a patient’s experiences [9]. However, as large amounts of textual data from social media have become available for training models, the use of Natural Language Processing (NLP) has been proposed as an alternative to determine a person’s mental state.
For instance, simple Machine Learning (ML) techniques such as logistic regression, Support Vector Machines (SVMs), and decision trees have been used successfully in various tasks of detecting mental health conditions. These methods usually outperform rule-based approaches by capturing more complex patterns in the data, such as non-explicit expressions of suicidal ideation, or changes in tone [10].
On the other hand, more complex machine learning approaches, like neural networks, provide a tailorability that can be efectively used for mental health applications. Due to the ability of models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to capture both short-term and long-term dependencies in text, deep learning models have significantly contributed to NLP [11, 12]. The use of dense vectors to represent sentences also give these models the ability to better capture semantic similarities [13]. All of these benefits make the deep learning approach more adaptable and efective in NLP than traditional ML approaches. Furthermore, word and sentence similarity metrics have become increasingly valuable in assessing mental health issues by analysing the semantic content of responses [14].
However, these type of architectures struggle with long-term dependencies. Arising to this need, pre-trained transformer models such as the Bidirectional Encoder Representations from Transformers (BERT) have transformed the field [ 15]. Transformers excel in transfer learning, where pre-trained models can be fine-tuned for specific tasks. These pre-trained models already have semantic knowledge embedded in them, and thus fine-tuning becomes much less computationally expensive and timeconsuming, while allowing the model to rapidly specialize to domain-specific tasks with minimal data, and delivering strong performance gains. This approach significantly reduces the need for large labelled datasets, which are often expensive and time-consuming to obtain. For instance, for the specific processing of medical texts, some specific transformers have been trained, like MedicalBERT or BEHRT [16]. These can be used in the specific case of mental health with good results [17].
Some more recent examples of applications of NLP have been Large Language Models, or LLMs. These have been successfully applied to the healthcare department, for instance, for diferential diagnosis [18] and medical text summarization [19], and also in more mental health related cases, like for suicide assessment [20]. These recent studies, along with the boom of LLMs, demonstrate that there is a great potential in LLMs for this kind of health applications.
Over the past editions of the eRisk shared task, several research groups have tackled the first task, exploring a wide range of strategies. Traditional machine learning pipelines initially dominated the ifeld, with participants performing feature engineering to retrieve the most significant lexical cues [ 21]. With the rise of deep learning, more recent approaches have adopted models like RNNs to capture temporal and semantic dynamics across user timelines [22]. However, most teams use transformerbased architectures, such as BERT and its derivatives, with excellent results [23, 24]. Additionally, methods like curriculum learning, weak supervision, and meta-learning have been proposed to handle the sparse and imbalanced nature of user-level labels. More recent approaches have also focused on retrieving or synthesizing data with LLMs to improve depression detection. For example, [25] proposed using ChatGPT-generated data to retrieve depression symptoms from social media, thus improving the performance. However, most prior approaches do not directly address the semantic alignment between training examples and clinical symptom definitions.
In order to take this semantic alignment into account, our idea focuses on a similarity-based approach, where the semantic similarity between the assessed sentences and the BDI-II questionnaire items was taken into account. Prior work has approached this problem through filtering systems to reduce the search space [26], as well as similarity mechanisms calculated with respect to the questionnaires [27]. While these methods have laid important groundwork, they often rely on rigid similarity metrics or static thresholds that may not adapt well to the subtle and context-dependent nature of social media posts. Furthermore, they lack a fine-grained semantic alignment with established clinical frameworks, such as the BDI-II questionnaire. To address these limitations, our work introduces two data reduction strategies based on semantic similarity to the BDI-II. This approach improves the relevance of selected content, optimizes training and inference time, and enhances the interpretability of the classification process, distinguishing our methodology from prior eforts in the eRisk series and contributing a new, clinically informed dimension to risk detection.
In the Pilot Task, as it was its inaugural year, we faced the challenge of charting new territory, as no directly comparable prior work existed. Nevertheless, we were inspired by existing applications of LLMs in diagnostic contexts [18] and their demonstrated capacity to assess personality traits [28]. Therefore, we hypothesized that leveraging LLMs as evaluators themselves could give valuable insights. Our goal was to comparatively investigate how both open-source and proprietary LLMs would perform relative to humans, especially in the absence of any specialized mental health training. To that end, our methodology was intentionally minimalistic: the developed models were provided only with the BDI-II questionnaire and stylistic prompting before interacting with the simulated personas they were tasked with assessing.
For the Pilot Task, no training data was made available. By contrast, for Task 1, a TREC-formatted sentence-tagged dataset was provided. The dataset consisted of user-generated Reddit sentences, with each sentence associated with one or more depression-related symptoms from the BDI-II questionnaire (see the first column in Table 1) as annotated by experts. Each user-generated sentence included metadata following an XML-like structure: • <DOCNO>: A unique identifier for the sentence, used for referencing. • <PRE>: The sentence that comes immediately before the target sentence in the user’s post or timeline, in order to provide context. • <TEXT>: The target sentence that is asked to evaluate. • <POST>: The sentence that comes immediately after the target sentence, ofering additional context.
Not all training data had the <PRE> and <POST> data, so we chose to not use them for simplicity’s sake. Apart from this, for training purposes, two comma separated value csv files were released. These ifles identified sentences from the training dataset (thanks to the document identifier, <DOCNO>) that were relevant to specific BDI-II symptoms, based on human annotation: • Majority vote. This csv included sentences that were labelled as relevant by the majority of annotators. This reflects a more inclusive judgment criterion and is useful for training models with greater coverage. • Consensus vote. This csv contained only those sentences that received unanimous agreement or full agreement among annotators about their relevance. These are considered high-confidence labels.
Not all the training data was annotated; in fact, more than 99% of the training dataset was unlabelled.
More particularly, a sentence could be annotated for a symptom but not the rest of the symptoms. The
dataset was also imbalanced, with more sentences being classified as irrelevant (0) than as relevant (
Furthermore, a main characteristic of our work has been the use of the BDI-II questionnaire [29] for both tasks. In the first task, it has been used to calculate the messages’ similarity with respect to the depression symptoms, and in the Pilot Task, it has been used as a prompting agent for the developed LLM personas. The left column in Table 1 shows the 21 symptoms related to depression according to this questionnaire.
For the depression symptom ranking task, we implemented several models based on the original multilingual BERT architecture, that is, the bert-base-multilingual-cased1 model [15]. All the models were fine-tuned on an Nvidia A100 GPU on two epochs with an AdamW optimizer and a learning rate of 2 − 5. The batch size was 32 and the maximum token length was 512. For reproducibility purposes, the code for this Task has been uploaded to https://github.com/anevarela/eRisk_ixa_ave.
The models tackled the task as a classification, and they had a 21-head output with a softmax activation function, enabling an inference for each sentence to all symptoms in the BDI-II questionnaire in a regression-like manner. Moreover, all of the models introduced the similarity of each sentence with respect to the 21 symptoms of the BDI-II questionnaire as training features. The output of the employed 1Available at https://huggingface.co/google-bert/bert-base-multilingual-cased multilingual BERT was concatenated with the 21 similarity values and fed to a final layer, that was then connected to the classifier heads.
The similarity of the sentences with respect to the 21 BDI-II questionnaire items was computed by means of Cosine Similarity. The model employed to transform the text into numerical representation was the Sentence Transformer mrm8488/distiluse-base-multilingual-cased-v2-finetuned-stsb_multi_mtes2. The similarity with respect to each symptom was calculated using the questionnaire’s items. Each symptom of the BDI-II has a severity ranging from 0 to 3 in a Likert scale point system, based on diferent statements. Take the following example from the BDI-II questionnaire: 14. Worthlessness 0. I do not feel I am worthless. 1. I don’t consider myself as worthwhile and useful as I used to. 2. I feel more worthless as compared to others. 3. I feel utterly worthless.
This shows that each symptom can have diferent degrees of severity, 0 (less severe) to 3 (more severe). To compute the similarity of one sentence of the dataset with respect to one of the 21 symptoms, we apply a weighted similarity across the symptom’s severity levels. The assigned weights, range from 1.0, for severity 0, to 2.5, for severity 3. This reflects the increasing diagnostic value and specificity of more severe statements, as the weights increased linearly with respect to severity with a step of 0.5.
While low-severity descriptions still indicate the user’s condition and are relevant, they often express
absence of the symptom and tend to be more general in terms of formulation. In contrast, high-severity
descriptions carry more distinct signals of symptom presence, making them more informative for
relevance estimation. Normalizing by the sum of weights ensures comparability across symptoms and
bounds the final score, as it can be seen in expression (
Regarding the use of the data, a 20% of the provided training set was used for the validation, randomly selecting users for this purpose. Moreover, oversampling was also introduced in the system to avoid classifying to the most prominent class, 0, as the data was highly imbalanced. It is also important to note that not all the training and validation instances were ultimately used, as only the labelled data were used for this (see Table 1). This is because, even if, ideally, there would be a pre-training stage to get the model used to the language of the desired task [30], it was skipped in this approach due to time and memory constraints. If the label of a certain symptom was not known, a − 1 mask was used to mark it. These masked sentences were not used to compute the loss at the pertinent symptoms.
Additionally, a weighting mechanism was incorporated into the loss computation to distinguish
between consensus and majority annotations. Specifically, predictions with consensus labels were
given twice the weight of those with only majority labels, emphasizing high-agreement samples during
training, as it can be seen in expression (
ℓweighted = (1 + (())) · ℓBCE(, ) The specifications above are common for all the models.
Although it may seem counterintuitive, instead of regular data augmentation approaches, we turned
to data reduction. Formally, our approach focuses on a subset of significant sentences with respect
to symptoms, both in the training and test stages. The training set data is reduced in an attempt to
keep sentences that are not really obvious, as in (
(, ) = ∑︀ (, )
∑︀ = = { = (, ) ∈ | ∃ : (, ) ≤ } { = (, ) ∈ | ∃ : (, ) ≥ }
As a result, our approach is characterized by two parameters (, ). Based on this, our team submitted ifve runs : 1. Base model. This model was trained in all the labelled training data using gradient descent. We submitted three runs based on this: a) base_all. The inference was conducted in all the provided test data. b) base_filter. This strategy included what we will call the test data selection strategy.
The approach consisted of ignoring all the instances from the test set that had a similarity,
as in (
A small summary of the models can be seen in Table 2. Notice that, for = 1, no sentence is going to have a higher similarity, and thus all the training data is taken into account; similarly, for = 0, no test data is eliminated, as the similarity will always be higher.
Run specifications: the models were trained and tested on diferent sub-set of the original training set, based on of similarity of the sentences with respect to the questionnaire. The training subset was selected based on a threshold , with sentences higher than it being excluded. The test dataset excluded sentences lower than a run base_all base_filter30 base_filter50 thresh_all thresh_filter50 1 1 1 0.5 0 0.3 0.5 0
Our work explores a combination of training data reduction (by excluding obvious training samples with ) and test data selection (by removing unrelated samples with ). This was validated in preliminary experiments and is consistent with the obtained results.
The Pilot Task for this year consisted of being able to develop a chatbot that would interact with twelve proposed personas. These personas would each have a certain depression score and symptoms associated to them, and the objective was to automatically infer these scores and symptoms.
We explored three primary approaches to conducting depression screening conversations. All developed approaches can be described as minimally trained base models (or personas) assigned the task of assessment based on the BDI-II questionnaire. That is, none of them were explicitly trained or ifne-tuned in a dataset related to mental health or depression. In contrast, the systems were prompted with the questionnaire and the task in hand, with no further information. This approach was mimicked in the manual approach, in which a human with no specific mental healthcare knowledge conducted the conversation, once given the BDI-II questionnaire.
Four diferent runs were presented. One run was manual, and the other three were based on LLMs. Two of the LLM approaches were based on the GPT-4 model [31], and the other one was based on an open-source model, Falcon [32]. This breakdown of all approaches explains each one with more details, and specifies the run related to them: • Manual approach (run 0). A human without specific clinical training conducts the conversation using the BDI-II questionnaire as a base. The dialogue is then analysed using a model previously trained for Task 1 to infer a BDI-II score and identify key symptoms, if any. The used model was the base_all. • GPT-based agents. Two versions of GPT personas were implemented: – Long GPT 3 (run 1). This version systematically goes through all BDI-II symptoms in detail and in order. The initial prompt to determine the system instructions emphasized that it was 3Available from https://chatgpt.com/g/g-67e55a078fd4819180b2a3d3651dd7e3-supportive-mood-checker a supportive persona in a therapist-like style, that needed to analyse depression symptoms as stated in the questionnaire in a conversational and emphatic manner, and then infer a score based on the used questionnaire. In the conversations, it was prompted to start with a general question and then, once the conversation was concluded, it gave a BDI-II score and top symptoms in the desired format. The BDI-II questionnaire was given as a reference document. The score for each symptom is not calculated; instead, the whole conversation is assessed and given an overall depression score. – Short GPT 4 (run 2). This version’s system instructions included a brevity aspect, asking so that it combined symptoms into fewer prompts to complete the assessment more quickly. This was based on the idea that an earlier detection would be beneficial, as it was stated in the task definition that a longer assessment would result on penalisation. Like its longer counterpart, this approach was prompted to start with a general question and ended with the overall inference of the persona’s mental state based on the BDI-II questionnaire. The BDI-II questionnaire was given as a reference document. • Open-source LLM (Falcon) (run 3). We employed the Falcon model, a lighter alternative to LLaMA [33], which, when given the task description, generates one synthetic question per BDI-II symptom, with a maximum length of 50 tokens. A second instance of the model then infers a BDI-II score and highlights key symptoms based on the responses. This approach also relied on the use of the BDI-II questionnaire scoring system and calculated the score for each symptom individually. The overall score was taken as a combination of all 21 symptoms, like it is done in the BDI-II questionnaire.
All the prompts for the used LLM models (both GPT and Falcon) can be found in Appendix A.
The models developed for this task, after inference of the test set, were evaluated using widely used information retrieval metrics [34]. Mean Average Precision (AP), Mean R-Precision, Mean Precision at 10 (P@10) and Mean NDCG at 1000 were used. These assess the efectiveness of the models in retrieving and ranking relevant items.
In the preliminary results obtained with the validation partition, applying test data selection appears viable for the Base model, but less so for the Threshold model, as this also included training data reduction. This distinction is intuitive: the Threshold model was never exposed to high-similarity cases during training, and thus lacked the capacity to generalize efectively to such instances at inference time.
Similarly, if this task were taken as a classification instead of a ranking based on the typical 0.5 limit, the validation metrics seem to confirm that the Base model performs better than the Threshold model. A F1-score of 0.85 is obtained with the Base model with no test data selection, while a 0.81 score is obtained with the Threshold model under the same test conditions. Nevertheless, employing similarity-based training data reduction remains promising, as it simplifies the learning process and reduces computational demands. In our case, 3.65% of the training labels were skipped due to this training data reduction, leading to a slightly more eficient approach. An eficient trade-of may be found, but this strategy exhibited limitations and our assumption that the high-similarity sentences may be “too obvious” may be incorrect.
The two base_filter runs show that applying test data selection can be beneficial. In fact, it can be confirmed during validation that the number of False Positives (FP) is reduced, at the cost of increasing the number of False Negatives (FN), when test data selection is applied. Figure 1 visualizes this degradation using a Sankey diagram, which illustrates the evolving confusion matrix as the test data selection increases. 4Available from https://chatgpt.com/g/g-67ee482cf2dc819195533b897fd63630-depression-screening-assistant
However, and as expected, the test data selection does contribute to the acceleration of the inference process. When ranking the test partition, for the 0.3 filter, more than 94% of instances were excluded because of low similarity with respect to the questionnaire, causing the inference time to drop to only a 5% of the time needed for all the instances. This seems to improve performance of the Base model by dropping a number of false positives, as it can be seen in Tables 3 and 4, which show our team’s performance in majority and unanimity voting, respectively. The diference is even greater with the 0.5 filter; although the performance is poor, the excluded sentences surpass 99%. This suggests a compromise can be found based on the trade-of between time and performance.
Our AP levels are not high, and they suggest that out of 10 retrieved sentences, only 1 or less is considered relevant across the full list. In contrast, our P@10 of 0.30 means 3 of the first 10 results are relevant. R-PREC evaluates precision at the rank equal to the number of true relevant items, and only achieves a maximum of 0.305 in our models. Finally, NDCG accounts for the position of relevant items in the ranked list, and only rises up to 0.345 for us.
Our best-performing model was base_filter30, achieving 0.102 AP in majority voting and 0.055 AP in unanimity. Comparing with the rest of the groups, our team’s performance was mediocre, particularly on the majority voting [6]. It is interesting to see that, comparing to the information loss some groups have from majority voting, our model seems to be more robust to this; our AP drop between majority and unanimity was only 0.047, whereas the average drop across top 5 teams was 0.1. Therefore, we have a strong conviction that our approach may be useful with a better fine-tuning, and particularly, that our approach to include weighted loss in the consensually labelled sentences is appropriate.
To value the performance of the Pilot Task, three main evaluation metrics were used, adapted from eRisk 2019 [35]: Depression Category Hit Rate (DCHR), Average DODL (ADODL), and Average Symptom Hit Rate (ASHR). The metrics shown in Table 5 are in accordance with the described models. The mean number of messages per run define the number of prompts generated by our models (or written by the user) to send to the personas.
As it can be seen in Table 5, run 0, related to the manual approach, and run 2, the short GPT model, are the ones that send the least number of messages, and while the short GPT model does pack a lot of information in the messages (both in the questions and the responses to the personas), the manual approach is poorer; the prompts are short, and the responses also seem to lack information based on the number of characters. On the other side, run 1 and 3, which are based in the long GPT approach and Falcon approach respectively, opt for a longer interview-style prompting, with approximately one question per symptom. This seems to work for the GPT model, as it develops richer questions and therefore obtains long answers, but the Falcon model, although more concise in its questioning, also obtains long answers from the personas.
The longest responses from the personas correspond to the short GPT model, maybe due to the information packing that comes from asking about several symptoms at the same time. This is also the model with the least number of prompts, making it the most appropriate for quick assessment.
The metrics obtained by our group in this shared task can be seen in Table 6. The asterisk (*) represents the manual run; it is important to mention that ours was the only manual run submitted.
Out of the four presented models, the results given by the evaluation metrics favour the GPT models. The manual and GPT-based models all score a DCHR metric of 0.33, showcasing limited capability of the models to efectively predict the depression category of the personas. A random guess in the DCHR metric has a 25% chance of hitting the correct category; our slightly higher metric showcases an increase in predictive capacity in this matter, with 4 out of 12 personas being classified correctly. However, the Falcon LLM model, showcasing a DCHR of 0.17, shows less capability than a random model for this severity classification.
The DCHR scores seem to be in accordance with the expected results, as the distribution of inferred BDI-II scores does not change significantly across models, except in the case of the Falcon model, which produces significantly lower results in general. This is shown in Figure 2. This is also corroborated by a statistical significance test, that showcases no diferences between Runs 0, 1 and 2 in terms of BDI-II score inference, and shows the 3rd run as a significantly lower score output.
In contrast, all the ADODL scores surpass 0.75, so the depression severity closeness level is predicted quite accurately, particularly in the case of the short GPT model, that rises the score up to a 0.83. Seen as a random model would get approximately a 0.67 score, we can confirm that we have efectively increased the predictive capability of the depressive score. However, our metrics highlight that, in mean, the best model will have a diference of 10.71 between the actual score and the predicted one.
On the other hand, the long GPT model shows the highest ASHR score (tied with other groups), so even if the performance is limited (0.29) the symptom identification capability of the model is the highest among the rest. The baseline for a random model is 0.19 in this case, so the Falcon model performs randomly in this case. This suggests that Falcon has an insuficient symptom coverage due to a lack of variety in prompts. In the rest of the models, on average, only 1 symptom is predicted correctly. Regarding the manual model (run 0), in which the base_all model developed in Task 1 was used for inference, we observed a poorer identification of the symptoms “Loss of interest in sex”, “Indecisiveness” and “Punishment feelings”, maybe due to confusion with other symptoms.
There are other observations to be taken into account, specially for the Falcon model (run 3). Falcon’s score inference appears influenced by the order of symptom presentation, consistently overemphasizing “sadness”. Furthermore, Falcon-generated conversations were repetitive, difering mainly in question order. Some prompts (e.g., regarding “past failures”) were incorrect, and all outputs were preceded by a token labelled "example" for unknown reasons.
It was also quite interesting to see that across all models, “loss of interest” and “pleasure” were among the most consistently detected symptoms.
As a side remark, our group was the only one to introduce a manual approach into the Pilot Task. The person conducting the manual conversation did not have formal training in psychological assessment and came from an engineering background. This was an intentional choice to explore how a non-expert might navigate the interaction. Contrary to initial expectations, the manual process was not as timeconsuming as anticipated, given that each step still required engaging with the GPT-based personas even with the Falcon and GPT-based approaches. However, the manual approach did require additional efort in monitoring the flow of the conversation while simultaneously introducing potentially relevant symptoms, a process which is not trivial with no clinical background. The main diference between this approach and the LLM-driven ones is that, in the others, an automatic symptom-checking strategy was used. This manual method was cognitively demanding and, at times, emotionally taxing, highlighting both the complexity and the emotional weight involved in simulating mental health assessments. Thus, beyond the accuracy of the evaluation, the use of LLMs provides advantages over face-to-face methods.
In general, the GPT models (runs 1 and 2) showed better performance than the manual and the Falcon approaches. However, the Manual approach DCHR score is the same as the GPT model ones, suggesting that a person with no more training than a mental health questionnaire can perform as well at identifying depression as a similarly prompted GPT model. It also is important to mention that, with no pretraining whatsoever and this being the first task regarding LLMs in the eRisk shared tasks, we obtained better results than expected.
In this work, we explored the two tasks made by our group, ixa_ave, for the eRisk shared task. Task 1 consisted of a ranking system based on depression symptoms for social media text, and the Pilot Task involved creating LLMs to assess the depressive score and the main symptoms of a series of GPT-based personas.
For Task 1, a dual data reduction strategy was developed to optimize training and inference eficiency and evaluate its impact on model performance. Our findings can be summarized in two points. Although the first reduction approach, based on instance removal for training, did not yield strong performance improvements, it demonstrated potential. We successfully reduced the dataset size, leading to faster training, and we believe a more similarity-aware strategy could further enhance results.
The second filtering method proved significantly more efective. It not only reduced inference time, but also showed performance improvements depending on the used filter. This suggests that selective data reduction can improve both eficiency and model quality when well-calibrated.
Overall, our results provide empirical support for similarity-based data reduction as a viable tool for accelerating training and inference, with minimal or even positive impact on output quality.
However, for future research, several directions could further strengthen and extend our findings. Firstly, experimentation with alternative threshold strategies for the training data reduction, and diferent filters for the test data selection, could be interesting, as diferent values for or could further optimize the balance between speed and model performance. Moreover, a pre-training stage could be introduced in the approach, so that the unlabelled instances could be used. This would allow the models to get used to the language, even if these instances are ultimately not used for fine-tuning. Furthermore, the exploration of other similarity techniques could be interesting to optimize the information retrieval from the questionnaire. Finally, other fine-tuning techniques such as triplet loss could be explored. This would enhance representation learning and improve classification performance, which could ultimately lead to better rankings.
For the Pilot Task, diferent prompting strategies were evaluated. Our results highlight several key insights. Firstly, the performance was modest across all runs, with the manual and GPT-based models achieving equal scores. This suggests that both human and LLM-based approaches can marginally outperform chance in depression assessment if they do not have specific knowledge of the subject apart from the used material; in our case, the BDI-II questionnaire. The Falcon model, in contrast, underperformed, suggesting limited capability in this task. The GPT models strikes the best balance across all metrics; however, concise, symptom-packed prompts can lead to more accurate estimations of BDI-II scores, while longer prompts lead to better symptom detection.
For future work, it would be interesting to use a specifically trained LLM model to tackle this mental health related task. This would give the models more domain-related knowledge that may yield better results. Furthermore, a symptom-wise inference could be explored. Evaluating each symptom individually may increase the needed time but could yield more accurate and interpretable results.
This work was partially funded by LOTU grant (TED2021-130398B-C22 funded by MICIU/AEI/ 10.13039/501100011033 and by the European Union NextGenerationEU/PRTR) and also by the Spanish Ministry of Science and Innovation (EDHIA PID2022-136522OB-C22); it has been also funded by the Basque Government (IXA IT1570-22).
During the preparation of this work, the authors used ChatGPT in order to: Check grammar and spelling, Paraphrase and Reword. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
You are a professional and compassionate medical expert conducting brief and focused assessments for depression using the Beck Depression Inventory (BDI-II) as a framework. Your approach is concise, eficient, and emotionally neutral, aiming to cover all 21 BDI-II symptoms through a streamlined series of short, natural-sounding questions. You avoid extended commentary or reflections, keeping responses minimal to maintain momentum while ensuring a respectful and caring tone.
You begin the conversation proactively with a brief, open-ended check-in, and then move quickly through questions that subtly touch on symptoms like sadness, pessimism, sleep, energy, appetite, and more. Your phrasing is casual yet intentional, prioritizing eficiency over depth while still being mindful of the user’s comfort. You do not ofer feedback or interpretation unless asked, and you never diagnose. If a user shares something concerning, you acknowledge it with minimal but empathetic validation and gently recommend speaking to a professional. Be empathetic but concise—respond in one to two phrases only. Move eficiently through all 21 BDI-II symptoms without lingering.
You are able to infer the user’s total BDI-II score (
You are a professional and compassionate medical expert conducting brief, focused assessments for depression using the Beck Depression Inventory (BDI-II) as a framework. You initiate each conversation with a gentle, open-ended emotional check-in, then guide the user through a streamlined set of fewer than 15 carefully phrased questions. Each question is crafted to be empathetic and natural while indirectly covering multiple BDI-II symptom areas. You group related symptoms—such as energy, sleep, and fatigue—into single prompts, and gently guide the user through a conversational flow that touches all relevant areas without being clinical or overwhelming. Your tone is warm, soft-spoken, and mindful, showing care without lingering or overexplaining. You keep your own responses brief to maintain momentum, typically using one or two phrases.
When a user shares something dificult, you respond with subtle, sincere empathy and, when
appropriate, suggest reaching out to a mental health professional. You avoid ofering
interpretation or advice unless asked directly. At the end, or upon request, you infer an approximate
BDI-II score (
Now please assess the user, who is called <persona name>, in the BDI-II scale, selecting up to the 4 most relevant symptoms. If the symptoms are not relevant, do not include them. The format should be JSON, like this example: { } "LLM": "Maya", "bdi-score": 27, "key-symptoms": ["Hopelessness", "Sleep Disturbance", "Concentration Problems", "Worthlessness"]
For reproducibility purposes, the code for the Falcon assessment of the obtained interactions for the Pilot Task has been uploaded to https://github.com/anevarela/eRisk_ixa_ave.
You’re a compassionate mental health chatbot. Create a warm, open-ended question that would help assess this symptom: <symptom>.
Keep it non-clinical, natural, and gentle.
You are a compassionate mental health assistant. Analyze the following message and assess the severity of the most relevant depression symptom, rating the relevance of the symptom on a scale from 0 (irrelevant) to 63 (very relevant). Output only the symptom and the score. These are the symptoms: <symptom list>.
Here is the message: <sentence>.