Depression is one of the most common mental health problems worldwide, afecting more than 280 million people. Early recognition and intervention are crucial to prevent severe consequences. This study participated in Task 1 of CLEF eRisk 2025, focusing on ranking the relevance of user-generated content to symptoms defined in the Beck Depression Inventory (BDI). Using various machine learning models and natural language processing techniques, including logistic regression (LR), Support Vector Machine(SVM), and BERT-based models, we aimed to advance early detection of depressive symptoms in online text, ofering new tools for future mental health early prevention.
Depression is a mental disorder that has a staggering impact globally, afecting more than 280 million
people, as noted by the World Health Organization [
The rise of social media and digital communication platforms has created an opportunity for mental
health researchers to analyze user-generated content in real time. Unlike traditional clinical assessments,
which are based on self-reports or structured interviews, online expressions such as tweets, Facebook
posts, and forum discussions provide researchers with unique insight into individuals’ emotional and
psychological states [
In recent years, a growing body of literature has explored the use of natural language processing
(NLP) techniques to assess mental health from online content. For example, Coppersmith et al. used NLP
methods to detect linguistic signals related to mental health from Twitter posts, showing how social
media text can help identify mental health concerns [
This paper summarizes our participation in Task 1 of CLEF eRisk 2025 [
This task focuses on detecting depression signals in user writings on social media. Participants are asked to build a system that could rank provided sentences for each of the 21 symptoms of depression from the Beck Depression Inventory II (BDI-II) questionnaire. A sentence is considered relevant if it contains the signal about the user’s condition of a particular symptom.
The oficial TREC-formatted annotated corpus from the eRisk shared tasks 2023 and 2024 were
provided [
In this study, a range of machine learning techniques were employed to analyze textual data and
perform binary classification of sentence relevance, ranging from classical machine learning approaches
to more advanced transformer-based deep learning models. The experimental design consisted of
multiple run groups, each applying distinct vectorization and modeling strategies. Initial experiments
used logistic regression and support vector machine (SVM) classifiers paired with CountVectorizer and
TF-IDF vectorization, respectively. In subsequent experiments, more advanced feature representations,
such as ClinicalBERT and Sentence-BERT (SBERT) embeddings, were incorporated to better capture
semantic context. Models were evaluated using standard classification metrics, including precision,
recall, F1-score, and computational eficiency. The final experiments explored a hybrid retrieval approach
that combined BM25 lexical matching [
During the training process, the datasets were merged and split into train and test sets in an 80/20 ratio. Relevance labels were binarized, assigning a value of 1 to relevant sentences and 0 to non-relevant ones. No external datasets were used during training or evaluation.
We submitted five runs in total in the requested TREC format: “result” (LR with CountVectorizer), “LR file
combined” (LR with TF-IDF representations), “SVM file combined” (SVM with TF-IDF Representations),
“BERT CONSENSUS” and “BERT MAJORITY”. Two types of evaluation schema were used to determine if
sentences were correctly ranked to a symptom or not: majority-based (a sentence was deemed relevant
if at least two of the three assessors marked it so) and unanimity-based (a sentence was deemed relevant
only when all three assessors agreed) [
From both results table 1 and 2, the best-performing approaches involved fine-tuning the BERT [
Given the scale of the eRisk 2025 test set (over 17 million unlabeled sentences), a symptom-aware keyword filtering pipeline was developed to pre-filter candidate sentences. This pipeline used symptomspecific keyword lists to reduce computational burden and improve inference speed. Filtered sentences were paired with their corresponding symptom prompts and passed to the BERT model for scoring. For each symptom, the top 1,000 sentences with the highest predicted relevance scores were selected and formatted according to TREC submission standards. The BERT model fine-tuned on unanimity-based labels achieved the highest scores on all five metrics in both ranking-based evaluation schema.
We tried fitting SVM classifier with ClinicalBERT [ 10] embeddings, which is pretrained on clinical corpora. This approach achieved a slightly higher accuracy of 0.79 on the validation set, with strong performance for non-relevant sentences (F1 = 0.85) but moderate efectiveness for relevant ones (F1 = 0.63). The macro F1-score was 0.74, and the weighted F1-score was 0.79. Although ClinicalBERT provided richer semantic context, SVM with TF-IDF representations ofered comparable results at a significantly lower computational cost, making it a viable alternative for large-scale or resourceconstrained applications.
In this experiment, Sentence-BERT (SBERT) [11] was adopted, ofering semantically meaningful sentence embeddings optimized for short text classification. Two classifiers – Support Vector Classifier (SVC) and Logistic Regression (LR) [12, 13] – were trained using embeddings generated by SBERT. The training data was derived from a merged 2023–2024 dataset, with sentence texts serving as input features and the binary “relevance” label as the output. To ensure consistency, the same 80/20 train-validation split from earlier experiments was used. Both SVC and LR models were trained on identical SBERT-based features to facilitate a fair comparison.
Despite SBERT’s semantic strength and computational eficiency, both classifiers achieved only moderate performance. Accuracy for both models were about 77% on the validation set, with strong precision and recall for the non-relevant class but lower metrics for the relevant class. These results suggest that while SBERT captures contextual meaning efectively, further optimization is required, particularly to address class imbalance and fine-tune model parameters. 3.2.3. BM25 with SBERT A hybrid retrieval framework was implemented that combined traditional BM25 lexical retrieval with semantic re-ranking using a pre-trained Sentence-BERT model. The two-step approach was designed to leverage BM25’s strength in lexical term matching while addressing its limitations in semantic understanding through contextualized embeddings. While BM25 performs efectively in retrieving documents with overlapping vocabulary, it often fails to capture semantically related candidates phrased diferently – an area where embedding-based models demonstrate superior performance.
NDCG
The evaluation on the validation set demonstrated a weighted precision of 0.47, recall of 0.34, and an F1-score of 0.39. BM25 retrieval alone produced strong recall by retrieving highly relevant documents, whereas precision improved significantly after applying Sentence-BERT-based re-ranking. However, precision (0.09) and recall (0.18) for Class 1 (relevant) sentences remained notably lower than for Class 0 (non-relevant) sentences, indicating that while semantic matching improved, the consistent identification of truly relevant examples remains a challenge.
The ranking evaluation demonstrates that fine-tuned BERT models consistently outperform traditional classifiers and unsupervised methods in retrieving clinically relevant sentences. Notably, the BERT models trained on unanimity-based labels achieve higher ranking scores across all metrics compared to those trained on majority-based labels. This suggests that when the labeling task is easier for human experts and/or they agree, the examples yield clearer relevance signals, enabling the model to rank pertinent sentences more efectively. Traditional machine learning classifiers such as logistic regression and SVM perform close to baseline levels in ranking metrics, highlighting their limited ability to capture nuanced clinical relevance for this task. The hybrid BM25 + SBERT approach improves lexical-semantic matching but still struggles with precision and recall, underscoring the importance of deep contextual understanding that transformer models like BERT provide. Overall, these results emphasize the value of transformer-based models trained on consensus labels for achieving improved retrieval performance in clinical sentence ranking tasks.
The results suggest that BERT outperformed traditional models such as logistic regression (LR) with term-frequency or TF-IDF features, primarily due to its ability to understand contexts within language. BERT’s superiority is likely attributable to its transformer-based architecture, which enables nuanced comprehension beyond the keyword-level focus of term-frequency based features. Among the two traditional vectorization methods, TF-IDF yielded better results than term-frequency, emphasizing the importance of weighting term, though both approaches fell short in capturing semantic meaning.
Despite these promising findings, the study had several limitations. A key issue was the imbalanced labels: sentences labeled as “relevant” (Label 1) were underrepresented, negatively impacting the performance of LR in particular. Future research could address this limitation through oversampling techniques or contrastive learning methods to enhance minority-class representation. BERT, while efective, also introduced scalability concerns due to its high computational cost, which constrained the number of models and hyperparameter combinations that could be evaluated.
This supports BERT’s potential for high-impact use cases where contextual understanding is essential. As NLP methods for classification tasks continue to evolve, integrating advanced NLP models like BERT or domain-specific variations, such as Opinion-BERT [ 14], which has been applied in mental health analysis to detect nuanced sentiment and psychological states, ofers a path toward deeper and more reliable insights [15].
During manuscript preparation, the authors used ChatGPT-4o for grammar and spelling check, then reviewed and edited the content as needed. They take full responsibility for the publication’s content. [10] E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, M. McDermott, Publicly available clinical BERT embeddings, in: A. Rumshisky, K. Roberts, S. Bethard, T. Naumann (Eds.), Proceedings of the 2nd Clinical Natural Language Processing Workshop, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 72–78. URL: https:// aclanthology.org/W19-1909/. doi:10.18653/v1/W19-1909. [11] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019. URL: https://arxiv.org/abs/1908.10084. [12] D. R. Cox, The regression analysis of binary sequences, Journal of the Royal Statistical Society:
Series B (Methodological) 20 (1958) 215–232. [13] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (1995) 273–297. [14] M. M. Hossain, M. S. Hossain, M. F. Mridha, M. Safran, S. Alfarhood, Multi task opinion enhanced hybrid BERT model for mental health analysis, Scientific Reports 15 (2025) 3332. URL: https:// www.nature.com/articles/s41598-025-86124-6. doi:10.1038/s41598-025-86124-6, publisher: Nature Publishing Group. [15] A. Gaurav, B. B. Gupta, K. T. Chui, BERT Based Model for Robust Mental Health Analysis in Clinical Informatics, in: 2024 21st International Joint Conference on Computer Science and Software Engineering (JCSSE), Phuket, Thailand, 2024, pp. 153–160. doi:10.1109/JCSSE61278. 2024.10613729.