In this work, we tackle the first task of the eRisk Lab for 2024: analyzing Reddit comments to identify symptoms of depression. Our objective is to use information collected from Reddit's various user communities to identify sentiments associated with symptoms of depression listed in the BDI-II questionnaire. As a place where people may openly discuss ideas and experiences, Reddit provides a unique perspective on how people express themselves, including their dificulties with mental health. Through utilizing this content created by users, we hope to bring to light how depression presents itself in digital environments.
Millions of people worldwide sufer from depression, a widespread mental health illness that raises
serious public health concerns [
Early detection of depression is critical since it allows for prompt support and care for those who are
afected. But conventional diagnosis techniques frequently depend on patients’ self-reported symptoms
during professional evaluations, which can be arbitrary and vulnerable to underreporting because of
stigma or ignorance [
Through the use of artificial intelligence (AI) algorithms to examine social media information,
researchers can find hidden patterns and possibly even diagnose depression. These realizations can
guide the creation of support networks, preventative initiatives, and treatments that are suited to the
requirements of those who are at risk [
eRisk seeks to close the gap among research and practice by arranging shared tasks and provide a
place for practitioners and researchers to work together to create eficient early risk detection. eRisk is
dedicated to utilizing digital data in order to facilitate the creation of solutions that are both scalable
and easily accessible, with the ultimate goal of improving the lives of those who are dealing with mental
health concerns [
Under the eRisk system, we focus on the first task for 2024, which aims to identify depressive
symptoms from user-generated content. This involves ranking sentences from a sample of Reddit user
postings based on their relevance to depressive symptoms. Participants are tasked with providing
rankings for the 21 symptoms outlined in the Beck Depression Inventory (BDI) questionnaire [
The Beck Depression Inventory (BDI) is a widely used self-report questionnaire carefully constructed
to assess the intensity of depression symptoms. Each one of the 21 items depicts a diferent depression
symptom, and participants rate their level of experience with each symptom during a certain time
period [
The first task in eRisk 2024 is an extension of the one in 2023. In 2023, the goal has been similar to this
year: score sentences from a selection of user writings based on their relevance to a depressive symptom
[
FormulaML’s approach to eRisk 2023 included preprocessing and encoding the dataset with Sentence
Transformers, namely the MiniLM-L3-v2 model. They used the BDI-II questionnaire as a query,
computed cosine similarity scores, and applied weighted scoring to assess sentence relevance. This strategy
allowed for a complete investigation of symptoms recognition and similarity assessment [
OBSER-MENH compared texts and symptoms using Sentence Transformers (ST), which convert them
into fixed-sized vectors. They used pre-trained models such as BERT to build these vectors, which can
represent phrases in a dense vector space. For example, the all-mpnet-base-v2 model converts phrases
into 768-dimensional vectors, whereas the all-distilroberta-v1 model is trained on a large dataset using
the distilroberta-base model. They also employed the all-MiniLM-L12-v2 model [
BLUE team used an approach inspired by current research to enhance data using LLMs, creating
synthetic Reddit posts associated with every BDI-II symptom to increase dataset diversity. While
they expected that adding more diverse data would improve results, their findings revealed that the
model using original BDI-II responses outperformed the model using produced data. They emphasized
the problem of the specificity of ChatGPT data and proposed improving prompts for semantically
comparable but heterogeneous material. Despite this, they acknowledged the useful character of the
created text and pointed out the potential of LLMs to generate mental health data for future research
[
In 2023, Formula-ML received the top scores across all parameters, followed by OBSER-MENH and then the BLUE team. In total there were 10 participating teams.
Regarding the dataset utilized in this study, it comprised two main folders: one containing the training data and the other the test data. Within the training data folder, a subfolder contained a total of 3107 TREC files, each file representing a distinct user.
Furthermore, the training folder encompassed two CSV files containing subsets of ’docnos’ and corresponding labels categorized by query. One CSV file documented labels assigned by the majority of annotators for each ’docno,’ while the other recorded consensus labels. In the former, a ’1’ label signified agreement among at least two out of three annotators regarding the text’s relevance, whereas in the latter, a ’1’ indicated unanimous agreement among all three annotators or disagreement among them.
Notably, the aggregated texts from all TREC files in the training set totaled approximately 4.2 million, illustrating that the annotated data represented only a fraction of the entire corpus. This disparity is expected, given the substantial volume of texts that are unrelated to any of the queries.
It is also important to note the inclusion of the test dataset folder, which has the same structure as the training dataset. This folder contains 553 TREC files, each with numerous sentences similar to the training dataset. These files are critical for evaluating the performance of models trained on the training dataset, since they allow us to analyze generalization capabilities and model eficacy on previously encountered data. This thorough assessment procedure guarantees the model’s predictive potential is strong and reliable, hence improving the general quality and validity of the study’s results.
Capturing semantic nuances in natural language processing (NLP) presents a major challenge due to
language’s innate complexity and ambiguity [
The major goal of this newly designed feature is to compute cosine similarity scores for Beck
Depression Inventory (BDI) symptoms and dataset sentences. Cosine similarity is a frequently employed
measure in NLP that assesses the similarity of two vectors by calculating the cosine of their angle
[
The four alternative responses to each of the 21 BDI symptoms are encoded into dense embeddings
using the paraphrase-MiniLM-L12-v2 model. These embeddings capture the semantic information
contained in each response, converting textual data into high-dimensional vector representations [
Apart from the fact that a sentence is relevant even if the person afirms they don’t have that symptom, a text is only relevant if it is talking about the user only, so: • A text such as “ I feel sad lately” would be labeled as 1 for the first BDI query which is ‘sadness’, but • A text such as “ My sister is very sad”/ “that is sad”, would be label 0 as it is not written from a subjective point of view
As this is an important aspect of our texts, I have created an additional feature called “first_person” which can get values of 0 or 1. If in that specific text there are pronouns such as: [’i’, ’me’, ’my’, ’mine’, ’myself’, ’im’], then it would be relevant for this new feature.
However, there are many sentences written in first person that actually are irrelevant for a specific query, such as this one: “im sure that symbolizes loneliness” labeled 0 for query 1, which denotes sadness.
To balance this impediment, I have added a condition to this “first_person” feature to only be labeled as 1 if there any of those pronouns are present in the text AND also the cosine similarity is greater than 0.4 for that query.
In the beginning phase of my research, I undertook a thorough review of numerous machine learning
models to establish a performance baseline. I specifically explored with models known for their
computing eficiency, such as Logistic Regression, MLPClassifier, DecisionTreeClassifier, and ensemble
models such as GradientBoostingClassifier, XGBClassifier, and LGBMClassifier. My motivation for
choosing those models was for a couple of reasons: computational eficiency and the possibility to
provide first glimpses into the dataset’s features [
Given the fact that Logistic Regression and MLPClassifier were the top performers according to f1_score at this point in my research, I have decided to continue working with these two models in my future undertakings.
The dataset has been split into training and testing subsets using StratifiedKFold, which preserves class distributions. This splitting approach preserves the consistency of the evaluation process by reducing biases and increasing the reliability of the outcomes [19].
In each iteration, the model’s hyperparameters are fine-tuned using a GridSearchCV pipeline. This pipeline systematically investigates a variety of configurations contained within the parameter grid [20], including for example for MLPClassifier parameters such as: hidden_layer_sizes, alpha, and learning_rate_init [21], and for LogisticRegression: C, penalty [22]. Its major goal is to determine the ideal parameters that result in the highest F1 results.
A key component of the pipeline is the RandomOverSampler, a mechanism used to solve class imbalance. This approach works by oversampling minority class instances within each fold, which corrects the dataset’s disproportionate distribution of class labels [23]. By boosting the proportion of minority class samples, the RandomOverSampler helps ensure the model is trained on a more balanced dataset, lowering the risk of biased predictions and improving the classification model’s overall performance.
Utilizing joblib, I’ve preserved the models that demonstrated the highest F1 scores across all five folds for each individual query. Given the performance variations between Logistic Regression and MLPClassifier across diferent query contexts, I’ve made the decision to retain both models for future use in making predictions on the test dataset. This approach ensures that we can leverage the strengths of each model depending on the specific characteristics and complexities of the queries encountered during testing, thus optimizing our predictive capabilities and ensuring robustness in our analysis.
I submitted three entries structured as follows: 1. The first submission served as a benchmark to gauge the performance of my models. For this submission, I exclusively relied on the cosine similarity scores generated by the paraphraseMiniLM-L12-v2 model. These scores were arranged in descending order for each query, and only the top 1000 scores per query were considered. 2. In the second submission, I utilized the probability scores outputted by my algorithms for each query. Depending on the query, I selected either MLPClassifier or Logistic Regression, based on their respective performance during the training phase. Subsequently, I computed the mean of the cosine similarity score and the algorithm’s probability score per query. 3. The third and final submission employed both MLP and Logistic Regression models per query, determined by their superior performance during training. However, a distinctive feature of this submission was the utilization of diferent thresholds to generate scores: • For queries with an F1 score ranging between 0.60 and 0.75, the new relevance score was computed as:
0.3 × prediction_probability + 0.7 × cosine_similarity_score • Queries with an F1 score between 0.75 and 0.85 utilized a relevance score calculated as: 0.5 × prediction_probability + 0.5 × cosine_similarity_score • For queries achieving an F1 score between 0.85 and 0.95, the relevance score was determined as:
0.7 × prediction_probability + 0.3 × cosine_similarity_score
In the presented tables we can see the outcomes of our team compared with the results of the team/teams that performed best, determined through voting by the annotators. Table 1 delineates the system performance rankings derived from a majority voting perspective, while Table 2 provides rankings based on unanimity among all three annotators.
Upon comparing the proposed methodologies, notable performance enhancements were observed with the approach integrating varied weights for cosine similarity and the probability score generated by the classification algorithm. Notably, the system named "MindwaveMLMiniLML12MLP_weighted" exhibited the highest performance, closely followed by "MindwaveMLMiniLML12MLP_0.5," which computed the mean between the probability score and cosine similarity. In contrast, "MindwaveMLMiniLML12" served as our comparative baseline. These findings afirm that while the improvements were not groundbreaking, the approach demonstrated eficacy by yielding higher scores in comparison to the baseline method.
In this research we have proposed an approach of ranking sentences based on their relevance to 21 depression symptoms derived from the Beck Depression Inventory (BDI) questionnaire.
Results from the evaluation process revealed insights into the performance of the developed computational models. Notably, models incorporating a weighted combination of cosine similarity scores and probability outputs from classification algorithms demonstrated superior performance compared to baseline approaches.
Feature engineering is one area that might benefit from some improvements. Further research into more complex methods for obtaining significant characteristics from textual data may result in more accurate depictions of the underlying semantics.
Experimenting with ensemble learning techniques may also yield further insights for improving system performance. Ranking results may be more reliable and robust if several models are combined and their combined intelligence is utilized [24]. Furthermore, examining domain strategies specifically designed for textual data related to depression may aid in bridging the gap between general pre-trained models and the particular demands of mental health domain tasks.