We present working notes for DS@GT team in the eRisk 2024 for Tasks 1 and 3. We propose a ranking system for Task 1 that predicts symptoms of depression based on the Beck Depression Inventory (BDI-II) questionnaire using binary classifiers trained on question relevancy as a proxy for ranking. We find that binary classifiers are not well calibrated for ranking, and perform poorly during evaluation. For Task 3, we use embeddings from BERT to predict the severity of eating disorder symptoms based on user post history. We find that classical machine learning models perform well on the task, and end up competitive with the baseline models. Representation of text data is crucial in both tasks, and we find that sentence transformers are a powerful tool for downstream modeling. Source code and models are available at https://github.com/dsgt-kaggle-clef/erisk-2024.
The eRisk challenge, as detailed in Crestani et al. [
Task 1 involves ranking documents relevant to symptoms of depression as identified in the BDI-II
questionnaire. The objective is to submit runs containing the top thousand documents pertinent to a
specific symptom. This is a standard information retrieval task commonly applied in search engines.
The evaluation is conducted against a pool of human assessors who are experts in the field, categorizing
sentences as either relevant or not relevant. There are two types of question relevancey (qrel) scoring:
majority and unanimity [
Inspired by last year’s eRisk competition participants, particularly the Formula-ML team who achieved the highest scores across the board, we chose to build our model using sentence transformers alongside traditional NLP methods. Our goal was to compare and contrast the performance of count methods, vector models, and rich semantic representations on early risk detection systems.
This task is well-suited for supervised learning approaches. We utilized logistic regression to make binary relevance predictions for each symptom in the questionnaire, optimizing for F1 scores and accuracy using the relevance labels from the QRELS provided by the lab as our baseline for binary relevance decisions. Our internal validation showed that diferent iterations of our models achieved over 70% in mean accuracy and mean F1 scores, with sentence transformers achieving the highest scores of 89% mean F1 and 90% mean accuracy. This gave us confidence in submitting our results using the sentence transformers and Word2Vec models.
A training and test TREC dataset contains user posts from Reddit. It contains over 500k users with almost 20 million sentences.
The dataset was formatted as TREC documents, each containing a unique document number (DOCNO) and a text field (TEXT) containing the post content. The test files contain additional fields PRE and POST fields with text content that may add context to the TEXT field. We compute some statistics on the dataset, including the number of users, the number of sentences, and the average and median number of words per sentence in Table 1. <DOCNO> s _ 0 _ 2 _ 4 < /DOCNO>
<TEXT > I f e e l d e p r e s s e d < / TEXT > < /DOC>
Two distinct modeling approaches were employed for Task 1: a baseline and a sentence-transformerbased solution. The baseline system uses classical classifiction models (e.g. Naive Bayes) trained on binary relevance labels as a proxy for relevance ranking. The sentence-transformer-based model does the same, but uses sentence embeddings as the underlying representation of the text.
We ran our experiments on Google Cloud Platform (GCP) compute instance sized n1-standard-2.
We used PySpark [
We collected the TREC formatted files and undertook extensive data cleansing, including the removal of special characters and the correction of formatting errors. Subsequently, we merged the cleaned ifles using a python script that compressed the documents into the Parquet format, a binary format optimized for fast querying and favored over CSV and other formats. To ensure both persistence and easy access during model development, we hosted the data in Google Cloud Buckets, and access to the data was facilitated through the Google Cloud API.
We trained a classifier for each question of the BDI-II to determine document relevancy to specific questions. Each document was evaluated against the classifier to obtain a relevancy probability, which was then used to rank the documents. The training data was vectorized using a count vectorizer, Word2Vec, and a text transformer. We hypothesized that framing the ranking task as a classification problem would yield satisfactory performance on the leaderboard. Additionally, we posited that unsupervised text representations with higher learning capacities would perform better in the relevancy classification task. See Table 2 for a summary of the modeling approaches.
We transform text from it’s natural representation into a representation that can be used by a machine learning model. The counting vectorizer treats each document as a vector by counting the frequency of each word in the document. TF-IDF is a normalization of the count vector that accounts for the term-frequency (TF) and inverse document frequency (IDF) of each word. Word2Vec is a neural network model that learns word embeddings by predicting the context of a word in a sentence, using either the continuous bag of words (CBOW) or skip-gram model.
We also used the sentence transformer model [
Classifier using vector of word counts, learns coeficients that determine how to weight words to fit a decision boundary.
Classifier using a word embedding that captures distributional semantics of the bag of words model. The unsupervised task should transfer knowledge to the simpler linear classifier.
Classifier using the inductive properties of transformer layers to model a sequence of words in an auto-regressive manner.
We had to aggressively filter out low-quality sentences since embedding documents took many hours for hundreds of thousands of documents out of millions. We used relevancy predictions from a simpler logistic regression model on count vectorized data to filter out irrelevant documents before transforming them with the sentence transformer model. We also experimented with filtering based on the compression ratio of the text, as we found both low and high compression ratios to be indicative of irrelevant documents. The compression ratio is the ratio of the size of gzip compressed text to the size of the original text. The mean compression ratio of the documents was 0.9 with a standard deviation of 0.1.
We report the results of our models on the public leaderboard in Table 3. After evaluating the hidden test set, our models score zero nearly across the board. We find that the transformer-based model scores an order of magnitude more than the models without the recall-precision and NDCG scores.
NDCG 0.010 0.003 0.001 0.000 0.000
MAP 0.000 0.000 0.000 0.000 0.000
Majority Voting R-PREC 0.009 0.001 0.000 0.000 0.000
NDCG 0.014 0.003 0.001 0.000 0.000
In this project, we assumed that a classifier could be repurposed as a functional ranker. However, our results on the leaderboard contradicted this assumption. One potential reason for the poor performance is that a classifier optimized for a specific metric (such as F1 or accuracy) may not be well-calibrated to the actual data distribution. Given more time, we would have explored proper ordinal regression through learning to rank methodologies.
During pipeline construction, we observed a non-trivial number of training examples with high
repetition. We would have preferred to explore more options to reduce the number of relevant training
examples. One potential solution to eliminate irrelevant documents would be to filter relevance using
keyword-based information retrieval algorithms [
For future iterations of this task, we propose leveraging various language models to explore the capabilities of transformer-based systems. Additional strategies could include employing prompt engineering, given that large language models are efective search engines; this could prove to be a compelling area of exploration for information retrieval systems. Other potential avenues include improving our pre-filtering process to remove non-relevant posts, fine-tuning pre-trained language models, and employing retrieval-augmented generation(RAG) to enhance the accuracy of predictions.
Eating disorders (ED) are serious mental health conditions characterized by abnormal eating habits.
Early detection and severity assessment are crucial for timely intervention and treatment. Task 3 aims
to explore the feasibility of automatically estimating the severity of symptoms of ED using the Eating
Disorder Examination Questionnaire (EDE-Q) based on the activity of social media users. We aimed
to leverage state-of-the-art NLP techniques and Machine Learning methods to design a pipeline for
detecting the severity of the signs of EDs. Our approach of BERT-based [
For this task, participants developed a system for predicting responses to an Eating Disorder Examination Questionnaire based on a Reddit post history. The EDE questionnaire is a 28-item self-reported questionnaire adapted from the Eating Disorder Examination (EDE). The questionnaire covers several domains: dietary restraint, eating, shape, and weight concerns. The responses range from 0 to 6, corresponding to the severity of symptoms. Our goal was to predict the responses to 22 of 28 EDE-Q questions based on user post history.
During the training phase of the challenge, the eRisk team released an entire history of user writings along with the corresponding answers to the EDE Questionnaire responses. The training data includes users from the 2022 and 2023 datasets, while the test data comprises new users from 2023. The combined training set consisted of 74 subjects. For each user or subject, we had access to a history of postings on Reddit and their responses to the EDE-Q questionnaire. We used the user responses as ground truth for training our machine learning models (discussed in the methodology section). The test set consisted of a history of postings for 18 users, structured similarly to the training set.
The goal of task 3 was to predict the responses of the 2024 users to the EDE-Q questionnaire. To achieve this, we needed to determine to what extent the characteristics associated with eating disorders are reflected in the social media user’s post and comment history. Note that the responses to the EDE-Q are integers ranging from 0 to 6. We approach this problem as a multi-label and multi-output classification task.
The eRisk lab organizers made datasets available for the 2022 and 2023 users. We processed TREC data into JSON using BeautifulSoup and etree.XMLParser. Preprocessing involved converting the JSON files to readable DataFrames in Pandas and cleaning the text data to remove noise, such as URLs, hashtags, and special characters. We tokenized the text and applied lemmatization, stemming, and stopword removal to normalize the data.
We leveraged BERT to generate text embeddings. For this task, we used the bert-base-uncased
[
BERT has a maximum input sentence length of 512 tokens, including the [CLS] and [SEP] tokens, and generates vectorized embeddings in R768. For posts that were too long to feed into the transformer model, we concatenated all user posts together in chronological order. We broke the resulting text into chunks of length , where = 512 − 2 = 510.
We hypothesized that the 768 dimensions of the sentence embeddings might be too large. Therefore,
we standardized the characteristics and performed principal component analysis (PCA) for
dimensionality reduction [
Given the substantial data requirements of deep learning models, we opted for classical machine learning models for our analysis. We selected five models: Random Forest, Extra Trees, XGBoost, Ridge Regression, and Support Vector Machines (SVM). Random Forest and Extra Trees are ensemble methods that combine multiple decision trees to improve performance and reduce overfitting. XGBoost is a gradient boosting algorithm that is known for its speed and performance on tabular data. Ridge Regression is a linear model that incorporates L2 regularization to prevent overfitting. SVM is efective for high-dimensional data and can handle non-linear relationships. For a model baseline, we used results from the runs for the 2023 challenge, which involved the same task.
To evaluate the performance of the models, the eRisk team provided several evaluation metrics: Mean Zero-One Error, Mean Absolute Error, Macro-averaged Mean Absolute Error, Restraint Subscale, Eating Concern Subscale, Shape Concern Subscale, Weight Concern Subscale, and Global ED (the global score).
Random Forest Classifier with high-dimensional embeddings performed the best overall, achieving an accuracy of 0.3303 and an MAE of 2.1091. This model benefited from the high-dimensional feature space, which likely captured the complex patterns in the data more efectively. After applying dimensionality reduction, the Extra Trees Classifier emerged as the top performer with an accuracy of 0.3273. This suggests that reducing the feature dimensions helped improve the model’s generalizability. Comparatively, the XGBoost Classifier showed lower performance across both feature sets, with an accuracy of 0.2606 and 0.2515 for high-dimensional and reduced-dimensional data, respectively. This may be due to the model’s sensitivity to hyperparameters, which requires careful optimization.
The Ridge Classifier and SVM showed moderate performance. Notably, the Ridge Classifier’s performance significantly dropped when trained on reduced dimensions data, with an accuracy of 0. 2000, highlighting its limitation in handling reduced feature spaces efectively. The SVM performed consistently across both feature sets, but its overall accuracy remained lower compared to Random Forest and Extra Trees.
We report the results of our models on the public leaderboard in Table 6. While our models generally met baseline metrics, they significantly outperformed the baseline in Mean Zero-One Error (MZOE), Mean Absolute Error (MAE), and MAE macro metrics on the test data. However, our models fell short in meeting the baseline for the Eating Concern (ECS) and Weight Concern (WCS) subscales, suggesting a need for additional techniques like topic modeling or semantic analysis. Despite these challenges, our team consistently ranked in the top 5 among five participating teams, with a total of 14 submissions. This highlights the robustness of our approach.
Our study achieved promising results in predicting the severity of eating disorder symptoms from social media content. Further research should focus on refining our models and incorporating advanced techniques to enhance performance, ultimately contributing to improved interventions and support systems for individuals with eating disorders.
Further research may explore the potential of deep learning models for predicting the severity of eating disorder symptoms from social media content. Deep learning architectures, such as recurrent neural networks (RNNs), ofer the ability to capture complex patterns in text data more efectively, which could lead to improved performance. Given that deep learning models often require large amounts of data, we would need to explore data augmentation techniques. We could generate additional data using techniques such as Language Model Fine-Tuning (LMFT) or services like TextSynth to increase the diversity and size of our dataset.
In this document, we presented our working notes for the eRisk lab of the CLEF 2024 conference, where we submitted entries for Task 1 and Task 3. In Task 1, we propose a system that ranks documents based on their relevance to the BDI-II questionnaire using a binary classifier as a proxy for ranking. The performance is lackluster, and future work would explore proper ordinal regression methods such as learning to rank. In Task 3, we developed a system that predicts the severity of eating disorder symptoms based on user post history. We use BERT embeddings to generate features for classical machine learning models, and prove competitive with the baseline models.
In all our tasks, we find that sentence transformers are a powerful tool in representing text data, and that the choice of model and feature representation can significantly impact the performance of the system. Source code and models are available at https://github.com/dsgt-kaggle-clef/erisk-2024.
Thank you to the DS@GT CLEF for subsidizing compute and storage used for this project. Thank you to the eRisk organizers for providing the dataset and evaluation for the competition.