This paper presents our participation in the CLEF eRisk 2024 competition, where we focus on the early detection of anorexia, eating disorders, and depression from social media data. For the first task (search for symptoms of depression), we explore diferent sentence semantic similarity models, achieving robust performance in identifying early depressive symptoms, achieving the second best results in most of the evaluation metrics. For the second task (early detection of signs of anorexia), we use an ensemble of traditional machine learning algorithms. In the eating disorders detection task, we use contextualized embeddings from BERT to represent the texts, and then, classify them with a neural network. The findings highlight the posibility and room for improvement of early intervention and the potential of social media analysis to provide timely support for individuals at risk.
The main goal of the CLEF eRisk 2024 competition is to promote the development of advanced approaches
for the early detection of various mental health issues through social media analysis [
This competition, part of the Conference and Labs of the Evaluation Forum (CLEF), has been a
significant event since its inception in 2017. It brings together researchers and practitioners to collaborate
on innovative solutions for identifying early signs of mental health disorders from social media data. By
analyzing textual content from social media posts, we can gain valuable insights into individuals’ mental
states and potentially provide early warnings for those at risk. Early detection is crucial for various
applications, from identifying potential sexual ofenders to detecting victims of suicidal tendencies,
enabling interventions before it is too late [
Our participation in the CLEF eRisk 2024 competition focuses on the development and refinement of
models to detect early signs of anorexia, eating disorders, and depression. Early intervention in these
cases is crucial for providing timely support and improving outcomes for afected individuals [
In the last decade, social media has become a vital platform for individuals to express their thoughts,
emotions, and ideas. This has opened up new avenues for the analysis of online data, which can be
leveraged for numerous purposes such as business and marketing strategies, political planning, stock
market predictions, and emergency awareness [
In the healthcare domain, social media posts have been instrumental in detecting disease outbreaks,
identifying smoking patterns, and recognizing adverse drug reactions, among others [
In this paper, we present our approaches and methodologies for each of the three tasks of eRisk 2024: search for symptoms of depression (task 1), early detection of signs of anorexia (task 2), and measuring the severity of the signs of eating disorders (task 3). We employ a variety of NLP techniques like BERT for creating word embeddings, Machine Learning, and Deep Learning, to develop robust models capable of accurately detecting signs of anorexia, eating disorders, and depression. For task 1, we we achieved high accuracy in the early detection of anorexia.
In this section, we describe the diferent approaches that we have used for each one of the tasks.
The first task involved ranking and classifying sentences from a collection of user posts based on their
relevance to symptoms of depression listed in the Beck Depression Inventory-II (BDI-II) questionnaire
[
We explored three diferent approaches: the first one based on sentence semantic similarity models, the second one based on a RoBERTa classifier model and an ensemble that combined the previous ones.
Our semantic similarity approach is based on Sentence Transformers [
Thus, these models allow us to obtain representations of the sentences (embeddings) so that later, by using mathematical formulas such as cosine similarity, the level of similarity of meanings between both texts can be extracted.
Cosine similarity is a metric used to determine the similarity between two vectors. It is calculated as
the cosine of the angle between the first and second vectors. A value of 1 indicates that the vectors
are identical, 0 means that they are orthogonal (unrelated), and -1 implies that they are opposite. It is
useful in semantic textual similarity, semantic search, or paraphrasing [
In this approach, the following Sentence Transformers models were used:
• all-MPNet-base-v2: To obtain this model, the microsoft/mpnet-base model was fine-tuned on a
dataset of 1B sentence pairs using a self-supervised contrasting learning objective. This particular
model maps sentences and paragraphs to a dense vector space of 768 dimensions and is typically
used for tasks such as clustering or semantic search[
Each model was used to calculate the numerical representations [
In other words, if a sentence has a high similarity to the sentences indicating sadness (symptom 1), it is classified as such. For example, suppose we have the sentence “I am very sad, really, sad sad sad” which belongs to symptom 1 (“Sadness”), the model would calculate its numerical representation and calculate the cosine similarity with those of the reference sentences indicating sadness (label 1). If the similarity is high with sentences labelled under the same symptom (1), the sentence is classified as indicative of sadness with label “1”.
Apart from the classification, the model also returned a decimal number between 1 and 10,
reinterpreted from its cosine similarity (since cosine similarity returns a decimal number from 0 to 1), as the
degree of relevance for that symptom in the BDI-II [
It is important to clarify that these models have a disadvantage in that they only label sentences once, i.e. sentence that may be related with more than one symptom at a time are not classified again, these models only address a multi-classification task but not multi-labelling. For example for the sentence “I am sad and crying” relevant for symptom 1 (“Sadness”) and 10 (“Crying”), it would be classified as “1” or “10” with a single degree of relevance.
In our second approach, we deal with the first task as a multi-labelling classification problem.
The architecture chosen was RoBERTa (a Robustly Optimised BERT Approach) [
In our study, we adjusted the model to predict only 21 labels (instead of the 28 in the original model),
which are the 21 symptoms described in the BDI-II [
The fitted model in addition to the classification, also provides the probability of each predicted label. This probability provided us with a relevance level for the test symptom in which the sentence was classified.
We also explored the combination of the previous approaches to deal with the task. We studied diferent ensembles of the sentence semantic models and the classifier RoBERTa. Based on our results during the development phase, our final submission was formed by aggregating the results from the all-MPNet-basev2, all-MiniLM-L12-v2, and RoBERTa classifier models by majority voting and averaging the ranking value.
For this task, the goal was to perform binary classification of users, determining whether they were at risk for anorexia or not. The model had to analyze the user’s sentences sequentially. If at any point the model deemed the user to be at risk for anorexia based on the sentences, it had to issue an alert. This alert was communicated to a server. Importantly, once a positive risk alert was submitted for a user, it could not be changed or retracted in later stages of analyzing that user’s data.
This binary classification task was tackled using a ensemble model of three classical algorithms, in addition to experimenting with diferent text representations. The ensemble models combine the predictions of several models to produce a final prediction, which can help reduce overfitting and improve generalisation by averaging the weaknesses and strengths of each individual model.
Specifically, our ensemble is composed of three diferent models: Support Vector Machines (SVM), Logistic Regression (LR) and Random Forest (RF). The texts were represented using embeddings produced by BART [22] and Doc2Vec [23].
Doc2Vec is an extension of the Word2Vec [24] model that allows the representation of documents as fixed-length vectors in a high-dimensional space. To do this, a neural network [ 25] is trained on a large corpus of text, where the network learns to predict words based on the surrounding context. As a result, documents with similar content or context will have similar vector representations, making it easier to identify relationships and patterns.
BART is a transformer model [
Its architecture is characterised by an encoder with an approach similar to the BERT (Bidirectional
Encoder Representation from Transformers) [
The choice of the best model was made on the basis of a GridSearch with a 10-split StratifiedKFold choosing its F1 as the best metric. The hyperparameters of the best models composing the ensemble with Doc2Vec-based text representation were as follows: • Doc2Vec: “vector_size”: 100, “window”: 5, “workers”: 4, “min_count”:2, “epochs”: 40. • SVM: “C”: 1,4, “kernel”: “rbf”. • LR: “C”: 0.0012, “class_weight”: “balanced”, “penalty”: “l2”, “solver”: “liblinear”.
• Random Forest: “max_depth”: 10, “min_samples_split”: 5, “n_estimators”: 120.
On the other hand, the hyperparameters chosen in the ensemble with text representation based on BART were as follows: • SVM: “C”: 1,8, “degree”: 4, “kernel”: “poly”. • LR: “C”: 0.0012, “class_weight”: “balanced”, “penalty”: “l2”, “solver”: “liblinear”.
• Random Forest: “max_depth”: 11, “min_samples_split”: 2, “n_estimators”: 140.
Two submissions were made for this tasks, being run 0 the one for the model with text representation based on Doc2Vec and run 1 for the one with BART embeddings.
The objective of this task was to estimate the levels of features associated with eating disorders based on user posts. This task is a continuation of the eforts made in 2022 and 2023. Participants were tasked with analyzing these postings to fill out a standard eating disorder questionnaire for each user. The questionnaire used in this task is the Eating Disorder Examination Questionnaire (EDE-Q) [27], specifically focusing on questions 1-12 and 19-28. The EDE-Q is a 28-item self-reported questionnaire adapted from the semi-structured Eating Disorder Examination (EDE) interview [28]. It is designed to assess the range and severity of features associated with eating disorders.
We utilized BERT (Bidirectional Encoder Representations from Transformers) to generate word embeddings for our text data, which were subsequently fed into a neural network for classification. Our neural network, was designed to process these embeddings and classify the text into one of 22 specified categories (possible answers to the eating disorder test). Its architecture consisted of an input layer taking the 768-dimensional BERT embeddings, followed by a fully connected layer mapping this to a 400-dimensional space, with a ReLU activation function. A dropout layer with a rate of 0.5 was used to prevent overfitting, followed by another fully connected layer reducing the dimensions to 200, again with ReLU activation. The final output layer mapped the 200-dimensional input to the 22 output classes. The model was optimized with cross-entropy loss and the Adam optimizer.
The detailed explanation for the metrics utilized can be found at the erisk 2024 overview paper in [
In task number 1, there were 9 teams in total. Being the team named NUS-IDS the one that achieved the best results. Most of the teams performed at least 3 diferent runs (submissions).
Team APB-UC3M APB-UC3M APB-UC3M APB-UC3M APB-UC3M
NUS-IDS
Run APB-UC3M_all-MiniLM-L6-v2
APB-UC3M_all-MiniLM-L12-v2 APB-UC3M_sentsim-all-mpnet-base-v2
APB-UC3M_ensemble APB-UC3M_classifier_roberta-base-go_emotions
Config 5
As can be seen in Tables 1 and 2, the semantic similarity models have been the ones that have obtained the best metrics, far above the classifier model RoBERTa.
Both unanimously and by majority, the semantic similarity models have achieved the second best metrics in Average Precision, R-PREC, NDCG and P@10. Even the model all-MiniLM-L12-v2 obtained the best metric of P@10 in majority labelling.
This indicates that semantic similarity models have been able to generalise much better than the classifier model RoBERTa with texts never seen before. In addition, a better metric on the precision in 10 (precision on the first 10 ranked items of each symptom) indicates that the method used to calculate the ranking of the sentence classification has been quite successful.
As for the classifier model, it has been shown that it has not been able to generalise as well as it should. A solution to this problem could be to train it for a longer time with a larger collection of sentences in order to try to improve its generalisation.
Finally, the ensemble model did not perform as expected. Its score was much lower than that of the models it incorporated. One of the reasons could be the decision to eliminate the multi-labelled sentences, we only kept the first occurrence of the sentence and its label.
In task number 2, there were 10 teams in total. Being the team named NLP-UNED the one that achieved the best results. Most of the teams performed 5 diferent runs (submissions).
team BioNLP-IISERB GVIS Riewe-Perla UNSL UMUTeam COS-470-Team-2 ELiRF-UPV NLP-UNED SINAI APB-UC3M #runs
Table 3 shows the time eficiency data for each of the models of each team after processing the total number of available posts (2,001). One reason for the longer elapsed time compared to other systems is the inability of the Doc2Vec-based text rendering model to use NVIDIA CUDA (Compute Unified Device Architecture) [29]. CUDA is a parallel computing platform and application programming interface model created by NVIDIA that allows developers to use NVIDIA GPUs (graphics processing units) for general-purpose processing such as training artificial intelligence models.
In Table 4, we can observe the results obtained for the decision-based evaluation (how accurate were the classifications into anorexics and non-anorexics) and the ranking-based evaluation (how accurate was the numerical value associated with each post based on its relevance). The model with text representation based on Doc2Vec although far behind the other teams, was able to rank the sentences more accurately than the text model based on BART [22]. However, it is also observed that the model performs worse on the speed-based metrics (latencyTP and speed).
Team APB-UC3M APB-UC3M NLP-UNED BioNLP-IISERB Riewe-Perla
P T y c n e t a l 9.00 2.00 14.00 4.00 6.00 1 F d e t h g i e w y c n e t a l 0.28 0.26 0.75 0.66 0.60 Team APB-UC3M APB-UC3M UNSL NLP-UNED NLP-UNED
There are several explanations why the presented models have not been able to obtain similar results to that of the other teams. In terms of speed, apart from the impossibility of using CUDA in the Doc2Vecbased model, there has also been a shortcoming in terms of the equipment used to communicate with the server. In other words, during the evaluation phase, there was no suficiently powerful equipment available that could accelerate the models.
On the other hand, regarding the accuracy of the models, a possible solution could be to change the type of ensemble used so that it is not carried out by majority voting, but rather specific weights are applied to each of the models that make up the [30], thus giving more relevance to the models that obtained the best metrics individually (e.g.: SVM). It should also be noted that due to a failure in the local machine used to communicate with the server, some messages were lost that could have been key in making decisions.
In task number 3, there were 5 teams in total. Being the team named SCaLAR-NITK the one that achieved the best results. Most of the teams performed at least 2 diferent runs (submissions).
Results depicted in table 6 show our team, APB-UC3M, achieved a MAE of 2.003, which is slightly higher than some of the top-performing teams such as SCaLAR-NITK, whose best run had a MAE of 1.874. Our MZOE was 0.869, indicating that our model had a moderate number of exact prediction matches, while our MAEmacro was 2.142. Our GED score of 2.647 suggests that our model’s predicted sequences were relatively close to the ground truth sequences.
In terms of subscale scores, our Restraint Score (RS) was 2.253, Eating Concern Score (ECS) was 1.884, Shape Concern Score (SCS) was 2.101, and Weight Concern Score (WCS) was 1.823. These results show that our model performed consistently across diferent subscales, although there is room for improvement, particularly when compared to the leading team’s scores.
team baseline baseline baseline APB-UC3M RELAI SCaLAR-NITK SCaLAR-NITK SCaLAR-NITK SCaLAR-NITK SCaLAR-NITK
The baseline results, especially the "all 6s" run, had a surprisingly strong performance, with a MAE of 1.937, which is close to our own. This indicates that a simplistic approach can still achieve competitive results, underscoring the complexity of improving upon simple heuristics in this task.
Overall, our model demonstrated a consistent performance but did not achieve the top results. The reason behind it could be due to the simplicity of the neuronal network used for the prediction.
We participated in all three tasks of the eRisk 2024 shared tasks [
Our models demonstrated a reasonable ability to identify early signs of depression from social media posts (task 1). The performance metrics showed that while the models were efective to an extent, there is still a gap between our approach and the desired level of accuracy. The challenge lies in capturing the subtle and varied ways depression symptoms can manifest in online behavior. One future approach we think of could be the improvement of the data quality. Expanding the dataset to include a wider variety of social media platforms and types of user interactions by including data augmentation techniques [31] and other social media datasets would help create more comprehensive models capable of generalizing across diferent contexts and user behaviors.
The models developed for detecting anorexia achieved poor results (task 2), revealing the need for more experimentation in this specific topic. Similar to Task 1, the variability in how individuals express anorexic symptoms online poses a significant challenge. The nuanced language and diverse expressions of anorexia necessitate more sophisticated models that can understand context and subtext better. To improve our results across the task, we propose the incorporation of sequential models such as GRU (Gated Recurrent Units) [32] or LSTM (Long Short-Term Memory) [33] which could potentially enhance the performance of our models by better capturing temporal dependencies and the sequential nature of user posts.
For task 3, our results were promising, particularly when compared to baseline models. Our approach, which involved the use of BERT embeddings and a neural network, provided a deeper understanding of user behavior, yet there is room for enhancing the precision and recall of our predictions. Future improvements could focus on enhancing the model’s ability to capture subtle nuances in user posts that are indicative of eating disorders, possibly through more advanced embedding techniques or more sophisticated neural network architectures.
This work was supported by ACCESS2MEET project (PID2020-116527RB-I0) supported by MCIN AEI/10.13039/501100011033/. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019).
URL: https://doi.org/10.48550/arXiv.1907.11692. [20] D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, S. Ravi, Goemotions: A dataset of ifne-grained emotions, 2020. arXiv:2005.00547. [21] A. N. Tarekegn, M. Ullah, F. A. Cheikh, Deep learning for multi-label learning: A comprehensive survey, 2024. arXiv:2401.16549. [22] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, CoRR abs/1910.13461 (2019). URL: http://arxiv.org/abs/1910.13461. arXiv:1910.13461. [23] Q. Le, T. Mikolov, Distributed representations of sentences and documents, in: International conference on machine learning, PMLR, 2014, pp. 1188–1196. [24] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient estimation of word representations in vector space, 2013. arXiv:1301.3781. [25] J. Schmidhuber, Deep learning in neural networks: An overview, Neural networks 61 (2015) 85–117. [26] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training (2018). [27] R. Murphy, S. Straebler, Z. Cooper, C. G. Fairburn, Cognitive behavioral therapy for eating disorders, Psychiatric Clinics of North America 33 (2010) 611–627. URL: https://doi.org/10.1016/j. psc.2010.04.004, cognitive Behavioral Therapy. [28] C. G. Fairburn, G. T. Wilson, K. Schleimer, Binge eating: Nature, assessment, and treatment,
Guilford Press New York, 1993. [29] R. S. Dehal, C. Munjal, A. A. Ansari, A. S. Kushwaha, Gpu computing revolution: Cuda, in: 2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), 2018, pp. 197–201. doi:10.1109/ICACCCN.2018.8748495. [30] I. D. Mienye, Y. Sun, A survey of ensemble learning: Concepts, algorithms, applications, and prospects, IEEE Access 10 (2022) 99129–99149. doi:10.1109/ACCESS.2022.3207287. [31] L. Perez, J. Wang, The efectiveness of data augmentation in image classification using deep learning, CoRR abs/1712.04621 (2017). URL: http://arxiv.org/abs/1712.04621. arXiv:1712.04621. [32] R. Dey, F. M. Salem, Gate-variants of gated recurrent unit (GRU) neural networks, CoRR abs/1701.05923 (2017). URL: http://arxiv.org/abs/1701.05923. arXiv:1701.05923. [33] R. C. Staudemeyer, E. R. Morris, Understanding LSTM - a tutorial into long short-term memory recurrent neural networks, CoRR abs/1909.09586 (2019). URL: http://arxiv.org/abs/1909.09586. arXiv:1909.09586.