This paper describes the participation of the SINAI team in the eRisk@CLEF lab. Specifically, two of the proposed tasks have been addressed: i) Task 1 on the search for symptoms of depression, and ii) Task 2 on the early detection of signs of anorexia. The approach presented in Task 1 is based on the use of a two-step detection approach using a transformer-based model, while the approach for Task 2 is based on calculating perplexity using two transformer-based models trained with causal language modelling: one on positive user data and the other on negative user data. In Task 1, our team has been ranked in the 7th position out of a total of 9 participating teams, with a 0.562 score of precision at 10 in the majority ranking. In Task 2, our team placed the 6th out of a total of 10 participating teams in the F1 score and we reached the best overall values after only two writings.
In this context, our primary objectives extend beyond merely creating high-performing systems. We
aim to comprehend the most efective methods and approaches applicable to similar scenarios. Rather
than assembling numerous features and systems into an ensemble of predictors solely for achieving
top rankings, our focus lies in identifying the best approaches aligned with our objectives: (
This work presents the participation of our research group, the SINAI1 team in Task 1: Search for symptoms of depression and Task 2: Early Detection of Signs of Anorexia. The rest of the paper is organized as follows: sections 2 and 3 describe in detail our participation in task 1 and task 2, respectively. Each of them is divided into subsections in which, first, we introduce what the task consists of, the data provided and the evaluation measures used. Secondly, the system developed and the methodology used are presented. Thirdly, the experimental setup is detailed. Subsequently, the results obtained and a discussion on them are presented. Finally, Section 4 shows the conclusions obtained after the participation in the eRisk lab and the perspectives for future work.
This task focuses on searching signals of depression in users from social media. In this case, the systems had to read the sentences, which are embedded within the preceding and following context, and determine their relevance to each of the 21 symptoms in the BDI Questionnaire. Additionally, each symptom has its own ranking, which will correspond to the 1000 most relevant for each distinct symptom.
The provided training data is annotated at sentence level regarding its relevance to symptom query. Each sentence is annotated by three annotators, and the dataset provides a majority vote on ground truth and the unanimous or aggregated label. The test data is composed of 551,311 users with a total of 15,542,200 sentences which have an average number of words of 17.98.
We have explored two diferent systems for this task. One submission is based on fine-tuning with data augmentation and uses a two-step detection approach (see Section 2.2.1) and the other is based on prompting to GPT-3 (see Section 2.2.2).
The SINAI_DR_majority_daug run trained a DistilRoBERTa base model2 using the training data provided
in its majority vote form and with data augmentation using the BDI-Sen dataset [7].
Data We used BDI-Sen which is a symptom-based dataset with relevant sentences that trace the
presence of clinical symptoms, for the labelling they used three expert annotators consisting of a
psychologist, a speech therapist, and a PhD student. This dataset contains a total of 18,510 sentences.
The data in the BDI-Sen dataset is labelled from 0 to 4, where 4 represents the control class, while values
from 0 to 3 indicate severity, with 0 being the mildest and 3 the most severe. We have modified these
numbers as follows: we consider values 0, ñ, and 4 as unrelated (0), and values 1, 2, and 3 as related (
After training the model, we computed the probability for each user to sufer each BDI Questionnaire symptom. We then ranked the scores from highest to lowest and selected the top 1,000 users with the highest probability for each symptom. Following this, we reapplied the previously trained model, 1https://sinai.ujaen.es/ 2https://huggingface.co/distilbert/distilroberta-base focusing only on users with high scores. Then we selected the top 1,000 sentences based on their ratings. By filtering at the user level first and then at the sentence level, this two-step procedure allowed us to refine the selection further and ensure the most relevant content was found for each symptom. We trained the model to perform a search for the 21 symptoms using multi-label regression on a compute node equipped with a single NVIDIA-A100 GPU with 40 GB of memory. To achieve this, we searched the best hyper-parameters with Optuna [8]. We trained the model over 10 epochs, set the learning rate to 1e-05 and a weight decay of 0.007, we also organized the training process into batches of 16 and used the AdamW optimizer. 2.2.2. Prompting GPT-3.5 The second proposed approach is based on Large Language Models (LLM) and based on the search for the 8 symptoms evaluated in the PHQ-8 (Patient Health Questionnaire-8) [9]: (a) Feeling down, sad, or hopeless. (b) Not enjoying things you used to like. (c) Trouble sleeping or sleeping too much. (d) Feeling tired all the time. (e) Changes in appetite, either eating too much or too little. (f) Feeling worthless or like a failure. (g) Dificulty concentrating on things. (h) Moving slowly or feeling restless and fidgety. This approach provides a solid foundation, as these symptoms can be indicative of underlying emotional conditions. By targeting these symptoms, a starting point is established to explore the 21 symptoms of the BDI, as many of them may be interconnected. Since the PHQ-8 only assesses 8 symptoms while the BDI assesses 21, with the assistance of a physician, we conducted a symptom mapping, which can be observed in Table 2.
The approach consisted of two steps: The first focused on generating 15 synthetic phrases with the generative model GPT-3.5 [10] for each of the 8 symptoms evaluated in the PHQ-8. The prompts used to generate these phrases were designed with the assistance of medical staf, aiming for as much detail as possible to provide GPT-3.5 with accurate and comprehensive information, thus simulating a real interview scenario. Additionally, the prompts were tailored to obtain phrases related specifically to Feeling bad about yourself — or that you are a failure or have let yourself or your family down Trouble concentrating on things, such as reading the newspaper Indecisiveness, Concentration Difior watching television culty Moving or speaking so slowly that other people could have noticed? Or the opposite — being so fidgety or restless that you have been moving around a lot more than usual depressive signs. The second step aimed to find relevant phrases in the user’s history. We utilized the synthetic phrases generated by GPT-3.5 as anchors to search for similar phrases in the user’s history using semantic similarity. This strategy combines the power of artificial intelligence with the richness of user contextual information.
Once we obtained phrases per symptom, we computed a phrase vector using a pre-trained transformer from SentenceTransformer [11]. Once we have obtained representations for synthetic phrases, we proceed with a semantic search within the user histories. The ranking of phrases was determined as follows: for each user history phrase, we computed the cosine similarity with each of the symptomrelated phrases generated by GPT-3.5, then summed this similarity across the 15 phrases for each symptom, and obtained a score. This score was assigned to each phrase in the history, thus determining the relevance of each phrase for each symptom.
The organizers conducted a labelling process involving three annotators who manually selected the relevant writings. This process led to the creation of two evaluation schemes: majority voting (requiring agreement from 2 out of 3 annotators) and unanimity (requiring agreement from all 3 annotators).
It is clear from Table 3 and Table 4 that the system obtains better results in the ranking with a majority vote than in the ranking with a unanimous vote. While AP, R-Precision, and NDCG measurements show only slight variations at P@10, a significantly larger shift is apparent.
We argue that this phenomenon stems from model overfitting, which occurs when there is a disparity in the representation of related and unrelated instances during the training phase.
This task focuses on early risk detection of signs of anorexia by processing posts from social media in strict order of publication. The participant systems had to read the posts (from several subjects) in the order in which they were created, process them and generate a response in order to get the next posts.
The task is faced from two diferent perspectives: as a binary decision problem, and as a ranking (regression) decision problem. As a binary decision problem, posts have to be labelled as positive (label 1, i.e. risk detected) or negative (label 0, no risk detected). The earlier the system detects an addiction, the better, as it is reflected with the ERDE and metrics proposed by the organizers and used to evaluate the systems, along with the well-known precision, recall and F1 scores. As a ranking decision problem, instead of assigning 0 or 1 labels, a score of the estimation of the risk of sufering such a disorder is computed. Diferent metrics as the ones used for information retrieval are considered to evaluate this second view of the task (P@10 or NDCG, among others).
The test data is composed of 366,886 posts by 784 diferent subjects in total: 92 subjects labelled as positive in risk detection of signs of anorexia, and 692 subjects not labelled as at risk.
We submitted a system based on the calculation of perplexity. For that, we trained two transformerbased models with causal language modelling, one with data from positive users and the other with data from negative users.
Nonetheless, though it was not submitted, another approach was performed, consisting of creating and employing emotion vectors, relying on several English lexicons, and training various classical Machine Learning (ML) algorithms with 80% of the provided observations of the dataset.
For the training and experiments conducted, only the data provided by the organisers consisting of data from the 2018 and 2019 editions was used. In total, the training set has 1,245 subjects, with 134 subjects labelled as positive for sufering the risk of anorexia and 1,111 subjects labelled as negative. Furthermore, the total amount of training data they give amounts to 799,186 writings. For both approaches, we split the data provided by organizers into training (996 subjects of whom 103 are labelled as positive and a total of 639,824 texts) and valid (249 subjects of whom 31 are labelled as positive and a total of 159,362 texts) datasets. From now on, whenever we refer to training data, we will refer to this subset.
Data pre-processing Each entry is pre-processed by removing URLs, line breaks, text only containing blank spaces or empty strings and texts whose number of tokens was lower or equal to 10 (this high threshold was chosen due to the large size of the texts of the dataset). Though for the following system used in this task titles will not be considered, in this case, if a record does not have text, the title will be used as such, avoiding a large number of empty entries to be removed. Table 5 shows the size of the datasets used Methodology First of all, it is necessary to introduce the employed algorithms for this approach. We have used the default hyperparameter values of the methods used from the scikit-learn library [12]. In particular, the ML methods used were: • Logistic regression with a maximum of 1000 iterations. • RandomForest using 20 decision trees and seed with value 45. • MLP with a maximum of 1000 iterations. • SVM (default). • Decision tree (default). • KNN (default).
• SGD with a limit of 1000 iterations, stopping criteria with value 1 × 10− 4 and 45 as a seed value.
In order to conduct the experiment, it is necessary to obtain valid English emotion lexicons. The employed lexicons in our experiments are NRC [13], ESN [14], DPM (DepecheMood) [15], DPM with normalized term frequency, and the intersection of words appearing in DPM and NRC using DPM emotion values. Each word has a value for each one of the considered emotions, which are Ekman [16] basic ones (anger, disgust, joy, sadness, surprise and fear) plus "don’t care" for DPM and finally, anticipation, negative and positive polarity and trust for NRC.
Regarding the training of the algorithms, the messages are grouped by user. Therefore, the generated array of each user will contain information about all the messages sent by the person. In the test phase, messages are not grouped all together by user and the arrays are re-generated each round, one message per user each lap.
To build the emotion vectors, the text for each observation will be tokenized by word and every time a word in the text also belongs to the lexicon currently being worked with, it is registered alongside its emotion values (if the word appears more than once, the values for that word are registered as many times as occurrences in the text there are). Therefore, the final result for each message’s feature vector will be a list of 4 sublists (maximum, minimum, average and standard deviation of the values of each gathered emotion) each one of those containing many positions equal to the number of "measured" emotions a lexicon has. Once the list of lists is created, the resulting data structure for each round is generated getting the maximum, minimum, average and standard deviation between the newly created array and the one from the previous round, being this the feature we will use in our classification experiments.
Results and discussion The results were obtained using the created validation set and measured utilizing common ML evaluation metrics. Before presenting the table, the runs that populate are presented: • Run 0: normalized DPM with KNN as best performing algorithm. • Run 1: ESN with KNN as best performing algorithm. • Run 2: intersection of words in NRC and DPM with normalized DPM emotion values being KNN the best performing algorithm. • Run 3: intersection of words in NRC and DPM with DPM emotion values being Random forest the best performing algorithm. • Run 4: intersection of words in NRC and DPM with DPM emotion values being stochastic gradient descent the best performing algorithm.
The runs were launched in an ordinary computer (i7-10700 CPU and 32 GB of memory). The 5 best results ordered by Macro-F1 are shown in Table 6.
Regarding the obtained score, DPM (DepecheMood) seems the most useful lexicon out of all those tested. Indeed, combined with NRC terms, it populates 4 positions within the top 5 best runs. Also, the K-nearest-neighbors algorithm provides the 3 best results regarding the Macro-F1 metric. What is more, it is worth mentioning the proficient results in ERDE5 and ERDE30 being this approach capable of detecting true positives acceptably fast. A future improvement involves a hyperparameter search to ifnd the best possible tuning for the employed algorithms.
Data pre-processing We apply a pre-processing of the training dataset which consists of only keeping the ‘text’ field from each writing user. Thus, we ignore titles of publications that do not have content in the text field because we suppose that if a writing does not have text then it does not contain valuable information. Again URLs, non-alphanumeric characters and blank spaces are removed. Moreover, we removed empty writings (when a text contains only punctuation, spaces or ‘[removed]’ target) and writings whose text contains less than four tokens. Once texts have been cleaned, we split the training dataset based on the label values. So we had two diferent datasets: one with texts coming from subjects labelled as positive (label = 1) and another with texts coming from subjects labelled as negative (label = 0). Table 7 shows some statistics about the datasets developed to train the systems. Of the 639,824 texts we had before the processing was applied in the training dataset we developed, we now have 411,498 texts. Methodology Once we have the data prepared, we proceed to train the models. We selected the GPT-23 model because it is freely available for download from HuggingFace4 and is straightforward 3https://huggingface.co/openai-community/gpt2 4https://huggingface.co/ to use and train. Additionally, GPT-2 performs well with the English language, which is the primary language of our dataset and can also handle other languages. We trained two models: • positive_model: using texts from subjects labelled as positives • negative_model: using texts from subjects labelled as negatives
Our main goal is to see which model finds a given text more familiar. We do this by calculating the perplexity for each text using both models. The model with the lower perplexity value indicates the text is more similar to its training data.
Next, we tested the models with our validation dataset. We ran experiments to find a good threshold for making predictions because the two models were trained on diferent amounts of data, making their perplexity values not directly comparable. We calculated the diference between the two perplexity values and checked if this diference was less than a certain threshold. We tested thresholds from 0 to 1,000 and then focused on 0 to 100 to find the best one. Then, if the perplexity from the positive_model is lower than that from the negative_model, the subject is classified as positive. However, a subject can also be classified as positive if the diference between the perplexities is less than a specified threshold, even if the positive_model perplexity is higher. This formula 1 determines whether a subject is labelled positive.
ppl_pos < ppl_neg or dif < threshold
(
dif : Diference between ppl_pos and ppl_neg threshold : Threshold for accepting the diference
Additionally, we also explored the role of context in our experiments: (
We applied the same pre-processing to each text as we did to the training data. We labelled the subject as negative if a text was too short or invalid. The scores we report are based on the perplexity values from the positive_model.
Experiments For the testing phase, the experiment set up was: • Run 0: refers to the text provided for the subject at each round.
(+, ) < (− , ) →− • Run 1: refers to the text provided for the subject at each round.
(+, ) < (− , ) OR (+, ) − (− , ) < 45 →− • Run 2: refers to the text provided for the subject at each round.
(+, ) < (− , ) OR (+, ) − (− , ) < 60 →− • Run 3: refers to the concatenation of texts provided for the subject.
(+, ) < (− , ) →− • Run 4: refers to the concatenation of texts provided for the subject.
(+, ) < (− , ) OR (+, ) − (− , ) < 45 →−
Both the model trained with positive texts and the model trained with negative texts are trained with 3 epochs, 0.01 weight decay, 1e-05 learning rate and batch size equal to 8. A maximum length of 512 tokens per document was set and the optimizer used was AdamW. All experiments in the pre-evaluation and evaluation phases were run on a compute node equipped with a single Tesla-V100 GPU with 32 GB of memory.
From the reported results provided by the organizers, we have extracted our scores, which are shown in Tables 8 and 9. 0.00 0.00 0.00 0.00
The first thing that is observable when looking at Table 8 is that there is no diference between runs 0, 1 and 2 (without contexts) and runs 3 and 4 (with contexts). So, the threshold estimated in the training phase is not valid for the test phase. We may think the test data is too far from the data used in training or that we have over-adjusted the thresholds we have set. Therefore the only observable diferences that can be discussed lead us to use context (concatenation of messages) or not to use context (taking into account only the current text).
The approaches based on contexts locate all positive users in the test set, the rest come close to achieving this goal. In addition, the prediction that only takes into account the current text is more accurate than with context but needs more messages in the detection of true-positive subjects.
Table 9 shows us that the scores sent are not valid values for this task.
Deep pre-processing of the training data should have been done to obtain better results, as some texts labelled as positive (because they came from a subject labelled as positive) were very similar to texts labelled as negative. For example, the positive_model was trained with the positively labelled sentence "They say dogs look like their owners....", which tokenised with the GPT-2 model has a very high cosine similarity to the negatively labelled sentence "Dogs are so much more cool than people." used to train the negative_model. We selected GPT-2 due to its established performance metrics and accessibility. However, we acknowledge that the use of a more recent and advanced model, such as GPT-3 or GPT-4, could potentially enhance performance and yield more impressive results. Future iterations of our research will consider integrating these more advanced models to improve accuracy and overall outcomes. Re-labelling of the messages, as for example was done in Fabregat et al. [17] in another edition of a similar task, as well as a search for hyperparameters and testing with other LLMs is also proposed as future work.
This paper describes our participation as the SINAI team in Task 1 and Task 2 of the eRisk@CLEF 2024 edition. The former is the continuation of the first edition in 2023 and consists of ranking sentences from a collection of user writings according to their relevance to a depression symptom, while the latter is the continuation of the 2018 and 2019 editions and consists of sequentially processing pieces of evidence and detect early traces of anorexia as soon as possible.
For Task 1, we have developed two diferent approaches: one is based on a transformer-model training with a two-step ranking, and the other is based on prompting GPT-3. As can be seen in the results, the pre-trained model with data augmentation works better than an LLM. In the two-step approach, the results are not entirely satisfactory, possibly owing to the model overfitting that could have been conducted with the dataset. For Task 2, we developed two diferent approaches, the emotion vectors approach and perplexity approach. We submitted the last one and achieved the best scores for recall and speed metrics. In future work, we plan to perform error analysis for both tasks to identify the main weaknesses of our systems and apply another preprocessing of the data.
This work has been partially supported by projects CONSENSO (PID2021-122263OB-C21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government. [8] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. [9] K. Kroenke, T. W. Strine, R. L. Spitzer, J. B. Williams, J. T. Berry, A. H. Mokdad, The phq-8 as a measure of current depression in the general population, Journal of afective disorders 114 (2009) 163–173. [10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [11] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019. URL: https://arxiv.org/abs/1908.10084. [12] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830. [13] S. M. Mohammad, P. D. Turney, Nrc emotion lexicon, National Research Council, Canada 2 (2013) 234. [14] S. Poria, A. Gelbukh, E. Cambria, A. Hussain, G.-B. Huang, Emosenticspace: A novel framework for afective common-sense reasoning, Knowledge-Based Systems 69 (2014) 108–123. [15] J. Staiano, M. Guerini, Depechemood: a lexicon for emotion analysis from crowd-annotated news, arXiv preprint arXiv:1405.1605 (2014). [16] P. Ekman, An argument for basic emotions, Cognition & emotion 6 (1992) 169–200. [17] H. Fabregat, A. Duque, L. Araujo, J. Martínez-Romo, Uned-nlp at erisk 2022: Analyzing gambling disorders in social media using approximate nearest neighbors., in: CLEF (Working Notes), 2022, pp. 894–904.