This paper presents the participation of the Vicomtech NLP team in the MentalRiskES shared task about the early detection of mental disorders in Spanish comments from Telegram users. We participate in two tasks: Task 1a, related to eating disorders, and Task 2a related to depression. For both tasks we propose a set of approaches based on supervised text classifiers using Transformers. We prioritise our experimentation in building low resource demand systems with the minimum low carbon footprint. With those highlighted features, our systems are developed to detect disorders as early as possible, involving an initial phase that automatically projects stream labels at message level, since not all messages contained in a stream are equally representative of the stream class. We obtain the best ERDE5 result in depression detection (0.27) and second-best in eating disorders (0.17).
requiring participants to identify potential risks as early as possible within a continuous stream of data. That is, participants were presented with user messages sequentially and required to emit a prediction at each step, thus emulating the dynamics of real-time analysis. Accordingly, systems were evaluated both for their risk detection correctness and speed.
This paper describes the approach developed by the Vicomtech NLP team to address two tasks of the MentalRiskES challenge: • Binary classification of eating disorders (Task 1a) : the goal of this task was to detect as soon as possible whether users sufer from anorexia or bulimia. • Binary classification of depression (Task 2a) : the goal of this task was to detect as soon as possible whether users sufer from depression.
Our team participated in these tasks with supervised Transformer-based [
The remainder of the paper is structured as follows: Section 2 provides a brief overview of the related work; Section 3 introduces the challenge data; Section 4 presents our approach to the tasks, including a detailed explanation of the aforementioned data preprocessing methodology; Section 5 reports and analyses the obtained results, both in the development phase and in the oficial evaluation; finally, Section 6 concludes the paper by summarising the key findings and suggesting avenues for future research.
Many eforts have been made to detect diferent mental disorders through written messages
on the internet, based on a text classification problem [
The solutions that have been developed by the CLEF participants cover a wide range of
diferent approaches. For example, the UNSL team at eRisk 2022 [
A diferent solution, which inspired our work, was the on proposed by the UNED-NLP team
[
The MentalRiskES dataset is a novel collection of content sourced from public groups on the
messaging platform Telegram. Each training example consists of a sequence of messages or
stream from one Telegram user, and the corresponding mental disorder risk label. Specifically, the
streams were labelled as suffer or control by 10 annotators, suffer being the positive class; the
ifnal label for each stream is suffer if 5 or more annotators voted thus, and control otherwise.
Streams vary in length, containing typically 20 to 40 messages, although many of them consist
of up to 100 messages. We refer the reader to the challenge overview article [
The organisers provided 175 training examples and 10 trial examples for each task. Guided by early experiment results and data analysis, we preprocessed these datasets as follows: 1. Resample the available data in order to obtain two development sets, with 15% and 10% of the available examples each—one for development purposes proper (Dev), and the other for testing purposes (Dev), respectively. 2. Filter out streams with 4, 5 or 6 votes from the resulting training set, as such controversial examples could introduce noise in the learning process. Consequently, we discarded 18 and 27 examples from Task 1 and Task 2 data respectively. 3. Detect and erase verbalised forms of emojis from the messages. As part of the data curation process, some emojis had apparently been transformed to their verbalised form (see examples in Table 7, Appendix A), which rendered some messages ungrammatical or even nonsensical. This preprocessing step was only applied to depression data (Task 2), seeing as the problem seemed to occur more often in that case.
No additional data was used other than that provided by the organisers. Thus, the final example distribution is as shown in Table 1, which also includes the oficial test. All references to train and/or development data should be hereafter interpreted as referring to these samples.
As can be seen, MentalRiskES is challenging in many respects, namely, the reduced size of the dataset and its imbalanced class distribution, but also the subjectivity of the problem, as evidenced by the spread-out annotator vote distribution.
While the provided challenge data comes annotated at stream level, early risk detection systems must be able to emit predictions at each step, that is, by having seen only partial stream sequences. Intuitively, then, a classifier trained with complete streams could have a hard time emitting correct predictions for partial streams, given that not all messages in a stream are bound to be equally representative of the stream category (e.g., some messages in positive streams could deal with topics unrelated to the target mental disorders).
In accordance with this premise, our approach to MentalRiskES has consisted of two distinct phases: first, automatically annotating streams at message level as suffer or control; second, exploiting the resulting silver corpus to train early detection systems. These phases are illustrated in Figure 1 and explained in detail in subsequent sections (4.1 and 4.2 respectively).
Both phases involve training BERT-like text classification models based on DiagTrast-Berto
[
The goal of this phase was to automatically detect the most representative messages of the suffer class in suffer training streams, and assign the control label to all other messages. That is, did not involve control streams, whose messages were directly assigned the control label. This phase corresponds to the leftmost box of Figure 1. In what follows, we introduce the key concepts of our proposal, namely, the average confidence Δ (ACΔ) and its threshold.
Train Dev
Message-Level Anotation 5 folds
Average confidence Δ
Average confidence Δ
Stream BERT classifier
Message-level annotation
Train
Dev x5 different annotation logics
Partial Stream
Sampling
Message BERT classifier
Our proposal builds on the intuition that, given an early risk classification model and an input message stream, if the confidence of the model for the suffer category is markedly higher with a particular message in the stream than without it, then it is sound to assume that the message must be representative of the suffer class.
Here, model confidence is defined as the result of applying a sigmoid function to the output logits of a BERT-like binary classification model, so that a value of 1 corresponds to a 100% suffer stream, and 0 to a 100% control stream. The confidence Δ of a message or messages is then the diference between the confidence of the model for the entire stream and the confidence of the model for the stream without said message(s), whereby a positive confidence Δ would imply that the message(s) contributed towards the suffer class. That is, the confidence Δ of a message is always relative to the stream it occurs in.
Figure 2 illustrates the confidence Δ scores of each message in a real stream of Task 1, eating disorders. The peaking messages are as follows (translations to English are given below each message; the whole stream can be consulted in Appendix A): (1) Ana Peso actual: 68 Peso meta: 50 Edad: 20 (9) Es posible vomitar las calorías?
Is it possible to vomit the calories? (15) Tips para bajar de peso en 3 días por favor urgente
Tips to lose weight in 3 days please urgent (19) Estoy tan gorda que hice reventar una falda Me siento muy mal
I’m so fat that I burst a skirt I feel so bad (23) Sabes que no tienes hambre y simplemente comes porque te sientes estresad@
You know you are not hungry and you just eat because you feel stressed (24) No puedes detenerte ni aunque pienses en qué vas a engordar y te verás horrible
You can’t stop even if you think you’re going to get fat and look awful
The confidence Δ scores of training set messages were computed in a k-fold validation fashion: the training dataset was divided into 5 folds with approximately the same number of streams; then, the messages of each held-out fold were processed with the confidence scores given by binary classification models trained on the other 4 folds. More specifically, we trained 5 models per fold with diferent seeds, for a total of 25 models, which allowed us to calculate an average confidence Δ (ACΔ) per message. The ACΔ of the development set messages (Dev) were obtained by averaging the scores of all the 25 models. Details about model architecture and training setup can be consulted in Section 4.3 and Appendix C.
Given a stream of messages and their ACΔ scores, we annotated messages as suffer if their score surpassed a given threshold, and control otherwise. In this work, we designed and tested multiple types of thresholds: Naive A message is considered suffer if its ACΔ is positive.
Unigram Local Maximum (LocMax1) A message is considered suffer if its ACΔ is greater than 20% of the maximum ACΔ of the stream.
N-gram Local Maximum (LocMax) A message is considered suffer if LocMax1 applies or if its dropping alongside consecutive messages (2 or 3) yields an ACΔ that is greater than 22% of the maximum ACΔ of the stream, in which case all the involved messages are tagged as suffer.
Unigram Global Maximum (GloMax1) A message is considered suffer if its ACΔ is greater than 20% of the average maximum ACΔ of all the streams.
N-gram Global Maximum (GloMax) A message is considered suffer if GloMax1 applies or if its dropping alongside consecutive messages (2 or 3) yields an ACΔ that is greater than 22% of the average maximum ACΔ of all the streams, in which case all the involved messages are tagged as suffer.
The ACΔ ratios (20% and 22%) we set empirically, along with a general minimum threshold of 0.004 (except for the naive approach).
Statistics about the resulting silver corpora (one per threshold strategy) can be found in Table 8 of Appendix B. Overall, less than 5% of the messages in each stream were labelled as suffer by the diferent annotation strategies, the first message labelled as suffer being on average in the third or fourth position. Appendix A includes a complete example of a stream annotated at message level.
In the second phase of the experimentation, we trained binary classification models that are able to recognise suffer cases of the target disorder seeing as few messages as possible. To that end, we sampled partial stream training and development sequences from suffer and control streams, leveraging our message-level annotations, in order to emulate the actual inference scenario of the challenge. This phase is represented in the rightmost box of Figure 1.
The goal of partial stream sampling (PSS) is to obtain sub-sequences or partial streams of each stream, with the label for a sub-sequence being suffer if any of the messages in it is suffer, and control otherwise. The process difers depending on the type of stream being sampled.
On the one hand, control streams only yield control samples. In this case, we extracted samples semi-randomly, assigning more likelihood to shorter stream lengths. In 10% and 5% of the cases, we sampled 2 and 3 sub-sequences respectively instead of just one.
On the other hand, suffer streams yield both suffer and control samples: • suffer samples extend from the beginning of the stream up to and including a suffer message. We extracted one sample per stream from the beginning of the stream up to the first suffer message. Additionally, we generated samples that extend up to the second or third suffer message in 20% of the streams, so as to maximise classifier recall. • control samples extend up to but excluding the first suffer message. These samples are meant to maximise classifier precision and were generated from 10% of the suffer streams. This process was applied once per stream, obtained what we will henceforth refer to as “1-time sample” or 1s datasets. By definition, 1s datasets only include content from the start of each stream. Furthermore, we applied the sampling method iteratively in a sliding-windows fashion (“exhaustive sampling” or ∀s), efectively obtaining more training and development samples.
These two sampling techniques, paired with the five message-level annotation versions (see Section 4.1) produced a total of ten dataset versions for early risk detection training, the sizes of which are reported in Table 9 (Appendix B).
In order to measure the impact of the diferent AC Δ thresholds and partial stream sampling methods, we trained and tested early risk detection models using the ten dataset versions (Table 9). Specifically, we trained 5 models per dataset with diferent random seeds. Then, their averaged results were studied to select the final models for submission to the challenge.
We also implemented baseline systems consisting of binary classification models trained directly on entire streams (i.e., on the kickof Train and Dev partitions described initially in Section 3). These baselines allowed us to measure the gains of our proposed method about automatically labelling streams at message level.
Details about model architecture and training setup can be consulted in Section 4.3 and Appendix C. Section 5 reports and analyses the results of all these models in our development test partition (Dev in Table 1), as well as the results in the oficial test set ( Test) of the models selected for submission.
All the classifiers implemented throughout our reported experiments consist of BERT-based binary classifiers. We followed the standard layer stack recipe: a BERT encoder, whose output for the token [CLS] is pooled and fed to a dropout layer, followed by a dense linear layer that produces the logits for the target categories—namely, suffer and control. During training, the cross-entropy loss is back-propagated to fine-tune the models. In inference, the final label for the given input is simply the most probable one.
Streams were fed to the encoder by joining all the messages through the special token [SEP] and passing this text through the corresponding BERT tokeniser. The cases where the resulting subword sequence surpassed the maximum allowed length of the encoder were handled diferently in each of the experimentation phases: • For message-level annotation models (Section 4.1.1), we simply truncated the stream to the maximum allowed length. Given the observation that suffer streams usually contain relevant information for the class already in initial messages, we considered this a good compromise between implementation simplicity and expected performance. • In the case of early risk detection models (Section 4.2.2), on the other hand, the most important message of the input sequence is by design the latest one—as the classifier receives messages from a live stream incrementally, it is the latest message that contains new information to consider. Thus, if necessary, the input sequences were truncated from the left, that is, discarding older content. Of note, the maximum input length was limited to 130 tokens during training and 512 in inference, as this setting produced better results in our preliminary experiments.
All the models are fine-tuned versions of DiagTrast-Berto [
This decision was also conditioned by concerns related to eficiency, as MentalRiskES organisers encouraged participants to implement systems with low resource demand and carbon footprint. For this reason, we did not consider experimenting with larger Transformer-based models, despite their potential superior performance.
Specific model hyperparameters and training setup can be consulted in Appendix C.
In this section, we will describe the results obtained in our development experiments and in the oficial challenge evaluation. It must be noted that despite our eforts to replicate the organisers’ evaluation methodology, we have not been able to obtain exactly the same metrics as those oficially reported on the test set. That is, the results obtained in our local experiments do not represent exactly the same metrics used in the final evaluation.
The results obtained by our models on the development test (Dev) are reported in Tables 2 and 3, for Task 1 (eating disorders) and Task 2 (depression) respectively. It must be noted that the reported results correspond in each case to the average of 5 models.
As can be seen, many of our models manage to surpass the simpler baselines trained on entire streams. With our proposed stream preprocessing method, we have managed to gain 11 and 15 F1-score points in Task 1a and Task 2a, respectively, despite the fact that the message-level datasets were labelled automatically, while the baselines are trained on gold labels.
Furthermore, it can be observed that models trained with the exhaustive datasets (∀s) managed to obtain better results than those of the 1-step datasets (1s) in almost all the cases. This observation applies both to binary classification metrics and, to a lesser extent, to early detection metrics. This is despite the fact that ∀s datasets contain information that was not sampled exclusively from the initial parts of the streams; on the other hand, it should be noted that ∀s datasets do contain many more training and development examples than 1s.
No such general remark can be made with respect to the impact of the diferent threshold heuristics. Although our models obtained the best results in both tasks with the GloMax1 threshold and ∀s dataset, the diferences are not particularly significant nor systematic. It is possible that a bigger test dataset is needed to measure the real impact of the diferent thresholds.
These results led us to submit the following models for oficial evaluation: Task 1a (eating disorders)
Task 2a (depression) • Run 0: ∀s + GloMax1 • Run 1: ∀s + GloMax • Run 2: ∀s + LocMax
• Run 0: Baseline • Run 1: 1s + GloMax1 • Run 2: ∀s + LocMax The oficial results of our 3 runs per task can be consulted in Tables 4 and 5 for Task 1a and 2a respectively. For benchmarking purposes, these tables also report the results of the organisers’ baselines and those of the best participants in terms of F1-score and/or F1 . Carbon footprint measurements have been compiled in Appendix D.
These results reveal that our systems are among the fastest of the participants, with some of the best trade-ofs between F1-score and speed of the entire task. We obtained the secondbest ERDE5 value in Task 1a, having only been slightly outperformed by the Roberta baseline (a model with double the parameters than ours). We also obtained the second-best F1 value. The best F1 value was achieved by the team CIMAT-NLP-GTO, who have a remarkably better F1-score but double our latency. On a minor note, it should be reported that we encountered a problem upon submitting the predictions of the first round, which afected our speed-related metrics.
The trends are similar for Task 2a, although the absolute metrics are overall worse than for Task 1a, in line with our development results. We hypothesise that this task may have been more challenging for all participants because the problem being modelled is more subjective or ambiguous, as the annotator votes in Table 1 suggest.
In general, our ERDE30 and F1-score values suggest that our systems are relatively premature in the classification. Other participants obtained better F1-score and ERDE 30 values than us by consuming more messages, which, in return, allowed them to be more accurate. However, most of our errors were committed in streams with a greater disagreement than average between the annotators, as we explain in the next section.
The false positive (FP) and false negative (FN) error counts on the Test are reported by task and run in Table 6. Run 0 (∀s + GloMax1) and Run 1 (∀s + GloMax) are quite balanced in Task 1a, but Run 2 (∀s + LocMax) makes twice FPs than FNs. This behaviour could be explained by the fact that LocMax is a laxer threshold than the former two. The error types are much more imbalanced in Task 2a, where our systems have made more than 50 FP errors each. As explained before, this task turned out to be markedly more challenging for all the task participants.
Interestingly, the errors committed by the diferent systems involve to a great extent the same set of streams. Having manually analysed them, we observed that many of the repetitive FP errors were triggered by messages that mentioned words or phrases related to eating disorders (example E1) or depression (E2), but that do not necessarily imply that the user themselves is at risk of sufering those. In contrast, the FN errors generally involve streams that clearly convey mental health risk but were outright missed by the systems (e.g., E3 and E4).
E1) Que opinan sobre hacer ayuno para bajar de peso
What do you think about fasting to lose weight E2) Y estoy aquí porqué mi Esposa fue diagnosticada con depresión y ansiedad
And I’m here because my Wife was diagnosed with depression and anxiety E3) Yo me siento horrible con el cuerpo que tengo [...] intenté bomitar pero nunca pude
I feel horrible with the body I have [...] I tried to vomit but I never could E4) Solo quiero ayudar y que los demás estén más felices [...] ya que yo no puedo [...]
I just want to help and make others happier [...] since I can’t [...]
Given the subjective nature of the task, we also analysed the annotator vote distribution specifically on the instances that induced the errors on our systems. Table 6 includes the average number of votes received by stream: in the case of FP streams (that is, true control streams), the annotator agreement is better as the vote average gets closer to 0; conversely, the annotator agreement for FN streams (that is, true suffer streams) is better as the vote average gets closer to 10. As we compare these values to the total average votes of the dataset (the bottom row of the table), we observe that the annotator agreement is somewhat worse, overall, for the streams that induced errors in our systems. That is, this results suggest that our systems committed errors on instances that present more dificulties than average even to human annotators.
This article described the participation of the Vicomtech NLP team in the MentalRiskES shared task. Specirfically, we have tackled the binary classification task for eating disorders and depression disorders, proposing Trasnformer-based systems that did not involve training with any external data nor the use of larger language models. Our proposal includes a novel messagelevel annotation algorithm based on the variation of confidence when dropping message from streams, and the training of DiagTrast-Berto models over these message-level annotated datasets.
Our systems managed to obtain competitive metrics in comparison to the results by other participants, having achieved the second-best F1 and ERDE5 values in Task 1a, and the best ERDE5 and the third-best F1 in Task 2a These metrics measure how fast models recognise the desired pattern. Our systems might have been slightly penalised in terms of ERDE30 or F1-score since we encouraged our models to be as fast as possible; however, further error analysis revealed that many of the errors were induced by data that produced notable disagreement among annotators.
Further work could focus on modifying our annotation algorithm, obtaining a BERT-like embedding by message, and looking at the attention layers of a trained a BERT model with a concatenation of those embeddings. Other studies could be leveraged training our systems with some external data from diferent sources, exploring diferent threshold heuristics, or modifying our system to prevent it from making premature classifications.
We would like to thank Aitor García Pablos for his insightful comments and recommendations. This work was partially funded by the Basque Business Development Agency, SPRI, under the ESAN project (grant agreement KK-2021/001179).
Table 8 describes the five versions of message-level annotations obtained by applying the diferent AC Δ thresholds (see Section 4.1) to the original streams. Table 9, in turn, describes the 10 dataset versions obtained after partial stream sampling (PSS) to the message-level annotated streams (see Section 4.2.1).
Our models were trained on a NVIDIA GeForce RTX 2080 GPU with 11GB of memory using
the AdamW optimiser [
Table 11 and Table 12 describe the estimated hardware electricity power consumption submitted
by the participants that we cite in Section 5.2 (oficial baselines were excluded from these tables,
as no footprint metrics were provided for those by the organisers) for the eating disorders task
and the depression task, respectively. These metrics were obtained using the Code Carbon
Python library [
ERD Team ∀s + LocMax ∀s + GloMax ∀s + LocMax CIMAT-NLP-GTO Team UMUTeam SINAI-SELA ∀s + LocMax 1s + GloMax1 Baseline