This paper summarizes the contributions of the PRHLTUPV team as a participant in the eRisk 2020 tasks on self-harm detection and prediction of depression levels from social media. Computational methods based on machine learning and natural language processing have a great potential to assist with early detection of mental disorders of social media users, based on their online activity. We use multi-dimensional representations of language, and compare various deep learning models' performance, exploring rarely approached avenues in previous research, including hierarchical deep learning architectures and pre-trained transformers and language models.
Mental health disorders a ect hundreds of millions of people worldwide; [
The CLEF eRisk Lab 3, organized every year since 2017, is dedicated specifically to identifying early signs of mental disorders from a user's social media posts, before the user was diagnosed with the disorder, for disorders including depression, anorexia and thoughts of self-harm [10{12]. Each year a new task is organized around predicting a speci c disorder: in 2017 and 2018 the shared tasks focused on depression detection, in 2019 a new task for anorexia prediction was organized, as well as a second task around predicting self-harm tendencies without any training data; in 2020 self-harm detection was again the topic, this time in a supervised setting. Datasets are collected from Reddit posts and comments selected from speci c relevant sub-reddits, annotated by automatically detecting self-stated diagnoses of users. Healthy users are selected from participants in the same sub-reddits, thus making sure the gap between healthy and diagnosed users is not trivially detectable. For the self-harm task, the dataset includes only posts published before any involvement in the self-harm related communities, which conditions any model trained on this data to be capable of very early prediction, and at the same time adds di culty to the task.
The language used by a speaker has been shown to contain strong indicators
of an altered mental state. These can manifest both explicitly, at the level of
the topics approached, or implicitly, at the level of the emotional charge of the
text (greater negative emotion [
Most previous research related to automatic mental disorder detection from
social media data have focused the study of depression [
This study summarizes our contributions as participants to the eRisk shared
tasks on self-harm detection and assessment of depression levels [
The rst task in eRisk 2020 consists of detecting whether a user is at risk of developing self-harm tendencies. Training data collected from Reddit was available, consisting of 340 users (of which 41 were labelled as positive) and their Reddit post history. Test data was provided as a stream of user posts, and candidate systems were asked to provide a decision (a binary number: a user is at risk or not), as well as a risk score (a real number), at each time step in the stream.
We participated in the task with ve di erent models. We implement several neural network architectures, as well as experiment with pre-trained models and strategies for sampling training data in order to improve results. Details of the architectures used and the experimental setup are described below. 2.1
Content features. We include a general representation of text content by transforming each text into word sequences. Preprocessing of texts includes lowercasing and tokenizing, removing punctuation and numbers; function words are not excluded. Most frequent 20,000 words were selected to form the vocabulary, and words not in the vocabulary were represented as a special "unknown" token. When passed as input to the neural networks, words within a sequence were encoded as embeddings of dimension 100. In order to initialize the weights of the embedding layers, we started from GloVe embeddings pre-trained on Twitter data. The choice of pre-trained embeddings was justi ed by their dimension, which is smaller than for other GloVe embeddings pre-trained on large corpora, leading to fewer mode parameters overall (in view of avoiding over tting problems). Nevertheless, even though the data for this task is also sampled from social media, the two platforms (Reddit and Twitter) have signi cant di erences as well; exploring the use of other embedding initializations (especially using embeddings pre-trained on longer texts) would be interesting in future experiments.
Style features. We aim at representing the stylistic level of texts through including function word and pronoun features. Function words have traditionally been used as stylistic markers, whereas increased use of pronouns, especially rst person pronouns, has been shown to correlate with mental disorder risk [24]. We include two separate stylistic features: rstly, we extract from each text a numerical vector representing function words frequencies as bag-of-words. Separately, we include a simple scalar feature meant to capture the rst person personal pronoun usage, by measuring the proportion of rst person pronouns relative to the total number of words used in a text. We complement these with features extracted from the LIWC lexicon, as described below.
LIWC features. The LIWC4 [
Emotions and sentiment. We dedicate a few features to represent
emotional content in our texts, since the emotional state of a user is known to be
highly correlated with his/her mental health. Several of the LIWC categories
aim to capture sentiment polarity and emotion content (negative emotion,
positive emotion, a ect, sadness, anxiety ). We additionally include a second lexicon:
the NRC emotion lexicon [
During the training phase as well as for testing, we do not consider social media posts individually as datapoints, since they are too short to be su ciently predictive. Instead, we generate our datapoints by grouping sequences of 50 chronologically consecutive posts into larger chunks, to obtain more consistent samples of text as our datapoints. Features are computed at chunk-level.
As a consequence, prediction is always done on chunks of 50 posts. When analyzing the input stream of test data, we form new chunks of the last 50 posts received periodically (after every 20 new posts), and feed them to the networks to generate predictions.
As we will describe in the following section, we use two types of architectures for modelling the input: sequential and hierarchical. We adopt a special strategy for predictions on the rst 50 posts in the stream: we pad the input data up to the size used during training (512 words in the case of the sequential setup and 50 posts of length 256 in the case of the hierarchical setup), but only submit the output score provided by the network, and as decisions (user is at risk or not) we submit zeros regardless of the output score, so as not to send premature alerts (since once a user is declared at risk, the decision can not be reverted).
Sequence sampling. For one of our runs, we employ a special strategy during the training phase. We attempt to augment the training data through generating "arti cial" chunks of user posts, aside from the ones formed naturally through chunking the user's post history in chronological order. We do this by sampling from the post history randomly, following an exponential distribution so as to sample with higher probability from recent posts (which are more likely to contain signs of the disorder). The chronological order of posts is maintained.
Rolling average of predictions. As previously mentioned, for most runs predictions are generated using the last 50 posts seen in the test data stream. For one of our runs, we use a di erent strategy, by computing a rolling average of the most recent 3 network outputs: in this way, we hope to obtain more robust results that are not dependent only on the last batch of 50 user posts, but take into account a larger window of context. 2.3
BiLSTM with attention. The rst model we consider is a bidirectional LSTM network, with attention. Input word sequences are truncated at maximum 512 words, with words encoded as embeddings, and passed as input to the BiLSTM layer with 256 units, which is then fed to an attention layer. The bagof-words features representing function word distribution are passed through a dense layer of 20 units; and the remaining extracted features (including pronoun, emotion and LIWC category usage) are concatenated into one vector. The output of the BiLSTM is concatenated along with the other features and the nal representation passed through an output layer that generates the nal prediction.
Hierarchical Attention Network. Hierarchical attention networks (HAN)
were introduced in [27] where they were used for review classi cation, by
representing a text as a hierarchical structure where a document is comprised of
sentences and a sentence is comprised of words. We propose that social media
data in our setup is very well suited to such a hierarchical representation; in our
case the hierarchy consists of user post histories, which are composed of social
media posts, which are in turn composed of word sequences. Especially since
the evolution of the mental state of a user is in itself a relevant indicator for
the development of a disorder, as shown in [
In the hierarchical setup, posts within a chunk (datapoint) are stacked to form a hierarchical structure: word sequences (truncated at 256 words), as well as the rest of vectorial numerical and bag-of-words features, are stacked to form bi-dimensional vectors. Bag-of-words and numerical features also follow a hierarchical structure, with a set of features extracted for each post in the group, and stacked together into bi-dimensional vectors. The hierarchical network is composed of two components: a post-level encoder, which produces a representation of a post, and a user-level encoder, which generates a representation of a user's post history. For encoding the word sequence at post-level, we use a convolutional layer with 100 lters of length 3. Each of the posts in the input datapoint is encoded with the post-level encoder, and then they are stacked to form a bi-dimensional representation, which is then concatenated with the other features, and passed to the user-level encoder. We choose to model the user-level encoder as an LSTM layer with attention, with 32 units. The output of the user encoder is connected to the output layer which generates the nal prediction. A depiction of the hierarchical architecture is shown in Figure 1.
Transformers. We experiment with state-of-the-art language models based
on transformer architectures, which have been shown to obtain high
performances on a wide range of NLP tasks, with minimal task-speci c training. We
use pre-trained BERT [
Ensemble. Finally, we use a simple ensemble model for one of our runs: predictions are generated through averaging the outputs of several other models on the received input. 2.4
The models and setups used for each of the ve runs submitted by our team are described below.
trained and ne-tuned BERT model. During ne-tuning, the sequence sampling strategy for data augmentation was used.
Run 1. BiLSTM. Run 1 consists of the BiLSTM model described in the previous section.
Run 2. Hierarchical CNN + LSTM. For run 2 we used the hierarchical attention network with CNN and LSTM layers. Due to memory limitations, we only generated predictions for the rst 50 posts in the stream: all subsequent predictions (for all datapoints in the stream) were based on these outputs.
Run 3. Ensemble. For this run, we used an ensemble of the rst three models: BERT, the BiLSTM and the hierarchical attention network. To obtain prediction scores, we averaged the outputs of the three networks for each input datapoint. A user is considered at risk if the obtained output exceeds the 0.5 threshold.
Run 4. Rolling average of BiLSTM. For our last run, we used the rolling average strategy described in the previous section, to obtain a smoothed version of the model's outputs. For each timestep, we averaged the output of the BiLSTM model for the most recent three inputs (chunks of 50 posts). 2.5
Table 1 show the o cial results obtained for each of our runs. Evaluation
measures included the traditional precision, recall and F1-scores computed at
userlevel, as well as some metrics speci cally designed for measuring how early risk
was detected: latency-weighted F1, which is the F1-score weighted by a
penalizing factor for late predictions, and ERDE [
Run BERT+seq-sampling BiLSTM HAN Ensemble BiLSTM+rolling iLab/run 1 SSN NLP/run 1 iLab/run 4 iLab/run 2 iLab/run 3 iLab/run 0
The best F1-scores were obtained with the BERT model using sequence sampling training, showing that pre-trained transformers are powerful for external tasks including detection of self-harm, and also that the sequence sampling strategy might be an e ective method for data augmentation. The second best results were obtained with the last model - the rolling average of outputs strategy brings signi cant improvement to predictions compared to the base model (simple BiLSTM). We attribute the poorer performance of the HAN and ensemble models to the small size of test data used for predictions ( rst 50 posts in the stream).
A second evaluation approach treats the task as a ranking task, by using the system's continuous risk scores and ranking users in order of risk according to these scores. Metrics speci c to ranking tasks are used to measure performance, including precision @ k (P@10), and Normalized Discounted Cumulative Gain @ k (NDCG@10, NDCG@100). In Table 2 we show the evaluation results for our systems using the ranking metrics, measured on the rst 500 posts in the input stream. Our models perform well on these metrics, the rst system obtaining perfect scores for both metrics measured @ 10. For comparison, we include the system that obtained best scores in terms of all ranking metrics @ 500 writings, submitted by the iLab team. 3
The second task consisted of predicting the level of depression of social media users, by predicting answers to a 21-question questionnaire for assessing depression, where each question can have one of four to six answers. Training data consisting of 20 labelled users was available beforehand. The test data consisted of 70 users' social media posts, and the participating systems had to predict their answers to each of the questions.
Several evaluation metrics were used, measuring how well the predictions match the true labels, from more ne-grained to more general levels, including: average hit rate (AHR), average closeness rate (ACR), average di erence between overall depression levels (ADODL), depression category hit rate (DCHR).
We participated with three di erent models in this task. The details of the models and features used are described below. 3.1
For the rst two models, we used a few of the same features described in the previous sections. We included the lower-dimensionality numerical features: LIWC and emotion categories, represented as continuous vectors. For obtaining userlevel representations, we averaged the values of these vectors computed for each of the user's posts. Since it has been shown that the evolution of certain behaviors and linguistic markers is in itself predictive of developing a disorder or not, we choose to capture the variation of the features extracted, by including in our feature vectors the standard variations (aside from the averages) seen in the distribution of each feature across a user's history of posts.
For our nal model, we tried to leverage pre-trained language models in order
to obtain semantic representations of the user's social media posts. To this e ect,
we extracted sentence representations from Universal Sentence Encoder (USE)
[
We chose to use simpler traditional machine learning models for this task with fewer parameters than the neural networks used in task 1, to suit the small size of the training data: we experimented with SVM and logistic regression models, using the features previously described.
All models were trained on the available training data, and the trained models were used to make predictions on the new data in the testing phase. For each of the models, we modelled the task as a multi-label multi-class classi cation problem, by training one model for each of the 21 questions, where each question can be assigned one of 4-6 labels (depending on the question).
LogReg-features The rst model used was a logistic regression model with the lexicon-based features represented as numerical vectors.
SVM-features For the second run, we used an SVM with RBF kernel, with the same features as for the previous run.
SVM-USE Our last model was an SVM with RBF kernel, and USE features. 3.3
Run AHR ACR ADODL DCHR LogReg-features 34.01% 67.07% 80.05% 35.71% SVM-features 34.56% 67.44% 80.63% 35.71% SVM-USE 36.94% 69.02% 81.72% 31.53% BioInfo@UAVR 38.30% 69.21% 76.01% 30.00% iLab run2 37.07% 69.41% 81.70% 27.14% relai lda user 36.39% 68.32% 83.15% 34.29%
Table 3 shows o cial results results for task 2, for all evaluation metrics. Our best models in terms of DCHR were the models using lexicon-based features, which obtained the maximum score of all participating teams on this metric. The model using USE features has better performance than the other two for the rest of the metrics. The good scores obtained with simple models and features suggest the problem may not be well suited to complex representations and architectures, possibly due to the small size of the training data. For comparison, we include in the table results of the systems that obtained best scores in terms of the other metrics (aside from DCHR). Overall, scores for this task were modest for all participating teams, suggesting predicting the level of depression is a di cult task. 4
In this paper we presented the contributions of the PRHLT-UPV team in the eRisk 2020 shared tasks: self-harm detection and the prediction of depression levels, based on social media text data. We used multi-dimensional features to represent various levels of the language, including content, style and emotion. In the rst task, where more training data was available, we experimented with di erent deep learning architectures, including hierarchical attention networks and transformers, as well as with di erent strategies concerning the experimental setup: such as sequence sampling for data augmentation, and rolling average for smoothing model outputs. For the second task we used traditional models such as SVM and logistical regression, with features including style and emotion features, as well as semantic sentence representations from pre-trained language models. We obtained best scores in terms of detecting the general depression category in the second task.
The work of Paolo Rosso was in the framework of the research project PROMETEO/2019/121 (DeepPattern) by the Generalitat Valenciana. 22. Shen, G., Jia, J., Nie, L., Feng, F., Zhang, C., Hu, T., Chua, T.S., Zhu, W.: Depression detection via harvesting social media: A multimodal dictionary learning solution. In: IJCAI. pp. 3838{3844 (2017) 23. Shen, J.H., Rudzicz, F.: Detecting anxiety through reddit. In: Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology|From Linguistic Signal to Clinical Reality. pp. 58{65 (2017) 24. Trotzek, M., Koitka, S., Friedrich, C.M.: Linguistic metadata augmented classi ers at the clef 2017 task for early detection of depression. In: CLEF (Working Notes) (2017) 25. Trotzek, M., Koitka, S., Friedrich, C.M.: Word embeddings and linguistic metadata at the clef 2018 tasks for early detection of depression and anorexia. In: CLEF (Working Notes) (2018) 26. Wang, Y.T., Huang, H.H., Chen, H.H.: A neural network approach to early risk detection of depression and anorexia on social media text. In: CLEF (Working Notes) (2018) 27. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classi cation. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. pp. 1480{1489 (2016)