This paper presents the contributions of the UPV-Symanto team, a collaboration between Symanto Research and the PRHLT Center, in the eRisk 2021 shared tasks on gambling addiction, self-harm detection and prediction of depression levels. We have used a variety of models and techniques, including Transformers, hierarchical attention networks with multiple linguistic features, a dedicated early alert decision mechanism, and temporal modelling of emotions. We trained the models using additional training data that we collected and annotated thanks to expert psychologists. Our emotions-over-time model obtained the best results for the depression severity task in terms of ACR (and second best according to ADODL). For the self-harm detection task, our Transformer-based model obtained the best absolute result in terms of ERDE5 and we ranked equal first in terms of speed and latency.
The availability of user-generated texts on social media such as Reddit and Twitter, makes it
possible to organize an early reaction to risks and threats as these are mentioned in conversations
between users. It has been shown that social media language data can be used for detecting
natural risks such as floods and earthquakes [
Task 1 For Task 1, we built a system for classifying Reddit users as potential pathological
gamblers based on their writings. This task was organized as an "only-test" task, with no
training data released by the organizers. The test data is collected from Reddit following the
procedure described in [
Task 2 For this task, participants were asked to develop a system for predicting early signs
of self-harm. The task was framed as a binary classification task (self-harm, no self-harm). As
for Task 1, the data consists of Reddit comments in English, collected following [
Task 3 In contrast to both Task 1 and Task 2, Task 3 is not focused on early prediction, but on estimating the severity of the users’ depression. As for Task 1 and Task 2, the data source is Reddit and all the texts are in English; for each Reddit user in the dataset (90 in total), the organizers collected their Reddit post history; furthermore, the organizers provided the answers to a depression questionnaire as filled by each user included in the dataset. The goal of Task 3 consists in estimating users’ response to the questionnaire given their history of Reddit comments.
We approached the three tasks using a combination of neural models and manually engineered features, developed by domain experts. We collected additional data from Reddit and hired expert psychologists to annotate it. We obtained the best results in Task 2 and Task 3 according to several key metrics.
We train our models in a supervised fashion using all the data released by the organizers. In
addition to that, we augment the released training splits by collecting additional data from
Reddit.1 For Task 1, we build a training and development set from scratch, since no data was
released by the organizers for this task. We follow the strategy described in [
We then collect all the comments and submissions from all the Reddit users who posted a text matching the queries. Furthermore, we collect all the comments and submissions posted to a manually compiled list of subreddits.2 We collected in total approximately 16 million texts.
Data Annotation Since we ran our initial queries against all of Reddit and since we collected
indiscriminately all the posts and comments posted to specific subreddits, we expected the data
to be noisy and to contain a lot of false positives.3 Considering the large size of the collected
dataset, sampling a random portion of the data for annotation would have probably lead to
a highly imbalanced label distribution. For these reasons, we adopted a "search as labelling"
approach: all the data was dumped into a PostgreSQL database [
We approach Task 1 in two steps. First, we train a text classifier on each text of each user independently, propagating the users’ labels (e.g., pathological gambler or not pathological gambler) to their writings, assuming that the information contained in a single post can potentially be 2Each subreddit is a forum dedicated to a particular topic. The complete list of crawled subreddits can be found in Appendix A.
3For example, we noticed that many submissions on depression-related subreddits are from people asking for help for their loved ones instead of being ill themselves.
3.1.1. Transformer Model with Alert-Emitting System
For modelling task 1, we train a Transformer-based text classifier using a pretrained, small
English uncased Bert model [
To build the alert-emitting system, we followed the work of the participants that obtained
the best results in the 2020 edition of this shared task [
5We use the implementation available in scikit-optimize.
The oficial evaluation setup of Task 1 is composed of two set of metrics: one for a decision-based
evaluation and one for a ranking-based evaluation. The decision-based evaluation provides an
estimation of models’ performance at classifying at-risk users, while ranking-based evaluation
is used to asses the goodness of a model at sorting users by their level of risk. For the
decisionbased evaluation, the standard classification metrics are used, i.e., precision ( P), recall (R) and
f-measure (F1), together with a set of metrics which take into account the time required to
emit an alert. We measure classification performance considering the number of posts required
to emit a correct prediction for each user (using ERDE5 and ERDE50, requiring 5 and 50
posts respectively) and considering the number of writings required by a system for finding
true positive instances (using latencyTP and latency-weighted F1). Table 3 shows the oficial
results as computed by the organizers on the test set. A detailed description of the metrics
can be found in [
We experimented with two types of models for approaching Task 2. The first model mirrors our work for Task 1, using a Transformer model to classify each post individually and then predicting a label for a user based on the probabilities assigned to their writings. A second model consists of a hierarchical LSTM-based architecture with attention (HAN) using a set of hand-crafted features. For both types of models, at inference time we use the alert-emitting system described in Section 3.1.1. Runs UPV-Symanto 0, 2 and 3 are based on HAN, while runs UPV-Symanto 1 and 4 are based on a Transformer.
6We don’t report here on the oficial ranking-based evaluation for Task 1, since due to a bug our model always predicts the negative class and thus all the metrics are equal to 0.
4.1.1. Transformer Model
The Transformer-based architectures that we use for modelling Task 2 are the same that we
used for Task 1, with the same hyper-parameters described in Appendix B.1.
4.1.2. Hierarchical Attention Network with Composite Features
In Task 2, we used a Hierarchical Attention Network (HAN)[
Content features. We include a general representation of text content by transforming
each text into word sequences. Preprocessing of texts includes lowercasing and tokenizing,
removing punctuation and numbers; function words are not excluded. Most frequent 20,000
words were selected to form the vocabulary, and words not in the vocabulary were represented
as a special "unknown" token. When passed as input to the neural networks, words within a
sequence were encoded as embeddings of dimension 300. In order to initialize the weights of
the embedding layers, we started from pre-trained GloVe embeddings [
Style features. We aim at representing the stylistic level of texts through including function
word and pronoun features. Function words have traditionally been used as stylistic markers,
whereas increased use of pronouns, especially first person pronouns, has been shown to correlate
with mental disorder risk [
LIWC features. The LIWC [
Emotions and sentiment features. We dedicate a few features to represent emotional
content in our texts, since the emotional state of a user is known to be highly correlated with their
mental health. Several of the LIWC categories aim to capture sentiment polarity and emotion
content (negative emotion, positive emotion, afect , sadness, anxiety). We additionally include
a second lexicon, the NRC emotion lexicon [
8http://www.liwc.net/ 9Time limitations at the inference stage prevented us from using the more sophisticated sentiment and emotion model available through Symanto’s Text Analysis API (https://www.symanto.com/api/), which we intend to explore
NRC features similarly to LIWC features, by computing for each category the proportion of words in the text which are associated with that category.
Experimental setup. We trained our models on the eRisk 2020 [
During the training phase, we do not consider social media posts individually as datapoints, since they are too short to be suficiently predictive. Instead, we generate our datapoints by grouping sequences of chronologically consecutive posts into larger chunks, to obtain more consistent samples of text as our datapoints. Features are computed at chunk-level. We use diferent values for : UPV-Symanto 0 uses chunks of 80 posts, while UPV-Symanto 2 uses 10 posts per chunk, and UPV-Symanto 3 uses 50 posts per chunk.
Hierarchical Attention Network. Hierarchical attention networks were introduced in [
In the hierarchical setup, posts within a chunk (datapoint) are stacked to form a hierarchical structure: word sequences (truncated at 256 words), as well as the rest of vectorial numerical and bag-of-words features, are stacked to form bi-dimensional vectors. Bag-of-words and numerical features also follow a hierarchical structure, with a set of features extracted for each post in the group, and stacked together into bi-dimensional vectors. The hierarchical network is composed in future work. of two components: a post-level encoder, which produces a representation of a post, and a user-level encoder, which generates a representation of a user’s post history. Each of the posts in the input datapoint is encoded with the post-level encoder, and then they are stacked to form a bi-dimensional representation, which is then concatenated with the other features, and passed to the user-level encoder. We choose to model the user-level encoder as an LSTM layer with attention, with 32 units. The output of the user encoder is connected to the output layer which generates the final prediction. A depiction of the hierarchical architecture is shown in Figure 1.
We use batch normalization and L2 regularization. Binary cross-entropy is used as a loss function. More details on the network’s configuration are found in the Appendix. Development results for Task 2, with the automatically optimized hyper-parameters for the alertemitting system: minimum number of posts ( ), maximum number of posts ( ), minimum threshold for emitting a prediction ( ). The bold values in each column denote the optimized metric. H: Hierarchical Attention Network; T: Transformer model.
model
The evaluation of task 2 mirrors exactly the setup of task 1. Table 4 reports the development results and Table 5 highlights the oficial results on the test set as released by the organizers.
Task 3 consists of filling a questionnaire with 21 questions related to the user’s mental state from the user’s Reddit post history.
ERDE5
NDCG@10
NDCG@100 # writings 1 100
H T H H T H T H H T
run UPV-Symanto 0 UPV-Symanto 1 UPV-Symanto 2 UPV-Symanto 3 UPV-Symanto 4 UPV-Symanto 0 UPV-Symanto 1 UPV-Symanto 2 UPV-Symanto 3 UPV-Symanto 4
5.1.1. Emotions over Time Model As one of our models, we chose an approach based on the evolution of emotions and certain psycho-linguistic features over time. Unlike other models used in this task, this approach models users not by extracting static features from their writings, but instead as time series describing their communication style related to emotions and self-expression over time.
Features. We use, to this efect, some of the features introduced in Section 4.1, namely: the 10 emotion categories in the NRC lexicon, and in addition 3 categories of the LIWC lexicon related to self-reference: I (usage of first person singular pronoun), we (usage of first person plural pronoun) and ppron (overall usage of personal pronouns). We compute scores for each of these categories for each post in the dataset, in a similar way to the feature extraction step for Task 2: for a given text and feature (lexicon category), we compute the number of words in the text corresponding to that category, normalized by the text length.
In order to obtain time series for each of the considered features, we compute the scores for a given user aggregated at the day level (computed over all texts posted in one day by a given user). In this way, we allow a fair comparison between users who have diferent habits in terms of frequency of posting, but who might nevertheless exhibit similar patterns in terms of emotion evolution over time. We also apply a rolling average of 100 days over the obtained scores, so as to reduce noise.
User Similarity over Time. In order to obtain predictions for a given user, we use the computed time series to define a similarity metric between users, and then predict answers to the questionnaire by imitating the answers of similar users in the training set. The similarity metric between users includes two factors: the static scores for the extracted features for the two users, as well as the correlations over time for the two users.
1. Static scores. We compute the average score for each of the considered features for a given user across all their writings (as a score between 0-1). The distance between two users will be computed as the arithmetic diference between the scores for two users (If is the distance between two users, the similarity between two users is then 1 − ). 2. Correlations over time. These are computed between the time series of feature scores corresponding to two given users. Since the time series for any two users are not guaranteed to have the same length, we attempt to "align" the two time series by finding the maximum correlation between them. We use a sliding window of the length of the smaller of the time series, and compute correlation scores for all possible alignments between the two time series, then take the maximum correlation score as the similarity between the two time series.
The final similarity score between two users is computed as the sum between the static and temporal component, both factors contributing with equal weight.
Predictions for a given user are computed separately for each question in the questionnaire,
as a weighted mean to the answers of the most similar 15 users in the training data, weighing
the answers (as integers) with the corresponding similarity scores, and rounding the result
to the nearest integer (in order to obtain a valid answer to the question10). In this way, we
approach the prediction of answers as a regression problem, by considering a continuous
range of possible answers, and are able to obtain good approximations for the overall level
of depression (obtaining high ADODL and ACR scores), even when the exact answer is not
correctly predicted.
5.1.2. Classification with Reddit BERT model
This model is trained in a two step approach. We first crawl posts from subreddits related to
mental health issues such as depression, self harm and anxiety. We group the data into 13
categories related to mental health and an additional category consisting of random posts. We
then train a balanced classifiers to discriminate between these 14 classes. We train a classifier
based on distilroberta-base [
In a second step we extract the [CLS] embedding of the pre-trained model for every post in the training and test datasets as well as the probability of the depression class. While the first is used as the main representation for classification the second one gives us a notion of relevance.
For every user in the training and tests sets we then average the embeddings of their posts to obtain the final user representation. Given this representation we train a classifier for each of the 21 questions. Since the dataset contains a small number of users we find it helpful to create multiple examples per users by sampling 80% of the user’s posts. This can be understood as a form of random dropout. Additionally we find it beneficial to restrict to the posts where the pre-trained model predicts a probability of > 0.0711 for the depression class. The motivation is that many posts are unrelated to the user’s mental state and that this filtering removes the noise introduced by these irrelevant posts.
10For questions where one answer had two variations ( and ), we ignored the variation and only considered the integer value.
11Recall that we trained a balanced classifier on 14 classes so that the average probability assigned to a class is ≈ 0.07
Following the setup used in the shared task we use the following metrics: Average Hit Rate (AHR), Average Closeness Rate (ACR), Average Diference between Overall Depression Levels (ADODL), Depression Category Hit Rate (DCHR) and the average of the former four metrics (MEAN).
In Task 3, we have considered that among the features used for mental disorder detection in previous literature, LIWC categories, emotions and personal pronouns usage have consistently been shown to be relevant for this task [27, 28, 29, 30, 31, 32, 33]. In previous work [34] we have noticed that the expression of emotions in relation with the use of personal pronouns reveals a specific pattern in users diagnosed with a mental disorders. For example, the use of “I” and personal pronouns in general present diferences in the correlation with all positive 12Random answer drawn from the train distribution of each questions. Metrics are averaged over 10 runs. ADODL
DCHR
MEAN ADODL
DCHR
MEAN emotions between depressed and not depressed people: the more depressed people use “I” and personal pronouns, the more they express positive emotions like joy,anticipation and trust, and the opposite happens with not depressed people. On this basis, in task 3 we considered that similarity in the expression of emotions and in the use of personal pronouns could be a predictor of responses in the Beck Depression Inventory. The good results obtained on the ACR and ADODL reinforce the idea that the expression of emotions, both in a static way and in its evolution over time, are a strong sign of development of depression.
We experimented with modelling all the three tasks simultaneously with a single Multi-Task Learning (MTL) model [35]. We aggregated all the datasets and trained a roberta-base [23] model, using a masked version of the cross-entropy loss, which does not penalize the model when training on instances with missing labels. These experiments did not provide good results.
For Task 1, the promising results obtained during development degraded terribly on the oficial test set due to a bug introduced right before the submission.
The choice of post chunking for training our hierarchical attention network for self-harm detection (Task 2) was motivated by preliminary experiments showing that classifying individual posts (using a network with a comparable architecture) does not achieve reasonable performance on the development data. We also experimented with diferent number of posts per chunk (including training and prediction), ranging between 10 and 90 posts per chunk and have seen that, overall, prediction performance (in terms of F1-score) increases proportionally with the size of chunks, while the models using larger chunks lose some performance in terms of latency-based metrics (latency-weighted F1 and ERDE scores). We tuned the early-alert decision mechanism applied to the trained models’ predictions (in terms of the diferent metrics evaluated on the development data) for choosing the configurations of our oficial runs.
In Task 3, an interesting question is why we obtain good scores in ACR and ADODL (that informs about scores predictions item by item and about overall depression level), but we fail to predict the depression category. What is observed is that we lose a lot of precision when we move from an ordinal scale, such as ACR and ADODL, to a categorical scale such as DCHR. We consider for future work the possibility to examine what patterns in ACR and ADODL are present in users who have been well classified and what patterns are present in those who have been misclassified in terms of categories for depression. It could be possible to apply a correction factor to certain extreme scores on certain items if we observe that these scores and these items play a major role in the errors of user’s classification to a certain level or category. Another interesting observation is that the classification-based models perform well on the development setup but not on test. A possible explanation is that the there is a discrepancy of users with minimal and moderate depression: 24% on test, 47% on dev and 40% on train (Appendix C). However, it is unclear why this afects the classification-based models more than other approaches.
Newly emerging AI-supported services are in a promising position in the use of early detection and prevention of mental health conditions. The necessity to implement such services into everyday life is becoming more relevant, especially considering recent incidents like the CoViD19 pandemic where many people sufered from psychological problems [ 36]. Digital mental health solutions (e.g. therapy programs, chat bots, smart device applications) is one of the largest use case areas in this sense. Early detection is a vital aspect of therapeutic interventions which makes the process more efective and prevents an aggravation of symptoms [ 37]. AIsupported systems could be used as a preliminary analysis tool in clinical settings to enable an early and preventive way of determining the type of the condition, severity of symptoms, and recommendations for a successful therapy concept [38]. Furthermore, digital solutions could solve the issue of accessibility, and stigmatization while providing individuals a healthy and unbiased way of self-help [39]. Besides the application in a clinical environment, AI-supported tools would be of use for general well-being practices. A holistic understanding of mental health requires not only the detection of problems but also the positive build-up of human psychology. Linguistic social media data is able to reflect diferent components of well-being, thus provide important insights into the everyday representations of mental health topics [40]. Early detection models developed by considering such insights could help to raise awareness on self-reflection and foster preventive lifestyle interventions. Academic and organizational institutions, which possess large application and impact areas on individuals, could use such tools to predict well-being of students and employees, and spot and support at-risk individuals [41, 42].
Apart from the overviews of the previous editions of the eRisk shared task [
There are many studies in both computational linguistics and psychology which approach the problem of analyzing the language of people sufering from a mental disorder, especially depression. Many of these studies perform simple quantitative analyses or use traditional machine learning models (such as logistic regression). Recently, more studies have started employing deep learning for mental disorder detection, generally using word sequences as features [44, 45, 46]. A few recent studies also use pre-trained transformers for detection of mental health disorders [47, 48].
Hierarchical attention networks have successfully been used for mental health disorder
detection in the past, including previous editions of the eRisk shared task: Mohammadi et al.
[49] use HANs for anorexia detection (obtaining best results at the eRisk 2019 shared task [
Most computational studies model mental disorder symptoms as static phenomena, whereas
the evolution of mental disorder markers, as well as their prevalence in texts posted by a user,
is an important indicator of mental disorder risk. We mention one previous study [
Emotions have been previously shown to be relevant for modelling mental disorders, but not many go beyond simple quantitative analyses. We mention an approach focused on a ifne-grained analysis of emotions, published three studies on depression, anorexia and self-harm detection [53, 54, 55]. Starting from Plutchik’s eight basic emotions [56], the authors use word embedding spaces to automatically identify sub-emotions, which they use as features for their classifiers, trained to detect depression [53], anorexia [54] and self-harm [55] respectively.
The "search as labelling" approach that we used to annotate our internal Reddit corpus is described in [57].
In this paper we presented the contributions of the UPV-Symanto team in the eRisk 2021 shared
tasks: gambling addiction and self-harm detection and the prediction of depression levels,
based on social media text data. We have used a variety of models and techniques, including
Transformers, hierarchical attention networks with multiple linguistic features, a dedicated
early alert decision mechanism, and temporal modelling of emotions. We ranked first in terms
of ACR and second in terms of ADODL for Task 3, exceeding the previous state-of-the-art
for this eRisk shared task [
The authors from Universitat Politècnica de València thank the EU-FEDER Comunitat Valenciana 2014-2020 grant IDIFEDER/2018/025. The work of Paolo Rosso was in the framework of the research project PROMETEO/2019/121 (DeepPattern) by the Generalitat Valenciana. We would like to thank the two anonymous reviewers who helped us improve this paper.
For augmenting the training data, we collected all the posts and comments from the following list of subreddits:
B.1. Transformer-based Model • batch size = 8 • optimizer = Adam • dropout = 0.1 • learning rate = 5e-5 • early stopping patience = 7 • epochs = 15 • maximum sequence length = 512 B.2. Hierarchical Attention Network • LSTM units (post encoder) = 128 • dense BoW units = 20 • dense lexicon units = 20 • LSTM units (user encoder) = 32 • dropout = 0.3 • 2 = 0.00001 • optimizer = Adam • learning rate = 1e-4 • early stopping patience = 5 • epochs = 25 • maximum sequence length = 256 • posts per chunk = 80
faster, cheaper and lighter, ArXiv abs/1910.01108 (2019). [23] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [24] A.-S. Uban, P. Rosso, Deep learning architectures and strategies for early detection of self-harm and depression level prediction, in: CEUR Workshop Proceedings, volume 2696, Sun SITE Central Europe, 2020, pp. 1–12. [25] L. Oliveira, Bioinfo@ uavr at erisk 2020: on the use of psycholinguistics features and machine learning for the classification and quantification of mental diseases (2020). [26] D. Maupomé, M. D. Armstrong, R. Belbahar, J. Alezot, R. Balassiano, M. Queudot, S. Mosser, M.-J. Meurs, Early mental health risk assessment through writing styles, topics and neural models (2020). [27] M. De Choudhury, S. Counts, E. J. Horvitz, A. Hof, Characterizing and predicting postpartum depression from shared facebook data, in: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, 2014, pp. 626–638. [28] M. De Choudhury, M. Gamon, S. Counts, E. Horvitz, Predicting depression via social media, in: Seventh international AAAI conference on weblogs and social media, 2013. [29] S. C. Guntuku, D. B. Yaden, M. L. Kern, L. H. Ungar, J. C. Eichstaedt, Detecting depression and mental illness on social media: an integrative review, Current Opinion in Behavioral Sciences 18 (2017) 43–49. [30] M. Trotzek, S. Koitka, C. M. Friedrich, Word embeddings and linguistic metadata at the clef 2018 tasks for early detection of depression and anorexia., in: L. Cappellato, N. Ferro, J. Nie and L. Soulier (eds.) CLEF 2018 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings.CEUR-WS.org, volume 2125, 2018. [31] M. Conway, D. O’Connor, Social media, big data, and mental health: current advances and ethical implications, Current opinion in psychology 9 (2016) 77–82. [32] P. Resnik, A. Garron, R. Resnik, Using topic modeling to improve prediction of neuroticism and depression in college students, in: Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1348–1353. [33] J. C. Eichstaedt, R. J. Smith, R. M. Merchant, L. H. Ungar, P. Crutchley, D. Preoţiuc-Pietro, D. A. Asch, H. A. Schwartz, Facebook language predicts depression in medical records, Proceedings of the National Academy of Sciences 115 (2018) 11203–11208. [34] A. S. Uban, B. Chulvi, P. Rosso, An emotion and cognitive based analysis of mental health disorders from social media data, Future Generation Computer Systems (In press) (2021). [35] R. Caruana, Multitask learning, Machine learning 28 (1997) 41–75. [36] B. Pfeferbaum, C. S. North, Mental health and the covid-19 pandemic, New England
Journal of Medicine 383 (2020) 510–512. [37] S. Graham, C. Depp, E. E. Lee, C. Nebeker, X. Tu, H.-C. Kim, D. V. Jeste, Artificial intelligence for mental health and mental illnesses: an overview, Current psychiatry reports 21 (2019) 1–18. [38] M. Ewbank, R. Cummins, V. Tablan, A. Catarino, S. Buchholz, A. Blackwell, Understanding the relationship between patient language and outcomes in internet-enabled cognitive behavioural therapy: A deep learning approach to automatic coding of session transcripts, Psychotherapy Research (2020) 1–13. [39] C. A. Lovejoy, Technology and mental health: the role of artificial intelligence, European
Psychiatry 55 (2019) 1–3. [40] H. A. Schwartz, M. Sap, M. L. Kern, J. C. Eichstaedt, A. Kapelner, M. Agrawal, E. Blanco, L. Dziurzynski, G. Park, D. Stillwell, et al., Predicting individual well-being through the language of social media, in: Biocomputing 2016: Proceedings of the Pacific Symposium, World Scientific, 2016, pp. 516–527. [41] E. Pogrebtsova, G. F. Tondello, H. Premsukh, L. E. Nacke, Using technology to boost employee wellbeing? how gamification can help or hinder results., in: PGW@ CHI PLAY, 2017. [42] S. Volkova, K. Han, C. Corley, Using social media to measure student wellbeing: a largescale study of emotional response in academic discourse, in: International Conference on Social Informatics, Springer, 2016, pp. 510–526. [43] G. Coppersmith, M. Dredze, C. Harman, K. Hollingshead, M. Mitchell, CLPsych 2015 shared task: Depression and PTSD on Twitter, in: Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, Association for Computational Linguistics, Denver, Colorado, 2015, pp. 31–39.
URL: https://www.aclweb.org/anthology/W15-1204. doi:10.3115/v1/W15-1204. [44] F. Sadeque, D. Xu, S. Bethard, Uarizona at the clef erisk 2017 pilot task: linear and recurrent models for early depression detection, in: CEUR workshop proceedings, volume 1866, NIH Public Access, 2017. [45] G. Shen, J. Jia, L. Nie, F. Feng, C. Zhang, T. Hu, T.-S. Chua, W. Zhu, Depression detection via harvesting social media: A multimodal dictionary learning solution., in: IJCAI, 2017, pp. 3838–3844. [46] Y.-T. Wang, H.-H. Huang, H.-H. Chen, A neural network approach to early risk detection of depression and anorexia on social media text., in: L. Cappellato, N. Ferro, J. Nie and L. Soulier (eds.) CLEF 2018 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings.CEUR-WS.org, volume 2125, 2018. [47] M. Matero, A. Idnani, Y. Son, S. Giorgi, H. Vu, M. Zamani, P. Limbachiya, S. C. Guntuku, H. A. Schwartz, Suicide risk assessment with multi-level dual-context language and bert, in: Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, 2019, pp. 39–44. [48] A. Zirikly, P. Resnik, O. Uzuner, K. Hollingshead, Clpsych 2019 shared task: Predicting the degree of suicide risk in reddit posts, in: Proceedings of the sixth workshop on computational linguistics and clinical psychology, 2019, pp. 24–33. [49] E. Mohammadi, H. Amini, L. Kosseim, Quick and (maybe not so) easy detection of anorexia in social media posts., in: CLEF (Working Notes), 2019. URL: http://ceur-ws.org/Vol-2380/ paper_74.pdf. [50] G. Rao, Y. Zhang, L. Zhang, Q. Cong, Z. Feng, Mgl-cnn: A hierarchical posts representations model for identifying depressed individuals in online forums, IEEE Access 8 (2020) 32395– 32403. [51] H. Amini, L. Kosseim, Towards explainability in using deep learning for the detection of anorexia in social media, in: International Conference on Applications of Natural Language to Information Systems, Springer, 2020, pp. 225–235. [52] A. S. Uban, B. Chulvi, P. Rosso, On the explainability of automatic predictions of mental disorders from social media data, in: International Conference on Applications of Natural Language to Information Systems (In press), Springer, 2021. [53] M. E. Aragón, A. P. López-Monroy, L. C. González-Gurrola, M. Montes, Detecting depression in social media using fine-grained emotions, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 1481–1486. [54] M. E. Aragón, A. P. López-Monroy, M. Montes-y Gómez, Inaoe-cimat at erisk 2019: Detecting signs of anorexia using fine-grained emotions., in: L. Cappellato, N. Ferro, D. Losada and H. Müller (eds.) CLEF 2019 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings.CEUR-WS.org, volume 2380, 2019. [55] M. E. Aragón, A. P. López-Monroy, M. Montes-y Gómez, Inaoe-cimat at erisk 2020:
Detecting signs of self-harm using sub-emotions and words 2696 (2020). [56] R. Plutchik, The emotions, University Press of America, 1991. [57] J. Attenberg, F. Provost, Why label when you can search? alternatives to active learning for applying human resources to build classification models under extreme class imbalance, in: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 423–432. URL: https://doi.org/10.1145/1835804.1835859. doi:10.1145/1835804. 1835859.