Developing methods for the early detection of mental disorders like depression and anorexia based on written text has become an important aspect with the rise of social media platforms. The CLEF 2018 eRisk shared task consists of two subtasks focussed on the detection of these two disorders and FHDO Biomedical Computer Science Group (BCSG) has submitted results obtained from four machine learning models as well as from a final late fusion ensemble. This paper describes these models based on user-level linguistic metadata, Bags of Words (BoW), neural word embeddings, and Convolutional Neural Networks (CNN). BCSG has achieved top performance according to ERDE50 and F1 score in both subtasks.
This paper describes the participation of FHDO Biomedical Computer Science
Group (BCSG) at the Conference and Labs of the Evaluation Forum (CLEF)
2018 eRisk task for early detection of depression and anorexia [
Studies concerning the effect of mental state on the language used by a person
have already shown various connections, beginning with observations of more
frequent uses of first personal singular pronouns in spoken language of depression
patients [
Results like these have lead to the development of tools that allow researchers
and therapists to evaluate written texts with a focus on the author’s mental state.
One such tool is the Linguistic Inquiry and Word Count (LIWC) software [
First results in the area of early detection of depression based on written
social media texts have been reported as part of the eRisk 2017 pilot task [
Similar to the task in 2017, the datasets of both subtasks consist of messages obtained from the social media platform reddit.com. The training data of the depression subtask is equivalent to the full training and test data of this previous task, while the anorexia subtask is based on completely new messages. An especially interesting aspect of reddit is that it allows users to create communities with specific topics called subreddits. There exists a wide variety of these communities, also including very active ones from a depression detection perspective, like /r/depression5, which is mainly used by people struggling with depression.
The messages contained in both datasets can consist of a separate title and text field depending on the type of message: Users can post content in terms of links or images (only title, link or image not included), text content (title and optional text), or as comment on another message (only text). Some messages in both datasets also include no text or title and can therefore be discarded. The number of documents per user ranges between 10 and 2000. In every week of the test phase, a chunk of 10% of each user’s messages is supplied to the 5 http://www.reddit.com/r/depression, Accessed on 2018-04-02 participants in chronological order, resulting in 1 to 200 documents per user each week. In both subtasks there are exceptions of this general rule because one anorexia training user (subject2167 of the control group), three depression test users (subject5161, subject5301, and subject8719), and two anorexia test users (subject4169 and subject7483) do not have any messages in the final week, resulting in only 9 messages of these users.
Table 1 displays the main characteristics of the two datasets. The average
amount of characters and unigrams per document was calculated based on a
concatenation of the text and title field. To calculate the number of unigrams,
the same preprocessing and tokenization as described in sections 4.3 and 4.4 was
utilized, retaining only words that occur in the writings of at least two users.
The participation of this team in the eRisk 2017 pilot task was based on a set
of user-level linguistic metadata features that were used as additional input for
every model. In this second eRisk shared task, only one submitted model (see
section 4.1) and the final late fusion ensemble (see section 4.5) use metadata
features. All text based features have again been calculated based on a
concatenation of the text and title field of each message. Still, this includes the same set
of features described in the previous working notes paper [27] and an additional
set of ten features obtained from the Linguistic Inquiry and Word Count (LIWC)
[
Figure 1 illustrates the correlation matrix of the complete metadata feature set and includes the class label information to indicate the relevance of each feature. Although some features—especially the pronoun counts—seem redundant at first sight, all of the original features are preserved as they are based on a Part of Speech (POS) tagging using the Python NLTK framework6 while LIWC features are based on a lexicon that also includes abbreviations or common misspellings. Most of the described features are averaged over all documents per user to obtain the final metadata feature vector, except for the counts of specific phrases like medication names or mentioned diagnoses which are summed. Finally, all averaged features are standardized to have unit variance and a mean of 0 and the summed features are converted to flags with a value of 1 for users that have used such a phrase in any document and -1 otherwise. trsvsneeeb irssvsnepeooun lrssaonponoun itttxeen liittee tngh itseaonanm isso itrseap itxye irsseon teghe ircsonodw i...rrssepngpno litcya ittne it tsene irvcssepoe sun lrsnpaonuon tsaP ssoP reP I""h It""hn ltxeeT tohnM LFW FER DRC FOG icedM iganD tyhM yanM ydepM illtnT xLe ts1 nA cuhA lcunonaF rcsouopnF rsebV itnogC ronoP rseoP lssaC s e tr u a e f 7 1 0 2 k s i R e s e tr u a e f C W I L h c e e p S tfr-o s a P y ilit b a d a e R s e s a r h P
Past tense verbs Possessive pronouns 0.94
Personal pronouns 0.91 0.96 "I" in the text 0.88 0.91 0.96 "I" in the title 0.13 0.16 0.18 0.21 Text length 0.93 0.96 0.97 0.92 0.13
Month 0.03 0.03 0.01 0.02-0.010.01 LWF 0.18 0.27 0.3 0.29-0.040.36 0.06 FRE 0.09 0.16 0.18 0.21-0.030.17 0.06 0.47 DCR 0.12 0.16 0.16 0.17-0.130.21 0.02 0.64 0.55
FOG 0.15 0.2 0.22 0.22-0.090.26 0.05 0.76 0.47 0.85 Medication names 0.03 0.07 0.1 0.13 0.14 0.08 0.04 0.18 0.16 0.13 0.14
Diagnosis 0.03 0.05 0.07 0.11 0.12 0.06 0.02 0.06 0.09 0.03 0.03 0.22 My therapist 0.01 0.05 0.06 0.09 0.07 0.04 0.03 0.09 0.02 0.07 0.09 0.21 0.19
My anxiety 0.07 0.11 0.11 0.15 0.11 0.1 -0.01 0.1 0.05 0.1 0.08 0.26 0.12 0.37 My depression 0.02 0.07 0.09 0.12 0.11 0.07 0.05 0.15 0.18 0.11 0.13 0.44 0.34 0.38 0.32
Title length -0.09-0.13-0.13-0.140.22-0.14-0.03-0.45 -0.6 -0.73-0.67-0.09-0.04-0.09-0.08-0.12
Lexicon words 0.16 0.23 0.25 0.3 0.14 0.22 0.04 0.48 0.63 0.44 0.43 0.22 0.16 0.18 0.17 0.25-0.44 1st pers. sing. pron. 0.21 0.29 0.32 0.41 0.3 0.25 0.07 0.33 0.54 0.39 0.41 0.25 0.17 0.23 0.23 0.32-0.490.67
Analytic -0.15-0.23-0.28-0.34-0.19-0.21-0.06-0.43-0.67 -0.5 -0.51-0.24-0.18-0.23 -0.2 -0.310.61 -0.8 -0.82 Authentic 0.19 0.24 0.28 0.39 0.19 0.25 0.09 0.39 0.45 0.22 0.28 0.27 0.17 0.2 0.19 0.3 -0.3 0.66 0.73 -0.6
Functional 0.19 0.26 0.3 0.34 0.15 0.28 0.05 0.62 0.67 0.56 0.52 0.24 0.15 0.17 0.17 0.25-0.490.92 0.68-0.820.67 Focus on present 0.11 0.17 0.22 0.27 0.2 0.18 0.05 0.38 0.6 0.44 0.42 0.22 0.15 0.2 0.17 0.28-0.490.77 0.65-0.820.49 0.76
Verbs 0.2 0.25 0.29 0.35 0.19 0.25 0.06 0.47 0.66 0.5 0.49 0.22 0.15 0.19 0.17 0.27-0.530.86 0.73-0.860.62 0.86 0.92 Cognitive process 0.19 0.25 0.3 0.34 0.08 0.29 0.04 0.58 0.52 0.45 0.43 0.21 0.12 0.17 0.15 0.24 -0.4 0.77 0.49 -0.7 0.57 0.81 0.67 0.74
Pronouns 0.18 0.26 0.29 0.34 0.22 0.23 0.05 0.4 0.66 0.52 0.5 0.21 0.17 0.21 0.2 0.29-0.590.82 0.87-0.940.61 0.84 0.81 0.87 0.63 Personal pronouns 0.2 0.29 0.32 0.37 0.25 0.25 0.05 0.36 0.61 0.44 0.45 0.23 0.18 0.22 0.21 0.32-0.550.76 0.9 -0.9 0.61 0.76 0.76 0.81 0.54 0.95
Class 0.04 0.11 0.15 0.19 0.05 0.11 0.04 0.26 0.25 0.2 0.23 0.42 0.31 0.33 0.22 0.54-0.210.35 0.44-0.420.39 0.34 0.36 0.36 0.34 0.38 0.41 1 0.8 0.6 0.4 0.2 0 -0.2 -0.4 -0.6 -0.8 -1 6 http://www.nltk.org/book/ch05.html, accessed on 2018-04-02
This section describes the five models that have been used to classify the test users of both subtasks. The models for both tasks are completely identical, use the same set of metadata features, and only vary slightly in their prediction thresholds as described below. In comparison to this team’s participation in the eRisk 2017 pilot task, the prediction thresholds were simplified: For each model, only a single prediction threshold value was chosen based on cross-validation on the training data to indicate whether a subject is classified as depressed. The number of documents already processed for a user is not used anymore as the new models are less prone to predict many false positives after processing only few documents. In addition, non-depressed predictions are now only submitted in the final week because early prediction of these cases has no effect on the score and later writings might still identify them as depressed. Selecting viable prediction thresholds is difficult as a balanced result according to both ERDEo and F1 is often hard to achieve. The goal for this participation was to use rather low thresholds to find depressed cases as early as possible without generating too many false positives.
In contrast to the previous participation of this team, only the first model and
the final ensemble utilize the updated set of user metadata features described in
section 3.1. The bag of words model, which achieved the best overall F1 as well as
second best ERDE5 and ERDE50 score in the previous task [
Bag of Words Metadata Ensemble - BCSGA
The first model is mostly equivalent to the first model used in this team’s
participation in eRisk 2017, except for the extended set of metadata features. It utilizes
an ensemble of Bag of Words (BoW) classifiers with different term weightings
and n-grams that are calculated on a user basis by first concatenating all
documents (text and title) of a user. The term weighting for bags of words can
generally be split into three components: a term frequency component or local
weight, a document frequency component or global weight, and a normalization
component [
All ensemble models use l2-norm for nd but varying local and global weights. The first one uses a combination of uni-, bi-, tri-, and 4-grams obtained from the training data. To build this first BoW, the 200,000 {1, 2, 3, 4}-grams with the highest Information Gain (IG) are selected, given by [14, p. 272]: I(U, C) =
X
X et∈{0,1} ec∈{0,1}
P (U = et, C = ec) · log2
P (U = et, C = ec) P (U = et) · P (C = ec)
, (2) with the random variable U taking values et = 1 (the document contains term t) and et = 0 (the document does not contain term t) and the random variable C taking values ec = 1 (the document is in class c) and ec = 0 (the document is not in class c). The raw term frequency of the resulting n-grams is used as local weight, while their IG-score is used as global weight. The second BoW utilizes a modified version of tf , namely augmented term frequency (atf ) [30], multiplied by idf : atf -idf (t, d) = a + (1 − a)
tft max(tf ) · log
nd
df (d, t)
(3)
with max(tf ) being the maximum frequency of any term in the document, the
total number of documents nd, and the smoothing parameter a, which is set to 0.3
for this model. This BoW, as well as the third one, contains all unigrams of the
training corpus. The local weight of the third model consists of the logarithmic
term frequency (logtf ) [
dft,+ max (1, dft,−) where dft,+ and dft,− is the number of documents in the depressed/non-depressed class that contain the term t. The final model of this ensemble uses the handcrafted user features described in section 3.1.
All three bags of words and the hand-crafted features were each used as
input for a separate logistic regression classifier. Due to the imbalanced class
distribution, a modified class weight was used for these classifiers similar to the
original task paper [
Bag of Words Ensemble - BCSGB The second model is similar to the first one, but it only includes the three bags of words in the ensemble and disregards the metadata features. Again, for the depression subtask any test subject with a probability of at least 0.4 was predicted as depressed, while users with a probability below 0.4 were predicted as non-depressed in the final week. The prediction threshold for the anorexia subtask was set to 0.3 in this case. 4.3
CNN with GloVe Embeddings - BCSGC
The third model consists of a Convolutional Neural Network (CNN) [
convolution document 100x50/300
CReLU activation 100 filters 2x300
The text classification network architecture used for this work is displayed
in Figure 2, which shows the use of 300 dimensional word vectors (and therefore
100 × 300 documents) as used for the next model BCSGD. It is similar to the
7 http://nlp.stanford.edu/projects/glove, Accessed on 2018-03-30
one-layer CNN for sentence classification described by Zhang and Wallace [31]
and consists of only a single convolutional layer, 100 filters with an equal height
of 2 and a width corresponding to the word embedding dimensions, and uses
1-max pooling to extract a single value from each filter. Due to the usage of
Concatenated Rectified Linear Units (CReLU) [
To obtain a final prediction per user, the 98th percentile of the outputs from all the user’s documents is calculated. This ensures that even depressed users that have very few documents with a high probability can be correctly predicted. For both subtasks, any subject with a final probability of at least 0.4 was predicted as depressed in each week, while probabilities below 0.4 again resulted in a non-depressed prediction in the final week. 4.4
CNN with fastText Embeddings - BCSGD
The second CNN model is based on the same architecture as the previous one but
utilizes 300-dimensional fastText [
Similar to the previous CNN model, the resulting 100 × 300 matrix of word embeddings obtained for each document was classified separately and the 98th percentile of the outputs was used as output for the corresponding user. This model was trained for 25 epochs using the same parameters as BCSGC. The prediction threshold for depressed predictions was set to 0.7 for both tasks, 8 https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly _available_reddit_comment/, Accessed on 2018-03-30 leading to a non-depressed prediction for probabilities below 0.7 in the final week.
CNN and Bag of Words Metadata Ensemble - BCSGE
The final model consists of a simple late fusion ensemble that has been calculated
as the unweighted mean of the outputs obtained from models BCSGA, BCSGC,
and BCSGD - the bag of words including metadata and the two CNN models.
Although these outputs have not been calibrated (e.g. by using Platt scaling [
Before examining the results of the described models in the two subtasks, it is necessary to analyze the utilized ERDEo metric for early detection systems. Since this metric is based on the absolute number of documents read per user before a true positive prediction, but these documents have to be read in ten equally sized chunks, the score is highly dependent on the number of documents available per user. Because at least 10% of each user’s documents have to be read by all participants, it is impossible to predict some users correctly depending on the parameter o that describes after how many documents the penalty for late predictions grows. This fact has already been described in more detail in another paper [28].
Table 2 displays the best ERDE5 and ERDE50 scores that are possible for the test data of the depression and anorexia subtask. These results are based on a perfect prediction in the first week of the tasks. As described in the abovementioned paper, only test users with less than 100 documents (less than 10 per chunk) have any effect on the ERDE5 score. This means that only predicting 26 of the 79 depressed test users correctly in the first week and ignoring all others still leads to an ERDE5 score of 7.78 (F1 = 0.50), while predicting only 12 of the 41 anorexia users in the first week also leads to an ERDE5 score of 10.23 (F1 = 0.45). ERDE5 alone, without the additional F1 score, is therefore hard to interpret.
To examine the weekly predictions obtained from the described models, Figures 3 and 4 show the cumulative number of positive predictions for the two subtasks and also visualize the proportion of true positives. For the depression subtask, this shows that the ensemble indeed lead to the most true positives but also many false positives. BCSGD seems to perform worse at first sight but indeed achieved a good balance between true and false positives because of its higher prediction threshold. As the comparison of both figures shows, the sn140 ito130 c ide120 rp110 ive100 itso 90 )p 80 e tru 70 ( fo 60 re 50 bm40 nu 30 itve 20 lua 10 um 0 C anorexia subtask was much easier using the same models and that it was possible to detect nearly all positive samples without too many false positives. Both examinations show a steady progression over the ten weeks for all models. 1 2 3 4 5 6 7 8 9
10
Tables 3 and 4 show the official results [
While the direct comparison of BCSGA (bags of words with linguistic metadata) and BCSGB (bags of words only) shows that the metadata features result in more positive predictions, the actual amount of true positives was only better for the depression subtask and resulted in a better ERDE5 score but worse s n o iitred50 c p e v iit40 s o p ) e ru30 tf( o re20 b m u n iltve10 a u um0 C 1 2 3 4 5
6
7 8 9 10
UNSLA UNSLD UNSLE 9.21 9.50 9.58 9.46 9.52 ERDE50 and F1. Similar to the task in 2017, the bag of words ensemble again obtained the best results in the depression subtask, while the CNN based on the self-trained fastText embeddings (BCSGD) and the ensemble using both the bags of words as well as the CNNs (BCSGE) achieved the best scores in the anorexia subtask. Overall, the models of BCSG achieved the second-best results in ERDE5 and the best results in all other scores except for another second-best result according to ERDE5%0 in the depression subtask.
As already described, the ERDEo score and especially ERDE5 should be discussed in more detail because of the fact that optimizing it can often lead to simply minimizing false positives by only predicting very few users at all. A detailed look at the results and the achieved ERDE5 scores shows that, for example, in the first week of the depression subtask both UNSLA and BCSGA have predicted 45 users as depressed of which 20 were indeed true positives. Still,
PEIMEXB RKMVERIA UNSLB UNSLD the resulting ERDE5 score differs drastically because the predicted users vary in the number of total documents and even though UNSLA only had five more true positives in the following nine weeks, while BCSGA already had ten more in the second week and a total of 53 in the end. Similarly, the leading model in ERDE5 of the anorexia subtask, UNSLB, had 19 true positives in the first week, while BCSGD already had 22, BCSGB had 21, and BCSGA had 20. In summary, ERDE5 produces highly misleading results because of the varying number of documents per user. 6
Again, the eRisk competition has been a challenging task concerning the early detection of mental health issues based on sequences of social media texts. The depression subtask had similar F1 scores but much better ERDEo scores based on a test set that was nearly as large as last year’s training and test set combined. The results of the anorexia subtask were surprisingly good, which probably is due to the nature of this dataset. Generally, the promising results with the test data of eRisk 2017 obtained only based on linguistic metadata [28] could not yet be confirmed in this year’s tasks. As already concluded in the same paper, finding a way to successfully integrate the metadata features into the neural network models is an interesting task for future research.
The examination of the task results again shows that a discussion about a meaningful metric should be a priority in the future. Both Flatency and ERDEo% include interesting ideas to improve the evaluation of early prediction models. Flatency contains a cost function that grows less rapidly and already incorporates the F1 score, which makes it more meaningful when viewed alone. ERDEo% is more viable for chunk-based shared tasks because it is calculated based on the proportion of read documents per user instead of the absolute number, which leads to results that are better interpretable than the standard ERDEo. A combination of these two ideas could be a promising basis for discussions about future early detection tasks. 7
The work of Sven Koitka was partially funded by a PhD grant from University of Applied Sciences and Arts Dortmund, Germany. 27. Trotzek, M., Koitka, S., Friedrich, C.M.: Linguistic Metadata Augmented Classifiers at the CLEF 2017 Task for Early Detection of Depression. Working Notes Conference and Labs of the Evaluation Forum CLEF 2017, Dublin, Ireland (2017) Available from: http://ceur-ws.org/Vol-1866/paper_54.pdf - Accessed on 2018-0329 28. Trotzek, M., Koitka, S., Friedrich, C.M.: Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences. arXiv preprint arXiv:1804.07000 [cs.CL] (2018) 29. Weintraub, W.: Verbal Behavior: Adaptation and Psychopathology. Springer Publishing Company (1981) 30. Wu, H., Gu, X.: Reducing Over-Weighting in Supervised Term Weighting for Sentiment Analysis. The 25th International Conference on Computational Linguistics (COLING 2014), pp. 1322–1330, Dublin, Ireland (2014) 31. Zhang, Y., Wallace, B.: A Sensitivity Analysis of (and Practioners’ Guide to) Convolutional Neural Networks for Sentence Classification. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, pp. 253–263, Taipei, Taiwan (2017)