Methods for automatic early detection of depressed individuals based on written texts can help in research of this disorder and especially offer better assistance to those affected. FHDO Biomedical Computer Science Group (BCSG) has submitted results obtained from five models for the CLEF 2017 eRisk task for early detection of depression that are described in this paper. All models utilize linguistic meta information extracted from the texts of each evaluated user and combine them with classifiers based on Bag of Words (BoW) models, Paragraph Vector, Latent Semantic Analysis (LSA), and Recurrent Neural Networks (RNN) using Long Short Term Memory (LSTM). BCSG has achieved top performance according to ERDE5 and F1 score for this task.
This paper describes the participation of FHDO Biomedical Computer Science
Group (BCSG) at the Conference and Labs of the Evaluation Forum (CLEF)
2017 eRisk pilot task for early detection of depression [
It is known that depression often leads to a negative image of oneself, pessimistic
views, and an overall dejected mood [
A similar task using Twitter posts was organized at the CLPsych 2015
conference [
The dataset presented in the CLEF 2017 eRisk pilot task consists of text contents written by users of www.reddit.com, which is a widely used communication platform for creating communities called subreddits that cover all kinds of topics3. Specifically, there is a very active community in the subreddit /r/depression4 for people struggling with depression and similar subreddits for other mental disorders exist as well. The registration of a free account using a valid mail address and a public user name is necessary to create content, while reading is possible without registration, depending on the subreddit. Users can post content as link (using a title and either a URL or an image), as text content (using a title and optional text), or as comment (using only the text field and no title).
The given dataset contains all three kinds of content written by 887 users and 10 up to 2,000 documents per user. Table 1 gives a summary of some basic characteristics of the training and test split. The task’s goal is to classify which of these users show indications of depression by reading as few of their posts as possible in chronological order. Each document contains a timestamp of publication, the title, and the text content, while title or text can be empty. There also exist 91 cases of documents with both an empty title and text. The URL or image of link entries is not provided in the dataset. The number of unique n-grams 3 https://redditblog.com/2014/07/30/how-reddit-works-2/, Accessed on 2017-04-24 4 http://www.reddit.com/r/depression, Accessed on 2017-04-24 contains all tokens with more than one alphabetical character (and the word “I”) that occur in at least two documents, also including numbers, emoticons, and words that contain hyphens or apostrophes. After examining the general characteristics of the given dataset, a detailed analysis of the text contents is necessary to get an insight into promising features and the specific properties of the domain. In order to find the most interesting n-grams of the given corpus, Information Gain (IG) or expected Mutual Information (MI) was calculated. In case of binary classification tasks, the information contained in each feature is given as [25, p. 272]:
I(U ; C) =
X
X et∈{0,1} ec∈{0,1}
P (U = et, C = ec) log2
P (U = et, C = ec) P (U = et)P (C = ec) , (1) with the random variable U taking values et = 1 (the document contains term t) and et = 0 (the document does not contain term t) and the random variable C taking values ec = 1 (the document is in class c) and ec = 0 (the document is not in class c).
Similar to the previously described selection of unigrams in the corpus, IG was calculated without stopword removal for all uni-, bi-, and trigrams that can be found in at least two documents. The obtained scores were then used to find the 100 features with the highest IG of the corpus as well as the 100 features with the highest IG that occur more often in the depressed class, which is both shown in Fig. 1.
i'm not ya much go something itseem ilke llaymtwimiggonaoesnsehdgisllyaet rehdotilhiki'itsnemwkyiybtisheohecualafivucnesaoenti o som lfee foam re dn a yatnodbut but i n i p lfee tocseabue ihdahittahneolapiamw thaw ttno
Both analyses give an interesting insight into the corpus that confirm previous research results described in the related work section. Comparing the two word clouds shows that the first person singular pronouns I, me, and my, which are frequently contained in documents of both classes, have the highest IG seen individually and are then found in some of the most important bi- and trigrams of the depressed class. The most important features of this class are, as could be expected, centered around depression and anxiety, while especially relationships (e.g. boyfriend, husband, partner, best friend), treatment (e.g. therapist, psychiatrist, medication), and look (e.g. acne, skin, makeup, alpha hydrox ) can easily be identified as frequent topics and are often combined with personal or possessive pronouns. Interestingly, although the sad emoticon :-( is part of the top features in the depressed class, the happy emoticons :-) and :) occur even more frequently in this class and have a higher IG. The frequent combinations as in “thank you :)” point to the conclusion that this is often a reaction to thoroughly helpful conversations.
When examining the text data further, it becomes evident that the posts sometimes include quotes taken from messages of other users. This could be misleading for classification tasks since the quoted user might show indications of depression, while the actual author of this message might not or vice versa. Luckily, quotes seem to be unfrequent and can be identified to some extend because they are always indented by a single space, do not contain line breaks, and are preceded and followed by an empty line. There is no way to distinguish them from similarly indented one-line paragraphs by the actual author. By using a regular expression, 4,266 quotes can be found in the training data and 4,423 in the test data. For all models described in this paper, the prefix quote_ was added to each token within a quote to make them distinguishable from the same words written by the actual author. 3.2
In addition to different document vectorization methods, a set of hand-crafted
features has been derived from the text data and was used in all approaches.
Several text statistics have been calculated and compared between the class of
depressed and non-depressed users in the given dataset. The most promising
features are displayed in Fig. 2 as box plot for each class. All features have
been calculated as mean over all texts of the same user. In addition to the
already mentioned counts of personal and possessive pronouns, past tense verbs,
and the word I in particular, four standard measures for text readability have
been calculated for the text content, namely Gunning Fog Index (FOG) [
60 40 20 0 5 originally developed by the U.S. Air Force without any available references 12 9 6 3 15 10 5 0
Class 0 - non-depressed 1 - depressed
The average of the months in which all texts of a user have been submitted was included based on the hypothesis that depressive symptoms can be intensified in the winter months. This is difficult to observe in the given dataset, since the age of the available texts depends on how frequently a user has posted due to the limitation to the last 2000 writings per user. Users with many and frequent writings therefore tend to have more samples from early summer 2015 (when the collection was created), while less frequent writers provide a more uniform distribution of texts over all months. Additionally, five features have been created for the users that simply count the occurrences of some very specific n-grams in all their documents. This ensures that some of the strongest indicators of depression can still be identified easily even when using averaged document vectors or just a large amount of documents. These features were used in boolean form by all described models and count the following terms without regard to case: – The chemical and brand names of common antidepressants available in the
United States (e.g.: Sertraline or Zoloft) obtained from WebMD6 – Explicit mentions of a diagnosis including the word depression (e.g.: “I was diagnosed with depression” or “I’ve been diagnosed with anxiety and depression”) – The term “my depression” – The term “my anxiety” – The term “my therapist” The mentioned terms have been picked carefully only from the training documents and have been designed to capture only statements referring to the personal situation of the author with the exception of the antidepressants. They could be extended for future research to include a more comprehensive list of medications or more general expressions of diagnosis (e.g. also including the terms “major depressive disorder” or “MDD”). Although the selected terms are not primarily helpful for early predictions, they are strong indicators to find already diagnosed individuals, which is important for the given task as well. It would also be interesting to include additional statistical features like the number of adjectives, adverbs, noun phrases, positive and negative emotions, and similar, as done for example by LIWC. Figure 3 displays the correlation of all user features without scaling and also includes the label information, where a higher value corresponds to the depressed class. It shows that all features are at least slightly correlated to the information whether a user is depressed or non-depressed. 6 http://www.webmd.com/depression/guide/depression-medications-antidepressants - Accessed on 2017-05-07
Past tense verbs Possessive pronouns 0.97
Personal pronouns 0.9 0.94 "I" in the text 0.81 0.86 0.96 "I" in the title 0.22 0.24 0.28 0.31 Text length 0.93 0.96 0.96 0.89 0.21
Month 0.06 0.04 0.01 0.01 -0.06 0.02 LWF 0.17 0.22 0.27 0.3 -0.02 0.33 0.09 FRE 0.13 0.16 0.19 0.23 -0.02 0.18 0.04 0.47 DCR 0.11 0.15 0.16 0.18 -0.13 0.21 0.01 0.7 0.53
FOG 0.11 0.13 0.17 0.19 -0.08 0.22 0.05 0.75 0.44 0.86 Medication names 0.02 0.04 0.05 0.07 0.07 0.05 0.02 0.1 0.09 0.06 0.06
Diagnosis 0.04 0.05 0.08 0.1 0.09 0.06 -0.02 0.05 0.09 0 0.02 0.13 My therapist 0 0.02 0.03 0.05 0.01 0.02 0.03 0.08 0.01 0.06 0.08 0.19 0.09
My anxiety 0.03 0.07 0.06 0.1 0.08 0.06 -0.01 0.08 -0.01 0.06 0.06 0.78 0.11 0.35 My depression 0.03 0.08 0.1 0.13 0.13 0.09 -0.01 0.11 0.12 0.06 0.08 0.37 0.33 0.21 0.52
Class 0.09 0.14 0.21 0.27 0.05 0.18 0.05 0.31 0.26 0.21 0.25 0.25 0.3 0.18 0.25 0.34
The findings for this specific dataset confirm that texts by individuals suffering from depression indeed contain more pronouns and especially the word “I”. Their texts are also slightly longer and more complex according to three of the four text complexity measures. This likely represents the difference between average users, who often post a large amount of short statements, and those who discuss problems and may even be looking for help.
Two conventional document vectorization models as well as three models
utilizing Long Short Term Memory (LSTM) [
The first model utilizes an ensemble of Bag of Words (BoW) classifiers with
different term weightings and n-grams. The term weighting for bags of words can
generally be split into three components: a term frequency component or local
weight, a document frequency component or global weight, and a normalization
component [
All ensemble models use cosine normalization (l2-norm) for nd but varying
local and global weights. The first one uses a combination of uni-, bi-, tri-, and
4-grams obtained from the training data: the 200,000 [
tft max(tf ) · log
nd
df (d, t)
with max(tf ) being the maximum frequency of any term in the document, the
total number of documents nd, and the smoothing parameter a, which is set to 0.3
for this model. This BoW, as well as the third one, contains all unigrams of the
training corpus. The local weight of the third model consists of the logarithmic
term frequency (logtf ) [
dft,+ max (1, dft,−) (4) where dft,+ and dft,− is the number of documents in the depressed/non-depressed class that contain the term t. The final model of this ensemble uses the handcrafted user features described in section 3.2.
All three bags of words and the hand-crafted features were each used as
input for a separate logistic regression classifier. Due to the imbalanced class
distribution, a modified class weight was used for these classifiers similar to the
original task paper [
The second model is based on document vectorization by using Paragraph
Vector [
For the Paragraph Vector classification of eRisk users, two separate models
have been trained based on the training documents using the Python
implementation in gensim 1.0.1 [
The output vectors of these two models were concatenated, as recommended by
the developers [
Finally, the average of all documents by each user was calculated to obtain
the average topic of everything the user has written. Figure 4 shows a
twodimensional representation of the averaged training document vectors calculated
by t-SNE [
Class non-depressed depressed
A logistic regression classifier was trained on the 200-dimensional averaged document vectors, using the same class weight equation as in the previous model with w = 4. The calculated class probabilities were again averaged with the probabilities obtained from the logistic regression based on the hand-crafted user features. Since this model depends more on the number of documents it has been trained on, the final predictions were based on the probability as well as the number of documents written by the user to prevent too many false positives. Depressed predictions were submitted for probabilities from 0.6 with at least 20 documents, 0.7 with at least 10 documents, and all probabilities above 0.9, while non-depressed predictions required a probability below 0.1 with at least 20 documents, 0.05 with at least 10 documents, or a probability below 0.01. 4.3
This and the following two models are based on a Tensorflow [
For this first LSTM approach, LSA was used to reduce the BoW vectorized documents to a viable number of dimensions based on Singular Value Decomposition (SVD). All documents were first transformed into a BoW by selecting only the 10,000 unigrams with the highest IG and using their term frequency multiplied by their IG as term weighting. LSA was then used to reduce these document vectors to 100 dimensions, which retained 90.32% of the original variance in the training dataset. To obtain an equal sequence length for all users that is viable as network input, the document sequences were modified to have a length of 25 documents: For users with fewer documents, zero vectors were appended, while two randomly selected consecutive document vectors were averaged for longer sequences, until the maximum length was reached. Adam was then used with a fixed learning rate of 1e−4, 64 units were added to the LSTM cell, a dropout keep probability of 80% was applied, and the network was trained for 300 epochs.
Similar to the previous model, prediction thresholds were based on the network’s output probability and the number of documents. Depressed predictions required a probability above 0.5 and at least 20 documents, above 0.7 and at least five documents, or above 0.9, while non-depressed predictions were submitted for probabilities below 0.05. 4.4
This fourth model utilized the same LSTM network as described for the previous one with identical parameters, except for a number of 128 hidden units in the LSTM cell and a training duration of 170 epochs. For the input sequences, documents were vectorized based on the two concatenated Paragraph Vector models of the second approach. Again, the resulting sequences of 200-dimensional document vectors were modified to have a unified length of 25. The model was configured to submit depressed predictions for any user with a probability above 0.3 and at least 50 documents, above 0.4 and at least 20 documents, or above 0.7, while probabilities below 0.01 resulted in a non-depressed prediction. 4.5
To have some additional time for model optimization and to compare the impact
on the ERDE score, the fifth model was not used to submit any predictions
until the last week. It is identical to the fourth model but uses two new,
200dimensional Paragraph Vector models that were trained on both training and
test documents. This is an unsupervised method that uses only text documents
without any label information. Also, this model uses a second fully connected
layer before the softmax layer, Rectified Linear Unit (ReLU) activation [
The final submissions to the CLEF 2017 early risk detection pilot task were
scored using the ERDE5 and ERDE50 score for early detection tasks defined
by the organizers as well as F1 score. The scores and the underlying precision
and recall values of all models have been published [
1 2 3 4 5 6 7 8 9
10
Week portion of true positives (blue bars only) per model after each week of the task. A horizontal line marks the 52 depressed samples in the test data.
t c70 i 60 s n o i d e r p e v i t i p ) ( f o r e b u n e a l u 50 s o e u40 r t 30 m 20 t10 v i 0 m u C Model
BCSGA BCSGB BCSGC BCSGD
BCSGE % 0 .0 4 6 e i h c30 A 20 0 % 4 score. The results of BCSG are highlighted. This plot is best viewed in electronic form.
Since results for BCSGE are only available for the last week, it was evaluated again for all weeks after the golden truth file was published. For this ex post analysis, separate Paragraph Vector models were trained using the training data and already released test data for each week. If BCSGE had been used from the first week, the results would have been 16.01% in ERDE5, 9.78% in ERDE50, and 0.46 in F1. While this ERDE50 score would have been the third best overall, the other scores show that this is still not well optimized and there are too many false positives. Future work will be used to examine the effect of hand-crafted features and preprocessing methods on the prediction results. A quick ex post analysis using the first two models BCSGA and BCSGB has shown that the selected hand-crafted features at least had a slightly positive effect (13.04% in ERDE5, 9.75% in ERDE50, and 0.63 in F1 for BCSGA without hand-crafted features), with the exception of the ERDE5 score for BCSGB, which would have been marginally better without hand-crafted features in contrast to a much worse F1 score (12.67% in ERDE5, 10.76% in ERDE50, and 0.37 in F1 for BCSGB). 6
The pilot task for early detection of depression has highlighted a variety of
challenges posed by this area of research. These challenges are not limited to the
task of distinguishing actual clinical depression from normal dejected mood as
well as other, more or less related mental disorders like anxiety disorders, PTSD,
or bipolar disorder. In the context of online platforms, there are also several
other frequent false positives that could be observed in this task: relatives of
depressed individuals and therapists offering advice can easily be mistaken for
depressed cases when giving too much weight to single words or phrases. Drug
users (which might indeed be an accompanying factor of depression [
The final results show that all chosen approaches are generally suitable for
early detection of depression and all of them are of interest for future research.
Due to the promising results using Paragraph Vector, optimizing these models
and applying similar word and document embedding methods like fastText [