This paper presents the systems developed by the UQAM team for the CLEF eRisk Pilot Task 2017. The goal was to predict as early as possible the risk of mental health issues from user-generated content in social media. Several approaches based on supervised learning and information retrieval methods were used to estimate the risk of depression for a user given the content of its posts in reddit. Among the five systems evaluated, the experiments show that combining information retrieval and machine learning approaches gives the best results.
The Early Detection of Depression Pilot Task was part of the CLEF eRisk 2017 workshop [16]. The pilot task challenge consists of performing early risk detection of depression by analyzing user-generated content from reddit 1 . Towards this goal, a system receives user-generated content as input, and should output a prediction regarding the user's susceptibility to depression. The pilot task dataset contains user-generated content, which is organized and processed chronologically. This allows for monitoring the user progress, and detecting risk as early as possible. Users are categorized as risk or non-risk (of depression). Each user produced a sequence of reddit posts, written within a given period of time. The pilot task was organized in two stages: training and test, each having a different dataset divided into 10 chunks. During training stage, a dataset containing a sequential set of posts per user was provided along with the user's category. All training chunks were made available, containing the complete user post sequence. During test stage, the dataset of test users was released sequentially (one release each week). Each release contained part of the user post sequence, corresponding to one chunk (from the oldest to the newest posts). Participant systems had to output predictions for users based on all current test chunks before the release of a new chunk. The predictions could be either the category of a user or no decision, up to the last week of the test stage where all the users had to be given a category.
We describe hereafter our prediction system based on an ensemble classification approach, which combines supervised learning, information retrieval, and feature selection methods. This report is organized as follows: the system resources are described in Section 3; the system modules, and the decision algorithm merging the module predictions are described in Section 4. Experiments and results are described in Section 5 while conclusions and future works are discussed in Section 6.
Social media content has been commonly utilized to develop approaches that support mental health care. The latest CLPsych Shared Tasks [5,18] have proposed participants to predict users in eminent risk of depression or Post Traumatic Stress Disorder (PTSD). These tasks made use of tweets or mental health forum posts. In [11], a sentiment analysis model was built with focus on user-generated social media content. It uses highly relevant sentiment lexicons and sentiment intensity measurements. The authors demonstrated that the approach outperforms other commonly used lexicons, as well as machine learning-based tools. The authors of [19] evaluated the usage of different features to analyze user posts from LiveJournal 2 , and compare discrepancies between posts from depression related online communities, and control (non-depression) related communities. Another approach was proposed by [17], relying on a statistical model based on the analysis of over 176 million tweets to identify communication patterns related to mental illness in Twitter, and to attempt predicting user behavioral patterns related to depression. We describe hereafter studies conducted mainly based on two research fields: supervised learning, and information retrieval.
Several studies were conducted towards identifying mental health issues in social media by using supervised learning methods. The choice of supervised algorithms varies according to the tasks and data at hand. However, the previous studies presented here generally rely on highly discriminative features to achieve state-of-the-art performance. This demonstrates the importance of attribute choice for such tasks. In [8], the authors presented a study on predicting depression from tweets by analyzing over 2 million posts of 476 users. The best performance was obtained with a SVM classifier and a set of behavioral features, such as occurrence of pronouns, usage of swearing and depression terms, tweet replies, as well as posting time and frequency. The work presented in [14] identifies user psychological stress in tweets. Features such as emotion words, smileys, tweet mentions, replies, and posting frequency were obtained from single tweets, and from all user's tweets. The best performance was obtained by a four layer Deep Neural Network (DNN). Previous works have also used Twitter data to identify language differences between users potentially presenting PTSD [6], or who attempted suicide [7]. In both these studies, the authors evaluated user-generated content using word and character language models. The findings point to characteristics of tweets associated to mental health issues, such as heavier use of emotions, usage of third person pronouns, anxiety terms, as well as high posting frequency. The authors in [23] analyzed Facebook3 status updates to predict user satisfaction with life. Their approach used feature selection of n-grams and topic extraction, aand built regression models based on the message level, and the user level. The results indicate that a cascade model, using message level predictions to inform user level predictions, performed best.
Information retrieval techniques are widely used to support knowledge discovery in the biomedical field. Most of the approaches are designed to help researchers and practitioners looking for relevant documents to support experiments or diagnoses. In the field of mental health, [10] reports an interesting study to support mental health maintenance of U.S. army soldiers. The goal is to aid health practitioners to perform efficient follow-ups on soldiers, since the suicide attempt rate among them is known to be high. The approach made use of the Veterans Informatics and Computing Infrastructure (VINCI) resource to process mostly unstructured health information, such as clinical notes. The authors built a search engine based on Apache Solr4 indexing these textual data to predict the risk of suicide attempt among soldiers. Even though only few pre-processing steps were utilized in this system, it provides promising performance, and covers a larger population than systems based on structured data.
The following Sections describe the resources utilized to build our systems.
The supervised learning-based systems rely on a set of depression-related dictionaries. The dictionary keywords are used to provide discriminative attributes for automatic classification. The dictionaries we utilized are lists of relevant feelings, medicine, drugs, and diseases, which are assumed to be related to depression. The feeling dictionary is composed of feeling words used in mental status exams 5 , and a conceptual feature map obtained from SenticNet [4]. The medicine dictionary lists antidepressant names or depression-related medicine, obtained from Wikipedia 6 . The disease dictionary is composed of depression-related disease names, from Wikipedia7 .
The drug dictionary contains a list of psychoactive drug names, such as hallucinogens, psychedelics, anxiolytics, and sedatives, also obtained from Wikipedia8 .
Classification To support developing the supervised learning method in our system, we have utilized the open-source machine learning framework Weka [24] 9 . The Weka framework provides standard implementations of several classification algorithms. It also provides modules to handle and process Attribute Relation Format Files (ARFF) files, which contain a matrix representation of the dataset in terms of instances versus features, allowing to easily perform feature selection.
Indexating The information retrieval method in our system relies on the open-source search platform Apache Solr. The Solr platform allows for building a search engine to perform full-text search in a document index. Both Solr search and index modules are built based on the Apache Lucene10 library. A Solr index is designed based on a schema, which is composed of a set of fields that represent a document object. Several pre-processing steps are also available in Solr, which can be applied at indexing time and also at query time.
To detect users in risk of developing depression, we have designed a multipronged approach that combines results obtained from both Information Retrieval (IR) and Supervised Learning (SL) based systems. The combination is performed by a decision algorithm. In Section 4.1, we explain how we utilized the CLEF eRisk training and test datasets in our experiments. The IR-based systems are described in Section 4.2 while the SL-based systems are presented in Section 4.3. Details on the decision algorithm are presented in Section 4.4. Finally, we briefly describe how we performed experiments to determine the best configuration for our approach in Section 4.5.
The CLEF eRisk training and test datasets are composed of user posts extracted from reddit. Both datasets are divided into a total of 10 chunks each, chronologically organized. Each chunk represents a sequence of writings for a given user in a period of time. Table 1 shows statistics on the eRisk 2017 pilot task datasets. We have utilized the chronological aspect of the user writings when processing both training and test data. When processing the training data, we have computed the user posting frequency, which is further described in Section 4.3. When processing the test data, we have considered single chunk and multiple chunk predictions, as further explained below. In order to output predictions in a given week, we have utilized the test data in two different ways: first, to obtain a list of predictions only considering the current test chunk; second, to obtain a list of predictions considering all test chunks released so far. Both list of predictions are taken into account when merging outputs from different models and systems.
We used an approach based on IR to retrieve similar documents from a test document used as a query. The intuition is that using the full content of a user post as a query should allow a search engine to retrieve semantically similar documents (posts). In our context, the similar posts are retrieved from the training corpus where they are already labeled according to the risk/no-risk state of the user who wrote them. We built two search engines relying on two different indexes created from the eRisk training corpus with and without indexing stop-words. We then considered the eRisk test documents as queries, which were submitted to both search engines.
For each test document d submitted to the search engines, we used the class (risk or non-risk) of the top n retrieved documents to compute a score S IR (d) reflecting how likely d has been produced by a depressed user. This can be compared to a k-nearest neighbors approach since we want to get the closest documents (neighbours) to a given document. The number of retrieved documents taken into account has been set experimentally to n = 20. S IR (d) is computed as follows:
where d i is the document retrieved by query d in position i, and
The test documents are then ordered according to their S IR score, and considered as risk candidates if their score is above a given threshold, which was experimentally set.
The search engines created in this approach rely on Apache Solr, and the BM25 probabilistic ranking algorithm [12]. We first indexed all the fields in the training set. Two indexes, I and II, were generated based on the same schema but applying different preprocessing steps, which are described in Tables 2 and 3.
For Index I, we indexed all the data with little pre-processing. Index II uses the same schema along with more pre-processing steps: stop-words removal, stemming (using the Solr built-in Porter Stemmer algorithm), and punctuation filtering.
Lowercasing Index II As Index I +
Table 2. Pre-processing steps by indexes Table 3 presents the fields used in the schema, i.e. all the fields available in the corpus (title, content, date, label). The Text field is a copy field that contains both content and title, and is used as the default search field. For better handling document-based queries, we utilized the built-in Solr MoreLikeThis (MLT) component 11 . Solr MLT enables retrieving documents that are similar to a given document, and is far more efficient compared to other classical search endpoints. 3. Indexed fields
The SL-based approach is based on the combined predictions of several classification models with different configurations. The SL models are designed using four classification algorithms and various feature types described below.
Features To design models for the SL-based systems, we have extracted discriminative features from the pilot task training dataset. Before extracting features, pre-processing steps were performed. These include word stemming, and normalization of URLs, smiley characters, as well as punctuation. The URLs and smiley normalization are relevant to better process the user-generated content, and help portraying the sentiment associated with a post. URLs can contain picture names, or words that refer to specific subjects. Smiley symbols are often used to represent an emotion, and during pre-processing they are replaced by actual words (e.g., :) or :-) are replaced by happy). All these cues are important since, if present, they might help representing a user's state of mind.
After pre-processing, four different feature types were extracted: n-grams, dictionary words, selected Part-Of-Speech (POS), and user posting frequency. N-gram features were extracted as of Bag-Of-Words (BOW), bigrams, and trigrams. Dictionary words were extracted based on the depression-related dictionaries described in Section 3.1. POS features were extracted by selecting the words annotated by the Stanford POS Tagger12 as either adjective (JJ), noun (NN), predeterminer (PDT), particle (RP), or verb (VB).
As an attempt to account for the temporal evolution of the psychological state of a given user, we computed the user posting frequency, which represents the user activity pattern. The posting frequency of a user is computed as the time lapse between the oldest and the most recent writings, divided by the number of writings a user has generated in total. Statistics on features extracted from the training set are presented in Table 4.
Classifiers To build the SL models we have used three classification algorithms: Logistic Model Tree (LMT) [13], an Ensemble of Sequential Minimal Optimization (SMO) [20] (ens SMO) classifiers, and an Ensemble of Random Forests [2] (ens RF) classifiers. The ensembles are composed of 30 different classifiers each. The 30 Random Forest classifiers composing the ens RF were designed with iteration values from 10 to 50 (with increments of 10), and tree depth values from 2 to 10 (with increments of 2), as well as unlimited.
The 30 SMO classifiers composing the ens SMO were designed with tolerance parameter values from 0.001 to 0.005 (with increments of 0.001), and epsilon for round-off error values from 1 to 5 (with increments of 1).
The decision algorithm merges the predictions from both IR and SL based systems. The IR-based candidates are ranked based on similarity, and each candidate is associated with a S IR score, as described in Section 4.2. Documents with highest scores are considered as candidates for the risk class. For the eRisk task, the high score threshold has been experimentally set to 0.7, i.e. all the candidates are documents d with a score S IR (d) such that S IR (d) ≥ 0.7.
The SL-based approaches are used to refine the list of candidates proposed by the IRbased systems. To be selected, a document from the IR-based list must be classified as risk by at least one of the SL-based systems. Candidates proposed by the SL-based system are also ordered according to the confidence of the prediction, and first ranked candidates are selected regardless of their presence in the IR list. The decision function ∆ can be formalized as follows:
where d is a test document, and 1 IR , 1 SL , 1 SLf are the indicator function respectively associated to the IR-based, the SL-based, and the SL-ranked-first lists of candidates. If ∆(d) ≥ 2, the document d is assigned the risk class, i.e. the user who generated this content is susceptible to depression.
In order to determine the most suitable configuration for the IR and SL based systems, as well as the threshold for the decision algorithm, we have performed several experiments utilizing the pilot task training data. The classification models were selected after performing experiments with all three classifiers using all feature types, or several feature types combined. Only the best performing combination of feature sets and classifiers were kept for the SL-based systems.
For the experimental evaluation, the pilot task training dataset was utilized as described in Section 4.
We submitted predictions on the test dataset obtained by five different systems. Four of these systems rely on a different ensemble configuration. The ensembles are either a merge of results obtained from the SL and IR based systems, or from a group of SL classifiers or IR-based systems. The five presented systems are described here:
-UQAMA is based on an ensemble approach, merging the output candidates from all SL-based systems (considering three classifiers and all features), with the output candidates from the IR-based systems. -UQAMB is based on candidates proposed by both IR-based systems only. We considered UQAMB as our baseline system.
-UQAMC is based on SL models built with a LMT classifier, and using as features either BOW or bigrams separately, and BOW or bigrams together with all the dictionary features. -UQAMD is based on SL models built with an ens RF classifier, using as features either BOW or bigrams together with all the dictionary features. -Lastly, UQAME is based on SL models built with an ens SMO classifier, using bigrams separately and together with all the dictionary features.
The user posting frequency was a feature used by all five systems. Table 5 present the results obtained by the five systems in terms of the metrics utilized by the CLEF eRisk pilot task. Besides F1, Precision, and Recall, the pilot task also evaluated systems using the early risk detection error (ERDE) [15]. The EDRE metric accounts for the imbalance problem on automatic classification, which could bias some classifiers. Additionally it penalizes late risk detection using a specific cost function, considering only the true positive scores, which are related to only the relevant (risk) documents.
In total, 8 teams participated in the CLEF eRisk 2017 pilot task, submitting a total of 30 different systems [16]. In obtained by our systems. Among our five presented systems, the best overall performance was achieved by UQAMA with the best F1 score and Recall. The best Precision was achieved by UQAMD, which is designed based on an ens RF classifier. The contribution of each method to the performance of UQAMA needs to be further evaluated, as well as the impact of the various experimental settings. Finally, an interesting observation was drawn from analyzing the user posts of candidates predicted as risk by our systems. The post content of such candidates often presented two major topics: "video games", and "sexuality or relationship issues". The relationship between "depression" and these two topics has been studied from a clinical perspective in several recent works [21,3,9,22]. Interestingly, the co-occurrence of these topics with risk of depression was also spotted by our systems.
This report describes the early risk prediction systems submitted to the CLEF eRisk 2017 pilot task. The system that performed best is based on a multipronged approach, which combines predictions from SL and IR based systems. SL-based systems made use of four major feature types, and three classification algorithms, LMT, ensemble SMO and ensemble RF. IR-based systems utilize two indexes, and users are ranked according to a similarity score based on the BM25 ranking algorithm [12]. The predictions obtained from both SL and IR based systems are merged by a decision algorithm. The results demonstrate that combining SL and IR approaches outperforms the results obtained by each approach applied separately.
Future work During our experimental phase, we have performed preliminary tests to evaluate the usage of three other methods: (1) simple rule-based classification using a sentiment analysis library, (2) deep learning-based classification using a Recurrent Neural Network (RNN), and (3) topic extraction using Latent Dirichlet Allocation [1]. Improving the system performance will involve further investigation of these approaches, as well as enhancement of the IR-based resources of the system.
Reproducibility Our system is publicly released as an open source software, and can be accessed at: https://github.com/BigMiners/eRisk2017