The 2017 CLEF eRisk pilot task focuses on automatically detecting depression as early as possible from a users' posts to Reddit. In this paper we present the techniques employed for the University of Arizona team's participation in this early risk detection shared task. We leveraged external information beyond the small training set, including a preexisting depression lexicon and concepts from the Uni ed Medical Language System as features. For prediction, we used both sequential (recurrent neural network) and non-sequential (support vector machine) models. Our models perform decently on the test data, and the recurrent neural models perform better than the non-sequential support vector machines while using the same feature sets.
Depression is responsible for almost 12% of all Years Lived with Disabilities
(YLDs), with nearly 350 million people su ering from it worldwide [
In this paper we present the ve early risk detection models we submitted
for the pilot task. We tried to leverage external knowledge sources beyond the
provided training data. We incorporated the depression lexicon created by [
The pilot task on early risk detection of depression focused on sequential
processing of contents posted by users in Reddit1 [
The task was divided into two stages: training stage and testing stage. The training stage started on November 30, 2016, when the entire text collection of 486 users was released, with their user-level annotations of depression. Participants then had a little over two months to develop their systems. The testing stage started on February 6, 2017 when the rst 10% of texts written by 401 previously unobserved users were released. For the next 9 weeks, new chunks were released, with each chunk including the next 10% of each user's text. After each release, and before the next release, systems had to make a three-way decision for each user: tag the user as depressed, tag the user as non-depressed, or wait to see the next chunk of data. If a user was tagged as either depressed or non-depressed, this decision was nal and could not be changed for future chunks of data. After the release of the 10th (and last) chunk of the data, the decision was two-way: tag the user as depressed, or tag the user as non-depressed. The prediction model was then evaluated based on its correctness and how early in the series of chunks was able to make its predictions. 3
A number of social media websites were considered as potential data sources for this shared task. Twitter2 was discarded because it provided little to no context about the user, is highly dynamic and did not allow them to collect more than 3200 tweets per user, which, in 140-character microblogs, represents only a small amount of text. MTV's A Thin Red Line (ATL)3, a platform designed to empower distressed teens to respond to issues ranging from sexting to cyberbullying, was also considered, but discarded as there were concerns about redistribution and 1 http://www.reddit.com 2 http://www.twitter.com 3 http://www.athinline.org problems regarding obtaining user history. Eventually, Reddit, a social media and news aggregation website, was selected because of its organization of contents among speci c subreddits, and the ease of collecting data using the API provided by Reddit itself.
For each user, the organizers collected the maximum number of submissions they could nd and were allowed to download through the API (maximum 2000 posts and comments per user). Users with less than 10 submissions were discarded. Original redditor IDs were replaced with pseudo user IDs for anonymization, and published along with the title, time and text of the posts.
After the data collection, the users were divided into two cohorts: an
experimental depressed group and a control (non-depressed) group. For the depressed
group, the organizers searched for phrases associated with self-declaration of
depression, such as diagnosed with depression, and then manually examined the
posts to lter down to just those redditors who explicitly said they were diagnosed
with depression by a physician. These self declaration posts were omitted from
the dataset to avoid making the detection trivial. For the non-depressed group,
organizers collected redditors who had participated in depression forums but had
no declaration of depression, as well as redditors from other random subreddits.
Their nal collection contained 531,453 submissions from 892 unique users, of
which 486 users were used as training data, and 401 were used as test data.
Statistics for that dataset are shown in Table 1.
This is a set of unigrams that has high probability of appearing in
depressionrelated posts. The list was collected from [
We used Metamap [
We considered both sequential and non-sequential learning models for the prediction task. For the non-sequential model, we used a support vector machine that observes the user's entire post history at once. For the sequential model, we used a recurrent neural network model that observes each of a user's posts, one at a time. We also used an ensemble model to combine both the sequential and non-sequential models.
As per the shared task de nition, classi ers were given the user's history in chunks (the rst 10% of the user history, then the rst 20%, etc.) and after each chunk, the classi ers were asked to make a prediction of \depressed", \not depressed", or \wait". All our classi ers were trained to make two-way predictions, \depressed" vs. \wait", and if a classi er predicted a user as depressed after seeing the rst n% of the history, that prediction was considered nal and the remaining 100 n% of the history was ignored. On the nal (10th) chunk, all users who had only ever been predicted as \wait" were classi ed as \not depressed". Note that our models never made post-by-post decisions; they always observed the entirety of the n% of the history they were given and then made a single prediction for the entire n%. 5.1
A support vector machine [
For the support vector machine (SVM) models we used, the feature vectors needed to summarize the entire history of the user. We converted the post-level raw count features to user-level proportion features (e.g., converting the number of times depression was used in each post to the proportion of all words in a all of a user's posts that were depression).
We used two out-of-the-box implementations of support vector machines:
{ Weka's implementation of the sequential minimal optimization algorithm [
Sigmoid
GRU GRU
Input Feature Vectors Depressed or Not-Depressed
1
1 + e x
Due to the sequential property of the data, we opted for machine learning
techniques that take advantage of this. We used Recurrent Neural Networks
(RNN) which have been successful in other natural language modeling problems
[
We trained recurrent neural networks that take a sequence of feature vectors, each representing a single post, and predict whether the user is depressed or not. An ensemble learning model takes the outputs of a set of other machine learning algorithms and uses them as inputs for classi cation. They are typically used to improve the performance of individual machine learning techniques by leveraging the di erent strengths of multiple approaches. For this task, We implemented an ensemble learning technique using the probability outputs of the nine individual models (3 from Weka, 3 from LibSVM and 3 from RNN: models used as features either the depression lexicon, Metamap outputs, or both). We used 5-fold cross validation for each model to calculate the probability of each user being depressed and then fed these probabilities to a Naive Bayes classi er, which served as the ensemble classi er. We used Weka's naive Bayes implementation with the default parameter settings. 6
We submitted ve di erent models for the task: { UArizonaA: An SVM model trained using LibSVM with the depression lexicon and Metamap outputs as features. { UArizonaB: An SVM model trained using Weka with the depression lexicon as features. { UArizonaC: An RNN model with both the depression lexicon and Metamap outputs as features. { UArizonaD: The ensemble model. { UArizonaE: An RNN model with the same structure as UArizonaC, but that always predicts \wait" until 60% of the test data is released.
All of these models were selected for their individual properties. UArizonaA was our most restrictive model, as it vigorously tried to not tag someone depressed, whereas UArizonaC was the most open as it tagged more users as depressed than any other models. The other 3 sat in between these 2. To make UArizonaA a Model Brief description E5 E50 F1 P R FHDO-BCSGA Ranked #1 for F1, #2 for E5, #2 for E50 12.82 9.69 0.64 0.61 0.67 UArizonaA LibSVM + lexicon + UMLS 14.62 12.68 0.40 0.31 0.58 UArizonaB WekaSVM + lexicon 13.07 11.63 0.30 0.33 0.27 UArizonaC RNN + lexicon + UMLS 17.93 12.74 0.34 0.21 0.92 UArizonaD Ensemble 14.73 10.23 0.45 0.32 0.79 UArizonaE RNN + lexicon + UMLS + 60%-wait 14.93 12.01 0.45 0.34 0.63 Table 2. Performance of the models. E5 and E50 are the shared-task-de ned Early Risk Detection Error (ERDE) percentages, P is precision, R is recall, and F1 is the harmonic mean of precision and recall. little more open towards depression tagging, we combined its 10th chunk output with UArizonaE's 10th chunk output.
The performance of our models are given in table 2, along with the performance
of the model that achieved the highest F1 in the shared task, FHDO-BCSGA.
The models were evaluated based on 5 performance measures: precision, recall
and F1, and 2 Early Risk Detection Error (ERDE) [
Our models were competitive with others in the shared task. UArizonaC ranked 1st out of 30 for recall, UArizonaD ranked 3rd for ERDE50, and UArizonaB ranked 4th for ERDE5. For precision and F1, our models were less impressive; both UArizonaD and UArizonaE ranked 11th for F1 and UArizonaE ranked 14th for precision. Overall, UArizonaD is the best of our models: it has the highest F1, the lowest ERDE50, and the second-best recall. 7
Our models fell short of the best system in the task for two main reasons. First, we attempted to predict depressed users from the beginning, even though the number of posts varies dramatically from user to user (from only 1 post per chunk to over 200 per chunk). A better strategy would have been to start making predictions after observing some threshold n posts, allowing us to predict early for users with many posts, while waiting until we have more information for users with few posts. Second, we did not su ciently explore the broad range of possible features. For example, we could have built a domain-speci c depression lexicon and used it instead of a previously collected lexicon, or we could have used more sophisticated techniques to represent posts as post-level feature vectors. 8
In this paper, we described the techniques for the University of Arizona submissions to the 2017 CLEF eRisk early risk detection of depression pilot task. We used features based on a depression lexicon and on the Uni ed Medical Language System (UMLS). We implemented sequential and non-sequential models and used ensemble methods to leverage the best of each model. We found that the ensemble model works better than the individual models, and waiting for more data before making a decision improves the traditional performance measures like precision, recall and F1. Whether it is acceptable to wait a decent amount of time to have better performance is still an open question { and we would like to work on that. We would like to establish a timeframe that can be deemed acceptable before making a decision so that the tradeo between correctness and speed of the decision is minimized.
We would like to express our gratitude to the organizers of the pilot task for their e ort towards building the rst annotated user-level depression dataset.