In recent years, people actively write text messages on social media platforms like Twitter and Reddit. The text shared on social media drives various applications including influenza detection, suicide detection, and mental illness detection. This work presents our approach to early risk detection of depression and anorexia on social media in CLEF eRisk 2018. For the two mental illnesses, our models combine TF-IDF information and convolutional neural networks (CNNs) to identify the articles written by potential patients. The official evaluation shows our models achieve ERDE5 of 10.81%, ERDE50 of 9.22%, and F-score of 0.37 in depression detection and ERDE5 of 13.65%, ERDE50 of 11.14%, and F-score of 0.67 in anorexia detection.
In this work, explore people sharing their opinions, experiences, and feelings, on social
media platforms from Twitter and Reddit. Textual information extraction can be used
for various intelligent applications in the real world such as healthcare, communication,
entertainment, journalism, and advertising. According to data from 2010 to 2018
reported by statista.com1, the number of Facebook users increased from 431 million to
2,234 million, and the number of Twitter users grew from 30 million to 330 million. As
of April 2018, Reddit had about 33 millions of users. In social media, life experiences
and conversation history from a large number of users are recorded. In recent years,
there is a variety of research focused on social media, including hate speech detection
[
In most cases, the detection task can be considered as a classification problem. Various learning models and linguistic features are explored to deal with different goals. For example, the detection of terrorist attack needs to take latency into account because it is extremely important to prevent an attack from happening. Similar situations also 1 https://www.statista.com hold in the detection of illnesses. In CLEF eRisk 20182, two tasks on early risk detection of mental illnesses are conducted. The goal is to find out potential patients of depression and anorexia as early as possible. In other words, we aim not only to accurately predict if a social media user is a patient of depression/anorexia, but also to minimize the revealed user information. In contrast to usual detection tasks, early risk detection is more challenging. In this work, we conduct an analysis on the datasets and propose a neural network-based approach to the two detection tasks. The rest of this paper is organized as follows. Section 2 briefly describes the CLEF eRisk 2018 task and the dataset. We present our model in Section 3. In Section 4, experimental results are discussed. Section 5 concludes this work. 2 2.1
Early risk prediction on the Internet (eRisk), which started since 2017, is a task held in
the Conference and Labs of the Evaluation Forum (CLEF) based on the consideration
that automatic detection models could be applied to identify the risk as early as possible
to help people avoid becoming victims of mental illnesses. In eRisk 2017 [
In this year, eRisk 2018 extends eRisk 2017 by introducing another mental illness, anorexia, to detect. In addition, the dataset of depression detection is also extended. Both tasks are organized in training stage and test stage. The training data is the writing history of users who are labeled as either risk or safe. The test data is composed of ten chunks released sequentially. For each chunk of a user’s data, the model has to make a decision among three choices: (1) The model does not want to emit a decision on this user in this time. (2) The model emits a risk on this user. (3) The model emits a nonrisk on this user. In Chunk 10, the last chunk, the undecided users should be determined as either risk or non-risk. 2.2
In eRisk 2018[
The words with the highest TF-IDF score in the risk and the normal groups in both datasets are listed in Table 3. The top words of the anorexia patients, marked as bold, denote cues to the illness.
For each user, their posts/comments are equally divided into 10 chunks based on the chronological order. Each post/comment or WRITING includes four fields: TITLE, DATE, INFO and TEXT. TITLE is the post title. For a comment, TITLE is always empty. INFO means the source of the message. TEXT is the body of the post/comment. The number of posts/comments varies from user to user. Moreover, there is no consensus on the total time of writing. Since it is difficult to obtain the standardized time as feature, our models take the information from only TITLE and TEXT into account. F-score and ERDE are the major metrics used in CLEF eRisk. Equation 1 shows the formula of F-score, where β = 1. ERDE complementally rewards early alerts because F-score is unaware of time. Equation 2 shows the latency cost function lco(k), where k is the number of textual items giving the answer, also called delay k times, and o is the parameter that controls the cost rate. The relationship between k and o is shown in Fig. 1. For a true negative or a true positive prediction, the ERDE is zero; for a false negative prediction, the ERDE is one; for a false positive prediction, the ERDE is set by Equation 3. In eRisk 2018, the averaged ERDE5 and the averaged ERDE50 are employed to evaluate the performance.
β =
(1 + β2) × true positive (1 + β2) × true positive + β2 × false negative + false positive
1 ( ) = 1 − 1+ − ERDE true positive = ( ) × truepositive (1) (2) (3)
We formulate the detection task as the problem of sentence classification. A classifier
based on convolutional neural network (CNN) [
The dataflow of the training procedure is shown in Fig. 2. We first compute the TFIDF for each word, and remove the words with low TF-IDF score in the sentence. Finally, the sentence classifier is trained with the refined sentences. The details are listed as follows.
Keyword Selection. See Fig. 2 (a), we select the top 300 words with the highest TFIDF, calculated in the risk documents. The toolkit TF-IDF Vectorizer is used to index and convert each word to a unique integer in the range between 1 and 300. Sentence Representation. The contents in TITLE and in TEXT from a WRITING are concatenated as a sequence of words. We discard the words other than the top 300 keywords. The rest of the sequence will be trained to encode as a vector by using the CNN-based sentence encoder. This step is important to convert an instance into a vector and an example of sentence encoding in Figure 2 (b).
Model Training. We regard the posts/comments written by risk users as positive instances, and those written by normal users as negative instances. Then, we train the CNN model3 to identify the potential patients and model architecture is shown in Figure 2 (c).
Based on the binary classification results, we design a strategy to predict the high-risk
users as early as possible. First, we perform the CNN classifier to predict every
post/comment in a chunk of a user. See Table 4. We emit a risk on this user if more
than θ1 of posts/comments are labeled as positive. On the other hand, we emit a
nonrisk on this user if less than θ2 of posts/comments are labeled as negative. Otherwise,
we do not emit on this user except in the last chunk. In the last chuck, we emit a risk on
the user if more than θ3 of posts/comments are labeled as positive. Otherwise, a
nonrisk is emitted. The thresholds θ1, θ2, and θ3 are real values between 0 and 1. We tune
them with the development set.
After the last chunk submitted, scoreboard reports [
There are different leading models in terms of ERDE5, ERDE50, F1, P and R. There
is a tradeoff between the different goals. The model with higher F-score usually suffers
from poor ERDE5. In addition, the performances of the same models in the depression
and the anorexia tasks are inconsistent. This result reveals the difference between these
two mental illnesses. Overall, early risk detection is challenging, especially when
multiobjectives are needed to optimize.
This work shows our proposed model that combines TF-IDF and CNN classification
for early risk detection of depression and anorexia. In CLEF eRisk 2018, our model
achieves decent ERDE5 in both tasks. According to the challenging issues discussed in
this paper, we will explore advanced methodologies for early risk detection. In future
work, we will improve the model according to the knowledge extracted from in-domain
resources such as Diagnostic and Statistical Manual of Mental Disorders (MSD-5) [