In this paper, we describe our participation in CLEF eRisk workshop. eRisk 2019 is the third edition of this track 3, which was rst introduced in 2017. In the current edition, the organizers are targeting Social Media users, namely Reddit, who may be under the risk of Anorexia, self-harm, and depression. We participated in both tasks of early risk detection of Anorexia and self-harm. Our predictions are based on Natural Language Processing using supervised machine learning with Support Vector Machines (SVM) and neural networks. SVM gave the best results among our ve submitted models with latency-weighted F1 of 0.58 and ERDE5 of 0.08 and ERDE50 of 0.04 for Anorexia detection task, while our more complicated neural network models did not show the desired performances.
NLP
LSTM Social Media (SM) provided a more vibrant environment for communication and sharing experiences. However, di erent means of interactions on SM platforms induced enormous amounts of data of di erent variations. Feeding the data to machine learning and data mining algorithms revealed much of user personality that could not be known even by their close family members 4.
Hence, the users' mental health state is not an exception and might be revealed and understood through the use of SM data as well.
In mental health treatment, professionals use the textual content produced
by people su ering from di erent kinds of mental illness to analyze and help
them in the treatment and diagnosis. This procedure could be costly to do on
site, and it may not result in enough data. Therefore, mining social media could
be a promising method to get more related content for analysis. Besides, it is
not only a cheaper source but also, a reliable way to reach more people that can
be at risk and in need of professional therapy [
eRisk 2019 introduced three collaborative tasks to deal with the early risk detection of di erent mental illnesses, namely Anorexia, self-harm, and depression, using textual data from related Reddit subreddits. We submitted predictions for the rst two tasks. We used ve models for both of the tasks, including variations of SVM and LSTM neural networks. In the following, we introduce our models and results. 2
The data was extracted from Reddit5. This website includes many communities to discuss certain topics and interests in what are called subreddits.
The rst task is related to the early detection of signs of Anorexia. This task has two stages training and testing. Training and Test datasets of the same task in eRisk 2018 were joined to be used as the training set for eRisk 2019. The data consists of users' messages, which are separate posts with a title and a textual content.
In earlier anorexia and depression detection challenges, users' posts were divided into ten chunks each with 10% of each user posts. In 2019 anorexia challenge, the test data is provided in an item-by-item manner. The predicting model must submit a decision for the user, i.e., positive or not, upon receiving each message from their stream. A score indicating the level of Anorexia must also be submitted after receiving each writing.
The second task is related to the early detection of Signs of Self-harm. The source of the data is the same as the rst task with the same test stage format. There is no training data for this task, so it is presented as an unsupervised learning task to motivate search based methods.
More information about the data is provided by [
We rst pre-processed the textual data by lemmatizing, lower casing, and then removing stop words, new-lines, and other unrelated terms or symbols related to the Reddit platform. After the cleaning process, we used di erent machine learning approaches to address the tasks. In our strategy, we aimed to use methods that are as simple as possible, but that led in earlier cases to competitive results [10] and then also try out more complex models that showed successes in related areas [12]. 3.1
For the rst model, we employed a traditional machine learning (ML) model, which is an SVM with a linear kernel. For the training phase, we concatenated all the writings for each labeled user, including text and title parts. Then, we applied a term-frequency transformer and then performed feature selection using chi-squared test to select the most signi cant 500 terms. On the resulting feature vectors from the described pipeline, we performed a 10-fold cross validation using LinearSVC provided by Python sklearn library. We used grid search to determine the SVM model parameters. Besides, we de ned a class weight to overcome the unbalanced data problem. The parameters that were used are: fpenalty = "l2"; dual = T rue; tol = 1e 3; C = 10; class weight = 1 : 4g. 3.2
This method is a two-steps classi cation. The idea is based on the fact that by observing each user's posts, we found that the stream contains many posts that are not related to the mental health issue and discuss other topics like games, movies or politics. Hence, the rst step is to lter out the writings that have a higher probability of being not related to the topics discussed frequently by mental health subreddit users and keep only those that are more likely to be about mental illness related issues. These most related writings pass to the next classi cation step to predict if the user is under risk or not. If no writings pass this lter, the user's label remains negative. In this model, we tried to narrow down the writings to get the ones that have a higher probability of having informative features on the users' mental health and examine if this will enhance the SVM performance by adding a pre- ltering phase.
To perform the rst step, we manually annotated writings in the training set that are not related to mental health. Our guidelines were to lter out any post that does not discuss any of the following issues: depression, food or cooking, physical activity, medications, life experience/story, self-harming/suicidal and eating disorder experiences. These topics are selected based on literature that mentioned topics written by people with mental health issues and by manually observing positive users postings in the data [11, 13]. Two people have performed the data annotation. We rst chose randomly 200 posts and annotated them. The inter-rater agreement computed using Kappa on these 200 posts is = 0:72.
We then extended the dataset and annotated more posts. In total, we have 660 posts. From this 660 posts, 230 (34.85%) are related to mental health, and 430 (65.15%) are not. We trained a linear SVM with similar features to the one trained for the rst model described in Section 3.1, but we selected the most signi cant 200 terms. We trained the SVM using 10-fold cross-validation on the 660 writings. The best parameter combination, found using grid search, produced 0.68 as the mean cross-validated F1 score. These parameters are: ftol = 1e 3; C = 1; class weight = "balanced"; gamma = "scale"g.
We consider this described classi er as a lter which does not allow the writing to pass to the next step if the probability of it not being related to mental health is higher than 95%. We chose this high probability to reduce the loss of true positives.
The second step classi er is responsible for the writings that pass through the previous ltering step. It is the same classi er described in Section 3.1. To train this classi er, we ltered the training data using the step-1 model and used the resulted data for training. 3.3
This model is based on a vanilla LSTM (Long Short Term Memory) neural network, which is a type of recurrent neural network.
Since the eRisk data is a stream of user's posts ordered by time, LSTM could be applicable for classifying users. For training this model, we did not use the full posts stream, but we used only 45 writings of the users' streams by which we took 15 writings from the beginning, another 15 from the middle and another 15 from the end of the stream. We chose the number of writings based on manual observation of the data in order to summarize the users' writing stream. The goal is to have a minimum and a representing number of writings to asses the users' risk. The features extracted from these writings are a concatenation of doc2vec features and term frequency features. We trained the model on the concatenation of all writings (including title and text) for each user.
The doc2vec model produces a 200 long vector which is a concatenation of the results of two doc2vec algorithms, namely Distributed Bag of Words and Distributed Memory with an output of 100 long vector each and trained in the same way described by [14] using Python implementation provided by gensim library. We concatenated the 200 doc2vec feature vector with a vector of term frequencies of the most signi cant 70 terms selected as described in Section 3.1.
We used Keras implementation of LSTM with 500 units for output, a dropout of 0.2, a dense layer of size 2 and Softmax function for the nal output. We used binary cross-entropy for the loss function and Adam as an optimizer. We de ned F1 metric for model evaluation.
In the test stage, the input to the trained model is given one writing at a time. Whenever new writings arrive, we concatenate it with the previous ones and use the concatenation as input to the model. When the stream of writings became longer, we started to arrange the input to concatenate only 45 writings the way we described above. 3.4
Attention is a mechanism used in deep learning models that have been quite
popular lately, and rst appeared in neural machine translation. Attention was
mainly introduced to address the ine ciency of sequence-to-sequence encoding
in memorizing longer sentences [
The model's input is similar to what we described for the previous simple
LSTM model in Section 3.3 but using only the doc2vec feature vectors, i.e.
a doc2vec for each writing. The 45 vectors formed an input layer for another
LSTM layer with 16 units and then followed by an attention layer. We use what
is called global/ soft attention, as described by [9], which is a simpli cation of the
attention mechanism in [
By manually comparing writings streams of positive labeled users and negative labeled ones, we noticed that the positive users' writings contain di erent kinds of topics and information that is more frequent in their feed. These topics, for example, can be related to diet, eating habits like fasting and food, and medications. Hence, the idea is to train a model that learns the importance of each writing depending on its topic.
This idea is similar to the model proposed in [12] for arguments classi cation. In their paper, Stab et al. use an average vector of embedding of each topic's terms to build the attention layer. These topics embeddings allowed to detect whether a sentence constitutes an argument or not by engaging the topic of the sentence in the detection model. For our case, we used doc2vec of paragraphs collected from web sources related to Eating Disorders in addition to posts from ED related r/EatingDisorders/ and r/AnorexiaNervosa/ subreddits, instead of topic terms that were used in the original paper. We used collected paragraphs from eight di erent web sources that contain articles about Anorexia and other eating disorders6,7,8. We used 15 paragraphs from the ED related websites chosen manually to be as diverse as possible to consider di erent aspects of the disorder. Besides, we added 15 writings from the mentioned subreddits. We selected the writings to cover di erent topics that users mention in these subreddits, such as body/weight information, diets, and recovering experiences.
The implemented model includes an inner-attention layer that receives both the input as doc2vev features from 45 writings as described before and the doc2vec feature of 30 paragraphs of related topics and an LSTM with 64 units with a dropout of 0.1 and a dense layer of 2 units for the output using Softmax function.
6https://www.mayoclinic.org/diseases-conditions/anorexia-nervosa/symptomscauses/syc-20353591 7https://www.psycom.net/eating-disorders/anorexia/ 8https://www.eatingdisorderhope.com/information/eating-disorder
We developed the ve models described in the earlier sections for Task 1, namely
Anorexia detection. eRisk organizers evaluate the prediction based on F1,
precision, recall, and ERDE measure, which was rst proposed in [
The organizers have added new ranking-based evaluation measures that are
based on the submitted scores that accompanied each received writing. The goal
is to rank the users according to their estimated risks [
SVM did not perform as it is expected, this could be due to the di erence between the training and testing settings, where the training is done using the stream of the whole writing for each user, whereas the judging is associated with each writing at a time. On the other hand, what is most tricky by this kind of problem is the coding for a user's writing stream. The NN solutions we used are initially used for a stream of words that constitute a sentence that needs to be classi ed, but in our case, we have several paragraphs/sentences that belong to one user which should be classi ed. It is most likely that our input encoding did not t with the test and evaluation strategy of the tasks. On the other hand, as mentioned before, the performance was better in terms of ranking evaluation when using more writings.
In our future work, we aim to investigate methods that suit the item-by-item test phase. Moreover, we need to investigate better encoding for the user input without losing much knowledge by encoding each post as one unit of information. 6
acknowledgement This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - GRK 2167, Research Training Group User-Centred Social Media. 9. Luong, M.T., Pham, H., Manning, C.D.: E ective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015) 10. Ramiandrisoa, F., Mothe, J., Benamara, F., Moriceau, V.: Irit at e-risk 2018. In:
E-Risk workshop. pp. 367{377 (2018) 11. Resnik, P., Armstrong, W., Claudino, L., Nguyen, T., Nguyen, V.A., Boyd-Graber, J.: Beyond lda: exploring supervised topic modeling for depression-related language in twitter. In: Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. pp. 99{107 (2015) 12. Stab, C., Miller, T., Gurevych, I.: Cross-topic argument mining from heterogeneous sources using attention-based neural networks. arXiv preprint arXiv:1802.05758 (2018) 13. Toulis, A., Golab, L.: Social media mining to understand public mental health. In: VLDB Workshop on Data Management and Analytics for Medicine and Healthcare. pp. 55{70. Springer (2017) 14. Trotzek, M., Koitka, S., Friedrich, C.M.: Linguistic metadata augmented classi ers at the clef 2017 task for early detection of depression. In: CLEF (Working Notes) (2017)