The CLEF eRisk 2018 challenge focuses on early detection of signs of depression or anorexia using posts or comments over social media. The eRisk lab has organized two tasks this year and released two di erent corpora for the individual tasks. The corpora are developed using the posts and comments over Reddit, a popular social media. The machine learning group at Ramakrishna Mission Vivekananda Educational and Research Institute (RKMVERI), India has participated in this challenge and individually submitted ve results to accomplish the objectives of these two tasks. The paper presents di erent machine learning techniques and analyze their performance for early risk prediction of anorexia or depression. The techniques involve various classiers and feature engineering schemes. The simple bag of words model has been used to perform ada boost, random forest, logistic regression and support vector machine classi ers to identify documents related to anorexia or depression in the individual corpora. We have also extracted the terms related to anorexia or depression using metamap, a tool to extract biomedical concepts. Theerefore, the classi ers have been implemented using bag of words features and metamap features individually and subsequently combining these features. The performance of the recurrent neural network is also reported using GloVe and Fasttext word embeddings. Glove and Fasttext are pre-trained word vectors developed using speci c corpora e.g., Wikipedia. The experimental analysis on the training set shows that the ada boost classi er using bag of words model outperforms the other methods for task1 and it achieves best score on the test set in terms of precision over all the runs in the challenge. Support vector machine classi er using bag of words model outperforms the other methods in terms of fmeasure for task2. The results on the test set submitted to the challenge suggest that these framework achieve reasonably good performance.
Early risk prediction is a new research area potentially applicable to a wide
variety of situations such as identifying people with mental illness over social media.
Online social platforms allow people to share and express their thoughts and
feelings freely and publicly with other people [
In this paper, di erent machine learning frameworks have been proposed to
accomplish the given tasks. The aim is to train a machine learning classi er
using the training set to identify anorexia or depression of the individual
documents of the test sets of these tasks. The performance of a text classi cation
technique is highly dependent on the potential features of a corpus. Therefore
the performance of di erent classi ers have been tested using both text
features and biomedical features extracted from the given corpus. In general, each
unique term of a corpus is considered as a feature and therefore the frequency
of the individual terms are considered to form the document vectors [
The empirical results for the rst task demonstrate that the ada boost, random forest and support vector machine classi ers using BOW features outperform the other frameworks using UMLS features and combining BOW and UMLS features. Furthermore, ada boost classi er using BOW features outperforms the other methods and it achieves best score on the test set in terms of precision over all the submissions in the eRisk 2018 challenge. For the second task, the experimental results show that the support vector machine classi er using BOW features outperforms the other frameworks using both UMLS features and combining BOW and UMLS features. The results on the test set submitted to the challenge suggest that these frameworks for task2 achieve reasonably good performance. However, there are some submissions for this pilot task, which beat the performance of this framework.
The paper is organized as follows. The proposed machine learning frameworks are explained in section 2. Section 3 describes the experimental evaluation. The conclusion is presented in section 4. 2
Various machine learning techniques have been proposed here to identify the documents related to anorexia from the given corpus, which is released in XML format. Each XML document contains the posts or comments of a Reddit user over a period of time with the corresponding dates and titles. We have extracted the posts or comments from the XML documents and ignored the other entries. Therefore the corpus used for experiments in this article contain only the free texts related to di erent posts over Reddit for individual users. Di erent types of features are considered to build the proposed frameworks to identify anorexia or depression of the individual documents using state of the art classi ers. 2.1
Di erent feature engineering techniques exist in the literature of text mining. We have considered both raw text features and semantic features in the proposed methods. 2.1.1
Bag Of Words (BOW) Features
The text documents are generally represented by the bag of words (BOW) model
[
Let us denote the number of documents of the corpus and the number of terms of the vocabulary by N and n respectively. The number of times the ith term ti occurs in the jth document is denoted by tfij ; i = 1; 2; :::; n; j = 1; 2; :::; N . Document frequency dfi is the number of documents in which a particular term appears. Inverse document frequency determines how frequently a term occurs in a corpus and it is de ned as idfi = log( dNfi ). The weight of the ith term in the jth document, denoted by wij , is determined by combining the term frequency with the inverse document frequency as follows:
N dfi wij = tfij idfi = tfij log(
); 8 i = 1; 2; :::; n and 8 j = 1; 2; :::; N
This weighting scheme is known as tf-idf weighting scheme. The documents can
be e ciently represented using the vector space model in most of the text mining
algorithms [
UMLS Features
We have also considered the UMLS concepts extracted from the text as features.
The UMLS stands for Uni ed Medical Language System and it is a
comprehensive list of biomedical terms for developing automated systems capable of
understanding the specialized vocabulary used in biomedicine and health care [
For the rst task we have retained only those terms related to some
manually selected semantic categories related to depression, namely, mental health
and behavioral dysfunctions, abnormalities, diagnostic procedures, signs and
symptoms, and ndings. For the second task, the terms belonging to the UMLS
concepts, namely, Protein, Activity, Disease, Food, Individual Behavior, Social
Behavior, and Vitamin are considered in the experiments as the other semantic
categories in UMLS are not related to eating habits or eating disorders.
MetaMap also normalizes the identi ed concepts of a term and provides a
concept unique identi er (CUI) for each of the concepts [
2.2
Di erent text classi cation methods have been implemented to identify
depression or anorexia in the given corpus using the BOW features and UMLS features
individually and by combining them. The proposed frameworks are developed
using ada boost, logistic regression (LR), Random Forest (RF), Support Vector
Machine (SVM) and recurrent neural network (RNN) classi ers.
SVM is widely used for text categorization [
Logistic regression performs well for binary class classi cation problem [
The Ada boost algorithm is an ensemble technique, which can combine many
weak classi ers into one strong classi er [
RNN is an useful classi er for sequential data because each neuron or unit can
use its internal memory to maintain information about the previous input. This
allows the network to gain a deeper understanding of the statement. In
principle, RNN can handle context from the beginning of the sentence which will
allow more accurate predictions of a word at the end of a sentence [
3.1 3.1.1
Task1
The corpus released as part of the rst task is a collection of posts or comments
from a set of users over Reddit [
Task2
The corpus released as part of task2 is also a collection of posts or comments
from a set of users over Reddit [
Ada boost, LR, RF and SVM classi ers are implemented in Scikit-learn6, a
machine learning tool in Python [
Task1 The data of the same challenge in 2017 has been released as the training set for this task. The corpus of the 2017 challenge was divided into training set and test set. The ground truths were available for both training and test set. We have used this training set to train di erent classi ers of the proposed frameworks in this article. The parameters of di erent classi ers are tuned using 10-fold cross validation technique on this training set of 2017 challenge. The test data of 2017 challenge is used as the validation set to evaluate the performance of the classi ers of the proposed frameworks using the ground truths. The classi ers using a particular type of features that perform the best on the validation set are chosen for implementation on the test set of this year. Subsequently, the results of the proposed frameworks on this test set have been submitted to the eRisk 2018 challenge.
3.2.2 The given training set of second task is further divided into two parts namely, training set and validation set. The new training set is build by randomly choosing 80% documents individually from anorexia and non-anorexia categories. Similarly the rest 20% of these categories form the validation set. The parameters of di erent classi ers are tuned using 10-fold cross validation technique on the newly formed training set and therefore the performance of these classi ers are tested on the validation set. The classi ers using a particular type of features that had shown better results than other such frameworks are submitted to the challenge. 3.3
The performance of the proposed method and the state of the art classi ers are evaluated by using the standard precision, recall and fmeasure and ERDE. The precision and recall for two class classi cation problem can be computed as follows:
Precision =
TP
TP+FP Recall =
TP
TP+FN Here TP stands for true positive and it counts the number of data points correctly predicted to the positive class. FP stands for false positive and it counts the number of data points that actually belong to the negative class, but predicted as positive (i.e., falsely predicted as positive). FN stands for false negative and it counts the number of data points that actually belong to the positive class, but predicted as negative (i.e., falsely predicted as negative). TN stands for true negative and it counts the number of data points correctly predicted to the negative class. The fmeasure combines recall and precision with an equal weight in the following form:
Fmeasure = 2
recall precision
recall + precision
The closer the values of precision and recall, the higher is the fmeasure.
Fmeasure becomes 1 when the values of precision and recall are 1 and it becomes 0
when precision is 0, or recall is 0, or both are 0. Thus fmeasure lies between 0
and 1 [
if d = negative AND ground truth=negative (TN)
The values of cfp and cfn depend on the application domain and the
implications of FP and FN decisions. The function lco(k) is a monotonically increasing
function of k, which is parameterized by o. The minimum value of o is considered
as 5 and the maximum value as 50. Note that ERDE lies in range [
It may be noted from Table 3 and Table 4 that the performance of all the classi ers using BOW features are better than the same using UMLS features. Moreover, Table 3 and Table 5 show that all the classi ers using the BOW features perform better than the same using the combination of BOW and UMLS features. This indicates that UMLS features have little in uence on the performance of the classi ers. It is manually checked that the number of UMLS features are too small and there are absence of biomedical terms related to depression in the documents. This may be the reason of poor performance. Consequently, we have submitted the results of Ada Boost, LR, RF and SVM classi ers using BOW features on the test set to the challenge.
We have also submitted a result of RNN using Fasttext embedding, as RNN has been widely used for text categorization in recent years. However, the performance of RNN on the validation set is not as good as the other classi ers using BOW features. The precision, recall and fmeasure of the same is 0.64, 0.60 and 0.62 respectively. Note that we have xed the sequence length of each sentence considered by RNN as 150 due to the limitation in the resources. The results of RNN may be improved by increasing the sequence length in the model, which is beyond the scope of this article.
The results of Ada Boost, LR, RF and SVM classi ers using BOW features and RNN classi er using Fasttext embedding on the given test set in terms of ERDE5, ERDE50, precision, recall and fmeasure are reported in Table 6. RKMVERIA, RKMVERIB, RKMVERIC, RKMVERID indicate the results of LR, SVM, Ada boost, and RF classi ers respectively using BOW features. RKMVERIE indicates the result of RNN classi er using fasttext embedding. Table 6 shows that precision of the RKMVERIC framework is better than the precision of the other RKMVERI frameworks and RKMVERIC achives the best score in terms of the precision of 45 submissions in the eRisk 2018 challenge. It can be seen from Table 6 RKMVERID performs better than other RKMVERI frameworks in terms of fmeasure and the same is the fourth best fmeasure in the competition. 3.4.2
Task2 We have reported the performance of Ada Boost, LR, RF and SVM classi ers on the validation set using BOW features, UMLS features and the combination of BOW and UMLS features respectively in Table 7, Table 8 and Table 9. The performance of these classi ers are measured in terms of fmeasure in these tables. It can be seen from Table 7 that the performance of SVM is better than the other classi ers in terms of precision recall and fmeasure. Table 8 shows that the performance of SVM is the best among all other classi ers in terms of fmeasure. It can be observed from Table 9 that Ada Boost classi er outperforms other classi ers in terms of fmeasure.
It may be noted from Table 7 and Table 8 that the performance of all the Text Classi ers Ada Boost Logistic Regression Random Forest Support Vector Machine classi ers using BOW features are better than the same using UMLS features. Moreover, Table 7 and Table 9 show that all the classi ers using the BOW features perform better than the same using the combination of BOW and UMLS features. This indicates that UMLS features have little in uence on the performance of the classi ers. We have manually checked that the number of UMLS features are too small, which may be a reason of poor performance. Consequently, we have submitted the results of Ada Boost, LR, RF and SVM classi ers using BOW features on the test set to the challenge. We have also submitted a result of RNN using GloVe embedding, as RNN has been widely used for text categorization. However, the performance of RNN on the validation set is not as good as the other classi ers using BOW features. The fmeasure of the same is 0.56. The results of Ada Boost, LR, RF and SVM classi ers using BOW features and RNN classi er using GloVe embedding on the given test set in terms of ERDE5, ERDE50, precision, recall and fmeasure are reported in Table 10. RKMVERIA, RKMVERIB, RKMVERIC, RKMVERIE indicate the results of SVM, LR, RF, and Ada Boost classi ers respectively using BOW features. RKMVERID indicates the result of RNN classi er using GloVe embedding. Table 10 shows that precision of the RKMVERIC framework is better than the precision of the other RKMVERI frameworks and it is the fourth best score among the precision of 35 submissions in the eRisk 2018 challenge. RKMVERIA performs better than other RKMVERI frameworks in terms of ERDE5, ERDE50, recall and fmeasure. 4
The eRisk 2018 shared task highlights a variety of challenges for early detection of depression and anorexia using the data over social forums. Depression is a type of mental disorder that has adverse a ects on feelings, thoughts and behaviors and can harm regular activities like sleeping, working etc. Anorexia is also a mental disorder distinguished by a refusal to maintain a normal body weight, intense fear of weight gain and disturbance in the perception of body shape and weight. However, it is generally di cult to identify depression or anorexia from di erent symptoms. The treatment for these diseases can be started on time, if the alarming symptoms are diagnosed properly. The aim of this challenge is to detect signs of such diseases from the posts or comments of individuals over social media. Various machine learning frameworks have been developed using di erent types of features from the free text to accomplish this task. We have examined the performance of both bag of words features and UMLS features using di erent classi ers to identify depression. However, it is observed that a few UMLS features exist in the corpus. Hence the proposed methodologies relied on the BOW features. The experimental results show that the performance of these methodologies are reasonably good. We have also implemented the RNN classier using the Fasttext and GloVe word embeddings. However, the performance of these RNN models are not so good as we have to x the sequence length of each sentence as 150 only due to limitation of resources. In future, we can implement RNN using higher length of word embeddings for better performance.