This paper provides an overview of eRisk 2018. This was the second year that this lab was organized at CLEF. The main purpose of eRisk was to explore issues of evaluation methodology, effectiveness metrics and other processes related to early risk detection. Early detection technologies can be employed in different areas, particularly those related to health and safety. The second edition of eRisk had two tasks: a task on early risk detection of depression and a task on early risk detection of anorexia.
The main purpose of this lab is to explore issues of evaluation methodologies, performance metrics and other aspects related to building test collections and defining challenges for early risk detection. Early detection technologies are potentially useful in different areas, particularly those related to safety and health. For example, early alerts could be sent when a person starts showing signs of a mental disorder, when a sexual predator starts interacting with a child, or when a potential offender starts publishing antisocial threats on the Internet. In 2017, our main goal was to pioneer a new interdisciplinary research area that would be potentially applicable to a wide variety of profiles, such as potential paedophiles, stalkers, individuals with a latent tendency to fall into the hands of criminal organisations, people with suicidal inclinations, or people susceptible to depression.
The 2017 lab had two possible ways to participate. One of them followed a classical
workshop pattern. This workshop was open to the submission of papers describing test
collections or data sets suitable for early risk prediction or early risk prediction
challenges, tasks and evaluation metrics. This open submission format was discontinued
in 2018. eRisk 2017 also included an exploratory task on early detection of depression.
This pilot task was based on the evaluation methodology and test collection presented in
a CLEF 2016 paper [
In 2018, the lab followed a standard campaign-style format. It was composed of two different tasks: early risk detection of depression and early risk detection of anorexia. The first task is a continuation of the eRisk 2017 pilot task. The teams had access to the eRisk 2017 data as training data, and new depression and non-depression test cases were extracted and provided to the participants during the test stage. The second task followed the same format as the depression task. The organizers of the task collected data on anorexia and language use, the data were divided into a training subset and a test subset, and the task followed the same iterative evaluation schedule implemented in 2017 (see below). 2
This is an exploratory task on early detection of signs of depression. The challenge consists of sequentially processing pieces of evidence –in the form of writings posted by depressed or non-depressed users– and learn to detect early signs of depression as soon as possible. The lab focuses on Text Mining solutions and, thus, it concentrates on Social Media submissions (posts or comments in a Social Media website). Texts should be processed by the participating systems in the order they were created. In this way, systems that effectively perform this task could be applied to sequentially track user interactions in blogs, social networks, or other types of online media.
The test collection for this task has the same format as the collection described in
[
The task was organized into two different stages: – Training stage. Initially, the teams that participated in this task had access to some training data. In this stage, the organizers of the task released the entire history of submissions done by a set of training users. All chunks of all training users were sent to the participants. Additionally, the actual class (depressed or non-depressed) of each training user was also provided (i.e. whether or not the user explicitly mentioned that they were diagnosed with depression). In 2018, the training data consisted of all 2017 users (2017 training split + 2017 test split). The participants could therefore tune their systems with the training data and build up from 2017’s results.
The training dataset was released on Nov 30th, 2017. – Test stage. The test stage had 10 releases of data (one release per week). The first week we gave the 1st chunk of data to the teams (oldest submissions of all test users), the second week we gave the 2nd chunk of data (second oldest submissions of all test users), and so forth. After each release, the teams had to process the data and, before the next week, each team had to choose between: a) emitting a decision on the user (i.e. depressed or non-depressed), or b) making no decision (i.e. waiting to see more chunks). This choice had to be made for each user in the test split. If the team emitted a decision then the decision was considered as final. The systems were evaluated based on the accuracy of the decisions and the number of chunks required to take the decisions (see below). The first release of test data was done on Feb 6th, 2018 and the last (10th) release of test data was done on April 10th, 2018.
Table 1 reports the main statistics of the train and test collections. The two splits are unbalanced (there are more non-depression cases than depression cases). In the training collection the percentage of depressed cases was about 15% and in the test collection this percentage was about 9%. The number of users is not large, but each user has a long history of submissions (on average, the collections have several hundred submissions per user). Additionally, the mean range of dates from the first submission to the last submission is wide (more than 500 days). Such wide history permits to analyze the evolution of the language from the oldest post or comment to the most recent one. 2.1
Evaluation measures
The evaluation of the tasks considered standard classification measures, such as F1,
Precision and Recall (computed with respect to the positive class –depression or anorexia,
respectively–) and an early risk detection measure proposed in [
However, standard classification measures are time-unaware and do not penalize
late decisions. Therefore, the evaluation of the tasks also considered a newer measure
of performance that rewards early alerts. More specifically, we employed ERDE, an
error measure for early risk detection [
ERDE, which stands for early risk detection error, takes into account the correctness of the (binary) decision and the delay taken by the system to make the decision. The delay is measured by counting the number (k) of distinct submissions (posts or comments) seen before taking the decision. For instance, imagine a user u who posted a total number of 250 posts or comments (i.e. exactly 25 submissions per chunk to simplify the example). If a team’s system emitted a decision for user u after the second chunk of data then the delay k would be 50 (because the system needed to see 50 pieces of evidence in order to make its decision).
Another important factor is that data are unbalanced (many more negative cases than positive cases) and, thus, the evaluation measure needs to weight different errors in a different way. Consider a binary decision d taken by a team’s system with delay k. Given golden truth judgments, the prediction d can be a true positive (TP), true negative (TN), false positive (FP) or false negative (FN). Given these four cases, the ERDE measure is defined as:
ERDEo(d; k) = 8> cfp if d=positive AND ground truth=negative (FP) >< cfn if d=negative AND ground truth=positive (FN)
lco(k) ctp if d=positive AND ground truth=positive (TP) >:> 0 if d=negative AND ground truth=negative (TN)
How to set cfp and cfn depends on the application domain and the implications of FP and FN decisions. We will often deal with detection tasks where the number of negative cases is several orders of magnitude larger than the number of positive cases. Hence, if we want to avoid building trivial systems that always say no, we need to have cfn >> cfp. In evaluating the systems, we fixed cfn to 1 and cfp was set according to the proportion of positive cases in 2017’s test data (e.g. we set cfp to 0.1296).
The factor lco(k)(2 [0; 1]) represents a cost associated to the delay in detecting true positives. We set ctp to cfn (i.e. ctp was set to 1) because late detection can have severe consequences (as a late detection is considered as equivalent to not detecting the case at all).
The function lco(k) is a monotonically increasing function of k: lco(k) = 1
1 1 + ek o (1)
The function is parameterised by o, which controls the place in the X axis where the cost grows more quickly (Figure 1 plots lc5(k) and lc50(k)).
The latency cost factor was only used for the true positives because we understand that late detection is not an issue for true negatives. True negatives are non-risk cases that, of course, would not demand early intervention (i.e. these cases just need to be effectively filtered out from the positive cases). The systems must therefore focus on early detecting risk cases and detecting non-risk cases (regardless of when these nonrisk cases are detected).
All cost weights are in [0; 1] and, thus, ERDE is in the range [0; 1]. Systems had to take one decision for each subject and the overall error is the mean of the p ERDE values. ) l(ck .04 Each team could submit up to 5 runs or variants. We received 45 contributions from 11 different institutions. This is a substantial increase with respect to erisk 2017, which had 8 institutions and 30 contributed runs. Table 3 reports the institutions that contributed to eRisk 2018 and the labels associated to their runs.
First, we briefly describe the main characteristics of the early detection systems implemented by these participants:
- FHDO, Germany. This is a joint effort performed by the several institutions in Germany (and led by the University of Applied Sciences and Arts Dortmund). This team submitted results for four machine learning models, together with an ensemble model that combined different base predictions. The models employ user-level linguistic metadata, bag of words, neural word embeddings, and convolutional neural networks.
- IRIT, France. This is a team composed of researchers from IRIT and LIMSI. Their experiments focused on investigating two types of textual representations: linguistic features vs vectorization. The team combined the representations in different ways and trained a number of machine learning models.
- LIRMM, France. This team, composed of researchers from different institutions in Montpellier, paid special attention to the temporal dimension. Their models try to capture temporal mood variation by sequentially analysing the available user submissions. The resulting mdoels have two learning stages and employ standard text vectorization methods.
- PEIMEX, Mexico & USA. This team submitted several runs as a result of a joint collaboration between multiple Mexican institutions and Houston University. Their approach makes a sentence-level analysis to detect sentences where users refer to themselves. The main intuition is that those sentences contain terms that better expose their interests and habits and, thus, they might reveal personality and psychological states. This extraction of sentences was followed by a novel feature selection process and a subsequent term weighting method.
- UDC, Spain. This team performed a standard machine learning treatment of the challenge. They formalized the task as a classification task and experimented with different features (text-based, semantic-based and writing-based). They implemented two independent models. The first was oriented to predict depression cases and the second was oriented to detect non-depression cases. To meet these aims, these researchers designed two variants, named as Duplex Model Chunk Dependent and Duplex Model Writing Dependent.
- UNSL, Argentina & Mexico. This is a team composed of researchers from a couple of Argentinian institutions (UNSL and CONICET) and INAOE, from Mexico. This team implemented a variant based on a model of flexible temporal variation of terms and another variant based on sequential incremental classification. The first model follows a semantic representation of documents that explicitly considers that the information available at each chunk is partial. The second model is a novel text classification approach that incrementally estimates the association of each individual to each class based on the accumulated evidence.
- UPF, Spain. This team, from Univ. Pompeu Fabra in Barcelona, implemented several machine learning models that follow a dynamic and incremental representation of the user’s submissions. The main focus of the experimentation was on testing different types of features, including linguistic features, domain-specific vocabularies and psychology-based features.
- UQAM, Canada. This team implemented a topic extraction approach and experimented with Latent Dirichlet Allocation and Neural Networks. The submitted runs represented the texts using unigrams, bigrams and trigrams and the team worked with 30 latent topics. The final estimations were supplied by a multilayer perceptron, together with a decision-based algorithm that classifies the users in a time-aware manner.
- TBS, Taiwan. This team is composed of researchers from two different institutions in Taiwan. Their models combine tf/idf evidence with convolutional neural networks (CNNs). The CNNs work with chunk-level evidence and are responsible of emitting the depression decisions. These decisions are based on classifying individual submissions made by each user.
- TUA1, Japan. The University of Tokushima in Japan sent results associated to a support vector machine classifier that works with tf/idf representations, a deep neural network and a simple keyword-based method.
Now, let us analyze the behaviour of the systems in terms of how fast they emitted decisions. Figure 2 shows a boxplot graph of the number of chunks required to make the decisions. The test collection has 820 users and, thus, each boxplot represents the statistics of 820 cases.
Some systems (RKMVERIB, RKMVERIC, RKMVERID, RKMVERIE, TBSA, UPFC, UPFD) took all decisions after the last chunk (i.e. did not emit any earlier decision). These variants were extremely conservative: they waited to see the whole history of submissions for all users and, next, they emitted their decisions. Remember that all teams were forced to emit a decision for each user at the last chunk.
Many other runs also took most of the decisions after the last chunk. For example, FHDO-BCSGA assigned a decision at the last chunk in 725 out of 820 users. Only a few runs were really quick at emitting decisions. Notably, most UDC’s runs and LIIRA had a median of 1 chunk needed to emit decisions.
Figure 3 shows a boxplot of the number of submissions required by each run in order to emit decisions. Most of the time the teams waited to see hundreds of writings for each user. Only a few submissions (UDCA, UDCB, UDCD, UDCE, UNSLD, some LIIRx runs) had a median number of writings analyzed below 100. It appears that the teams have concentrated on accuracy (rather than delay) and, thus, they did not care much about penalties for late decisions. A similar behaviour was found in the runs submitted in 2017.
The number of user submissions has a high variance. Some users have only 10 submissions, while other users have more than a thousand submissions. It would be interesting to study the interaction between the number of user submissions and the effectiveness of the estimations done by the participating systems. This study could help to shed light on issues such as the usefulness of a large (vs short) history of submissions and the effect of off-topic submissions (e.g. submissions totally unrelated to depression).
Another intriguing issue relates to potential false positives. For instance, a doctor who is active on the depression community because he gives support to people suffering from depression, or a wife whose husband has been diagnosed with depression. These people would often write about depression and possibly use a style that might imply they are depressed, but obviously they are not. The collection contains this type of non-depressed users and these cases are challenging for automatic classification. Arguably, these non-depressed users are much different from other non-depressed users who do not engage in any depression-related conversation. In any case, this issue requires further investigation. For example, it will be interesting to do error analysis with the systems’ decisions and check the characteristics of the false positives.
Figure 4 helps to analyze another aspect of the decisions emitted by the teams. For each user class, it plots the percentage of correct decisions against the number of users. For example, the last two bars of the upper plot show that about 5 users were correctly identified by more than 90% of the runs. Similarly, the rightmost bar of the lower plot means that a few non-depressed users were correctly classified by all runs (100% correct decisions). The graphs show that the teams tend to be more effective with non-depressed users. This is as expected because most non-depressed cases do not engage in depression-related conversations and, therefore, they are easier to distinguish from depressed users. The distribution of correct decisions for non-depressed users has many cases where more than 80% of the systems are correct. The distribution of correct decisions for depressed users is flatter, and many depressed users are only identified by a low percentage of the runs. This suggests that the teams implemented a wide range of strategies that detect different portions of the depression class. Furthermore, there are not depressed users that are correctly identified by all systems. However, an interesting point is that no depressed user has 0% of correct decisions. This means that every depressed user was classified as such by at least one run.
Let us now analyze the effectiveness results (see Table 4). The first conclusion we can draw is that the task is as difficult as in 2017. In terms of F1, performance is again low. The highest F1 is 0.64 and the highest precision is 0.67. This might be related to the effect of false positives discussed above. The lowest ERDE50 was achieved by the FHDO-BCSG team, which also submitted the runs that performed the best in terms of F 1. The run with the lowest ERDE5 was submitted by the UNSLA team and the run with the highest precision was submitted by RKMVERI. The UDC team submitted a high recall run (0.95) but its precision was extremely low.
In terms of ERDE5, the best performing run is UNSLA, which has poor F1, Precision and Recall. This run was not good at identifying many depressed users but, still, it has low ERDE5. This suggests that the true positives were emitted by this run at earlier chunks (quick emissions). ERDE5 is extremely stringent with delays (after 5 writings, penalties grow quickly, see Fig 1). This promotes runs that emit few but quick depression decisions. ERDE50, instead, gives smoother penalties to delays. This makes that the run with the lowest ERDE50, FHDO-BCSGB, has much higher F1 and Precision. Such difference between ERDE5 and ERDE50 is highly relevant in practice. For example, a mental health agency seeking an automatic tool for screening depression could set the penalty weights depending on the consequences of late detection of signs of depression. 3
Task 2 was an exploratory task on early detection of signs of anorexia. The format of the task, data extraction methods and evaluation methodology (training stage followed by a test stage with on sequential releases of user data) was the same used for Task 1. This task was introduced in 2018 and, therefore, all users (training+test) were collected just for this new task.
Table 2 reports the main statistics of the train and test collections of Task 2. The collection shares the main characteristics of Task 1’s collections: the two splits are unbalanced (of course, there are more non-anorexia cases than anorexia cases). Contrary to the depression case, the number of users is not large (and, again, each user has a long history of submissions). The mean range of dates from the first submission to the last submission is also wide (more than 500 days). 3.1 Each team could submit up to 5 runs or variants. We received 35 contributions from 9 different institutions. All institutions participating in Task 2 had also sent results for Task 1. Table 5 reports the institutions that contributed to this second task of eRisk 2018 and the labels associated to their runs.
Most of the teams implemented the same type of models and use them for both tasks (with minor modifications, such as the inclusion of anorexia-related lexica). We refer to section 2.2, where the reader can see a brief description of each group’s variants. The interested reader is also referred to the working note papers to see a full description of the experiments performed by each team.
The behaviour of the systems in terms of how fast they emitted decisions is shown in Figure 5, which includes boxplot graphs of the number of chunks required to make the decisions. The test collection of Task 2 has 320 users and, thus, each boxplot represents the statistics of 320 cases. The trends are similar to those found in Task 1. Mosf of the systems emitted decisions at a late stage with only a few exceptions (notably, LIIRA and LIIRB). LIIRA and LIIRB had a median number of chunks analyzed of 3 and 6, respectively. The rest of the systems had a median number of chunks analized equal to or near 10.
Figure 6 shows a boxplot of the number of submissions required by each run in order to emit decisions. Again, most of the variants analyzed hundred of submissions before emitting decisions. Only the two LIIR runs discussed above and LIRMMD opted for emitting decisions after a fewer number of user submissions. In Task 2, again, most of the teams have ignored the penalties for late decisions and they have mostly focused on classification accuracy.
Figure 7 plots the percentage of correct decisions against the number of users. The plot shows again a clear distinction between the positive class (anorexia) and the negative class (non-anorexia). Most of the non-anorexia users are correctly identified by most of the systems (nearly all non-anorexia users fall in the range 80%-100%, meaning that at least 80% of the systems labeled them as non-anorexic). In contrast, the distribution of anorexia users is flatter and, in many cases, they are only identified by less than half of the systems. An interesting result is that all anorexia users were identified by at least 10% of the systems.
Table 6 reports the effectiveness of the systems. In general, performance is remarkably higher than that achieved by the systems for Task 1. There could be a number of reasons for such an outcome. First, the proportion of potential false positives (e.g. people engaging in anorexia-related conversations) might be lower in Task 2’s test collection. This hypothesis would need to be investigated through a careful analysis of the data. Second, the submissions of anorexia users might be extremely focused on eating habits, losing weights, etc. If they do not often engage in general (anorexia unrelated) conversations then it would be easier for the systems to distinguish them from other users. In any case, these are only speculations and this issue requires further research.
The highest F1 is 0.85 and the highest precision is 0.91. The lowest ERDE50 was achieved by FHDO-BCSGD, which also has the highest recall (0.88). The run with the lowest ERDE5 was submitted by the UNSL team (UNSLB), which shows again that this team paid more attention to emitting early decisions (at least for the true positives).
Overall, the results obtained by the teams are promising. The high performance achieved suggest that it is feasible to design automatic text analysis tools that make early alerts of signs of eating disorders. 4
This paper provided an overview of eRisk 2018. This was the second year that this lab was organized at CLEF and the lab’s activities concentrated on two tasks (early detection of signs of depression and early detection of signs of anorexia). Overall, the tasks received 80 variants or runs and the teams focused on tuning different classification solutions. The tradeoff between early detection and accuracy was ignored by most participants.
The effectiveness of the solutions implemented to early detect signs of depression is similar to that achieved for eRisk 2017. This performance is still modest, suggesting that it is challenging to tell depressed and non-depressed users apart. In contrast, the effectiveness of the systems that detect signs of anorexia was much higher. This promising result encourages us to further explore the creation of benchmarks for text-based screening of eating disorders. In the future, we also want to instigate more research on the tradeoff between accuracy and delay.
We thank the support obtained from the Swiss National Science Foundation (SNSF) under the project “Early risk prediction on the Internet: an evaluation corpus”, 2015.
We also thank the financial support obtained from the i) “Ministerio de Economía y Competitividad” of the Government of Spain and FEDER Funds under the research project TIN2015-64282-R, ii) Xunta de Galicia (project GPC 2016/035), and iii) Xunta de Galicia – “Consellería de Cultura, Educación e Ordenación Universitaria” and the European Regional Development Fund (ERDF) through the following 2016-2019 accreditations: ED431G/01 (“Centro singular de investigacion de Galicia”) and ED431G/08.
Institution Submitted files FH Dortmund, Germany FHDO-BCSGA FHDO-BCSGB FHDO-BCSGC FHDO-BCSGD FHDO-BCSGE IRIT, France LIIRA LIIRB LIIRC LIIRD LIIRE LIRMM, University of Montpellier, France LIRMMA LIRMMB LIRMMC LIRMMD LIRMME Instituto Tecnológico Superior del Oriente del Estado de Hidalgo, Mexico PEIMEXA Instituto Nacional de Astrofísica, Óptica y Electrónica, Mexico PEIMEXB Universidad de Houston, USA PEIMEXC & Universidad Autónoma del Estado de Hidalgo, Mexico PEIMEXD PEIMEXE Ramakrishna Mission Vivekananda Educational and Research Institute, RKMVERIA Belur Math, West Bengal, India RKMVERIB RKMVERIC RKMVERID RKMVERIE University of A Coruña, Spain UDCA UDCB UDCC UDCD UDCE Universidad Nacional de San Luis, Argentina UNSLA CONICET, Argentina UNSLB INAOE, Mexico UNSLC UNSLD UNSLE Universitat Pompeu Fabra, Spain UPFA UPFB UPFC UPFD Université du Québec à Montréal, Canada UQAMA The Black Swan, Taiwan TBSA Tokushima University, Japan TUA1A TUA1B TUA1C TUA1D Table 3. Task 1 (depression). Participating institutions and submitted results 10.0 7.5 k n u h c#5.0 2.5 10.0 7.5 k n u h c#5.0 2.5 10.0 7.5 k n u h c#5.0 2.5 10.0 7.5 k n u h c#5.0 2.5
FHDO-BCSGA FHDO-BCSGB FHDO-BCSGC FHDO-BCSGD FHDO-BCSGE
LIRMMA
LIRMMB
LIRMMC
LIRMMD
LIRMME RKMVERIA
RKMVERIB
RKMVERIC
RKMVERID
RKMVERIE
UDCA
UDCB
UDCC
UDCD
UDCE TBSA
UNSLA
UNSLB
UNSLC
UNSLD
UNSLE
UPFA
UPFB
UPFC
UPFD LI RA
LI RB
LI RC
LI RD
LI RE
PEIMEXA
PEIMEXB
PEIMEXC
PEIMEXD
PEIMEXE n 1it000 i r 500 0 2000 1500 s g w # 500 0
FHDO-BCSGA FHDO-BCSGB FHDO-BCSGC FHDO-BCSGD FHDO-BCSGE
LIRMMA
LIRMMB
LIRMMC
LIRMMD
LIRMME RKMVERIA
RKMVERIB
RKMVERIC
RKMVERID
RKMVERIE
UDCA
UDCB
UDCC
UDCD
UDCE TBSA
UNSLA
UNSLB
UNSLC
UNSLD
UNSLE
UPFA
UPFB
UPFC
UPFD LI RA
LI RB
LI RC
LI RD
LI RE
PEIMEXA
PEIMEXB
PEIMEXC
PEIMEXD
PEIMEXE Institution Submitted files FH Dortmund, Germany FHDO-BCSGA FHDO-BCSGB FHDO-BCSGC FHDO-BCSGD FHDO-BCSGE IRIT, France LIIRA LIIRB LIRMM, University of Montpellier, France LIRMMA LIRMMB LIRMMC LIRMMD LIRMME Instituto Tecnológico Superior del Oriente del Estado de Hidalgo, Mexico PEIMEXA Instituto Nacional de Astrofísica, Óptica y Electrónica, Mexico PEIMEXB Universidad de Houston, USA PEIMEXC & Universidad Autónoma del Estado de Hidalgo, Mexico PEIMEXD PEIMEXE Ramakrishna Mission Vivekananda Educational and Research Institute, RKMVERIA Belur Math, West Bengal, India RKMVERIB RKMVERIC RKMVERID RKMVERIE Universidad Nacional de San Luis, Argentina UNSLA CONICET, Argentina UNSLB INAOE, Mexico UNSLC UNSLD UNSLE Universitat Pompeu Fabra, Spain UPFA UPFB UPFC UPFD The Black Swan, Taiwan TBSA Tokushima University, Japan TUA1A TUA1B TUA1C
Table 5. Task 2 (anorexia). Participating institutions and submitted results
UNSLA UNSLB UNSLC UNSLD UNSLE Fig. 5. Number of chunks required by each contributing run in order to emit a decision.
FHDO−BCSGAFHDO−BCSGBFHDO−BCSGCFHDO−BCSGDFHDO−BCSGE
LIRMMA
LIRMMB
LIRMMC
LIRMMD
LIRMME PEIMEXA
PEIMEXB
PEIMEXC
PEIMEXD
PEIMEXE
RKMVERIA
RKMVERIB
RKMVERIC
RKMVERID
RKMVERIE LIIRA
LIIRB
TBSA
TUA1A
TUA1B
TUA1C
UPFA
UPFB
UPFC
UPFD UNSLA
UNSLB
UNSLC
UNSLD
UNSLE