In this paper we describe the participation of our research group at the CLEF eRisk 2019. The eRisk goal is the early detection of atrisk people by means of machine learning techniques based on language usage. This year eRisk edition was divided into three tasks, T1, T2, and T3. The rst two were focused on early detection of anorexia and self-harm on Reddit users. T3 focused on measuring users' severity of depression. To carry out this task, models had to automatically ll the standard BDI depression questionnaire based on the evidence found in the user's history of postings. We used the same classi er, SS3, to carry out these three tasks with the same hyper-parameters con guration. SS3 is a recently introduced text classi er[1] that was created with the goal to deal with early risk detection scenarios in an integrated manner: it naturally supports incremental and early classi cation over text streams and additionally, it has the ability to visually explain its rationale. The nal results for all these three tasks show that SS3 is a very robust and e cient classi er. SS3 was the fastest method and obtained the best ERDE and overall best ranking-based measures in all the tasks. Additionally, it obtained the best P recision, F 1 and F 1latency for task T2. Finally, in task T3, it obtained the best AHR and ACR values, and the second-best ADODL and DCHR. This was quite remarkable taking into account that the same classi er was used here to ll users' BDI questionnaires, which is a task completely di erent from the other two \yes or no" tasks.
The detailed description of each task and the used performance measures, and
full lists of results are given in [
As it is described in more details in [
g!v(w) = (gv(w; c0); gv(w; c1); : : : ; gv(w; ck)) where ci 2 C (the set of all the categories). That is, g!v is only applied to a word and it outputs a vector in which each component is the gv of that word for each category ci. For instance, following the above example, we have: gv(`sushi`) = (0:85; 0:09; 0:5; 0:02); gv(`the`) = (0; 0; 0; 0);
The vector g!v(w) is called the \con dence vector of w". Note that each category ci is assigned a xed position in g!v. For instance, in the example above (0:85; 0:09; 0:5; 0:02) is the con dence vector of the word \sushi" and the rst position corresponds to f ood, the second to music, and so on.
It is worth mentioning that the computation of gv involves three functions, lv, sg and sn, as follows:
gv(w; c) = lv (w; c) sg (w; c) sn (w; c) { lv (w; c) values a word based on the local frequency of w in c. As part of this process, the word distribution curve is smoothed by a factor controlled by the hyper-parameter . { sg (w; c) captures the global signi cance of w in c, it decreases its value in relation to the lv value of w in the other categories; the hyper-parameter controls how far the local value must deviate from the median to be considered signi cant. { sn (w; c) sanctions lv in relation to how many other categories w is signi cant (sg (w; c) 1) to. That is, The more categories ci whose sg (w; ci) is high, the smaller the sn (w; c) value. The hyper-parameter controls how sensitive this sanction is.
For those readers interested in how these functions are actually computed, we
highly recommend you to read the SS3 original paper[
For all 3 tasks T1, T2 and T3 we carried out the classi cation of each user,
incrementally as in [
X w2W Hs g!v(w) (1) where W Hs is the subject's writing history. Note that for all the tasks !ds was a vector with two components, one for the positive class and the other for the negative one. The policy to classify a subject as positive was performed analyzing how !ds changed over time, as shown with an example in Figure 1. Subjects were classi ed as positive when the positive value in !ds exceeded the negative one4, for instance, the subject in the gure was classi ed as anorexic after reading the 42nd writing.
Finally, this year, models performance were also evaluated based on rankingbased measures for tasks T1 and T2. Thus, models were asked to provide an estimated score of the level of anorexia/self-harm along with the binary decision (0/1). To compute this score, SS3 performed the di erence between the positive con dence value and the negative one (i.e. d[positive] d[negative]) and returned it along with each decision. For example, in Figure 2 is shown how this score changed as more writings were read, for the same user shown previously in Figure 1. 3
As mentioned earlier, we used the same classi er with the same hyper-parameters for these 5 runs, and instead, we mostly focused on changing aspects related to how we trained our models. For each one of the 5 runs, we performed the following: { UNSL#0 : we trained the model using only the data in the \train" folder, i.e. we trained the model using the same data as for the eRisk 2018. { UNSL#1 : the same as with the previous one but this time allowing SS3 to compute the global value not only for words but also for pair of words (bigrams), i.e. SS3 learned to compute gv(w0; c) as well as gv(w0w1; c) for each w0; w1 seen during training. { UNSL#2 : the same as run#0 but training with all available data for training this year, i.e. all the data in both \train\ and \test\ folders. { UNSL#3 : the same as in the previous run, but this time also taking into account bigrams of words (as in run#1). { UNSL#4 : the same as in run#2 but letting SS3 to take into account only words whose global value where greater than 0.3, i.e in Equation 1 SS3 assigned gv(w; ci) = 0 to all w and ci such that gv(w; ci) < 0:3.
The main global results obtained in this task could be summarized as follows: { As it is shown in Table 1, UNSL#0 obtained the best ERDE5 and UNSL#4 the best ERDE50. Note that most of the ERDE values are relatively close to each other, this is due to the way ERDE was computed.5 The larger and more unbalanced the dataset is, the asymptotically atter and closer the 4 Except for task T3 in which we did not perform an "early stop", and therefore every user was classi ed after processing the entire writing history. 5 cf p = #positive users therefore each user, misclassi ed as false positive, added a value #users of #users = #positive users = 8152 = 0:0001 to the nal/reported ERDE (and 0.001 cfp 73
#users2 in case of false negative or true positive after the o threshold). ERDE values are (as it was with this task). Thus, decimals do matter a lot when it comes to ERDE measure. For instance, in this task, just a small di erence of 0.009 (0.9%) in ERDE actually means that 12% of either all anorexic users or non-anorexic ones were not properly classi ed, which is a signi cant di erence. { Regarding the new F 1latency, we did not obtain remarkable results, being 0.13 points below the best one. This is mainly due to this new measure being introduced this year and only after the tasks ended. Therefore, SS3 could not be optimized to obtain a better F 1latency value. As said before, we used the same hyper-parameters for the 15 runs of the 3 tasks, these hyper-parameters were selected to optimize ERDE measures, which in turns produced an SS3 model that prioritizes the recall6 and speed7 above the precision, which is not bad taking into account that we are dealing with early risk detection tasks (every positive subject not detected is a life at risk!). Despite this, our best F 1latency value (.55) was quite above the average (0.38) and was positioned 11th out of the 50 contributions and 5th out of the 13 research groups. Additionally, we also decided to compute the F 2latency which gives a little more of importance to recall than to precision. This improved our results, making our best F 2latency value (.67) to be positioned 7th out of the 50 contributions and 3rd out of the 13 research groups, and only 0.07 points below the best one. { As it is shown in Table 2, SS3 was the fastest method to process the writings from each server response, processing each users' writing in about 8s8, this 6 This is due to the way ERDE measure is computed, the false positive cost (cf p) is really small compared to the false negative cost (cf n). 7 On average, SS3 classi ed users after reading the 2nd or 3rd post. 8 Note that much of this 8s were wasted waiting for network communication, since this number includes the latency of receiving and sending the response from and to the API RESTful server.
contrast with other methods that obtained a better F 1latency but required
more time, such as CLaC, INAOE-CIMAT or lirmm. For instance, CLaC
was 232 times slower than SS3 and took 11 days and 16h to process only
109 of the 2000 writings, whereas SS3 processed all the 2000 writings for the
5 runs in only 23h. This could suggest that some research groups possible
incorporated some sort of o ine (or manual) processing into their models.
It is worth mentioning that the fact that SS3 was the fastest model was
not due to the type of machine we used9 but rather due to SS3 naturally
supporting incremental classi cation. To put this point in context, it is
important to note that ERD is essentially a problem of analysis of sequential
data. That is, unlike traditional supervised learning problems where learning
and classi cation are done on \complete" objects, here classi cation must be
done on \partial" objects which correspond to all the data sequentially read
up to the present, from a (virtually in nite) data stream. Algorithms
capable of naturally dealing with this scenario are said to support incremental
classi cation. As it is described in more details in [
writings.11 Note that these results are not a minor aspect, since they are implying that both: (a) the score (con dence value) given by SS3 correctly values/ranks positive subjects, that is, the global value, gv, is correctly capturing the degree of importance of each word for the positive class12 (see Figure 3 for a top-100 word cloud selected by gv); and (b) since SS3 is valuing/ranking users correctly, it means there is much room for improving the classi cation performance by choosing a better policy to actually classify them |perhaps using global information across di erent users could lead us to better classi cation performance, instead of classifying them locally, simply and prematurely just when the positive value exceeds the negative one. 11 Those rst NDCG@100 values, .87 and .88, were obtained by UDE#1. It is worth mentioning that it took UDE 1 day and 6h to process those 500 writings (or 2 days and 12h for those 1000 writings) whereas it took SS3 only 5h to process them (9 times faster). 12 which, as we will see later, this is also re ected by the promising results obtained in task T3.
For this task, unlike T1, the training set was not provided, and therefore we had to build our own dataset to train SS3. To achieve this, we tried out creating di erent datasets, for instance, collecting tweets and Reddit posts related to self-harm, or using the datasets already available for anorexia and depression, as it is described in more details below: { UNSL#0 : we collected Reddit posts related to self-harm and stored them in a single txt le to represent the positive class. For the negative class, we used the negative documents in the \train" folder for task T1 (anorexia). This run obtained the best precision (.71) but, among the other 4 runs, the lowest recall (.41) along with UNSL#1 (.39), both using the same dataset. { UNSL#1 : this run used the same dataset as the previous one, but this time
SS3 took into account also bigrams of words (as in UNSL#1 for T1). { UNSL#2 : for this run, we trained SS3 using a dataset built using the Reddit posts (the same used in the runs above) and tweets related to self-harm. We created a single le with all these tweets and posts related to self-harm (about 40MB in size) and used it to learn the positive class. For the negative class, we used the negative training documents for the eRisk 2018 depression task. Additionally, as in run#4 of T1, SS3 was con gured to ignore words whose global value was less than 0.3. Among the other 4 runs, this one had the best Recall (.9) but the worst values for precision (.2) and F1 (.32). { UNSL#3 : here we trained SS3 using the training documents for T1 (anorexia) and also using the training documents for the eRisk 2018 depression task. This run had a similar performance to run#4, although its recall was a little bit worse. { UNSL#4 : SS3 was trained using the same documents as in the previous run (i.e. anorexia + depression 2018) but this time adding those of run#0. This run had the best F 2latency (.64) and ERDE values, 8.20% and 4.93% for ERDE5 and ERDE50 respectively.
Since no training data was released, we did not have any validation set to check if our models were learning properly, i.e. we did not know on which data (a) Words (b) Word bigrams our models were going to be evaluated. In order to know whether the learned model made sense or not, after training, we asked SS3 to give us a list of words ordered by global value for the positive class, and checked if the list made sense to us. Fortunately, as shown in Figure 4, the generated list of words matched what we expected.
The global results obtained in this task could be summarized as follows: { As it is shown in Table 4, UNSL#0 obtained the best precision (.71), F 1 (.52) and F 1latency (.52), and UNSL#4 the best ERDE5 (8.20%) and ERDE50 (4.93%). { Once again SS3 was the fastest method, processing all the writing of each response in about 5s (as shown in Table 5). { Again, SS3 was the method that obtained the best overall performance in ranking-based evaluation since, as shown in Table 6: it obtained the best ranking performance P@10 and NDCG@10 for all the 4 rankings; the best NDCG@100 for rankings made after processing 1 and 100 writings (.67 and .86 respectively). Additionally, for the ranking made after processing 500 and 1000 writings, SS3 obtained the second-best NDCG@100 (.79 and .78 respectively), the rst ones were obtained by Fazl#1 (.84). It is worth mentioning it took Fazl 4 days and 17h to process those 500 writings (or 9 days and 10h for the 1000 writings), whereas it took SS3 only 3h (almost 60 times faster!).
It is worth mentioning that, unlike the other runs, UNSL#3 was trained only using data from anorexia and depression and yet it obtained good results. In fact, if we had sent only this run, among all participants, SS3 would have still obtained the best ERDE values (8.78% and 5.44%), the best F 2latency (.63) and the second best F 1latency (.45, rst would have been .46). This, added to the results obtained by the other 4 runs, shows us that SS3 is a classi er quite robust to deal with cross-domain scenarios.
This task was really di cult since it was not a single \yes or no" problem but
a problem involving multiple decisions, one for each one of the 21 questions.
To make things even harder, as with Task 2, no training data was released
either. Fortunately, early depression detection is a task we had some previous
experience working with since we had participated in the two previous eRisk
labs (2017[
Then, after SS3 classi ed a subject, the obtained conf idence value was divided into 4 regions, one for each BDI depression category. This was carried out by the following equation: c = bconf idence value 4c
And nally, the subject depression level was predicted by mapping the percentage of conf idence value left inside the predicted c region to its corresponding BDI depression level range (e.g. (0:5; 0:75] ! [19; 29] for c = 2 = \moderate depression") by computing the following: (2) (3) depression level = minc +b(maxc minc +1) (conf idence value 4 c)c (4) Where minc and maxc are the lower and upper bound for category c, respectively (e.g. 19 and 29 for \moderate depression" category).
In order to clarify the above process, we will illustrate it with the example shown in Figure 5. First, SS3 processed the entire writing history computing the conf idence value (given by Equation 2) and then, the nal conf idence value (0.941) was used to predict the depression category, \severe depression" (c = 3), by using the Equation 3. Finally, the depression level was computed by the mapping given by Equation 4, as follows: depression level = 30 + b(63 30 + 1) (0:941 4
3)c = 30 + b34 = 30 + 25 = 55 (3:764 3)c = 30 + b34 0:764c (5) Fig. 5: Diagram of the depression level computation process for subject 2827. As reader can notice, after processing all the subject's writings, the nal con dence value (0.941) was mapped into its corresponding depression level (55).
At this point, we have transformed the output of SS3 from a 2-dimensional vector, d, into a BDI depression level (a value between 0 and 63). However, we have not covered yet how to actually answer the 21 questions in the BDI questionnaire using this depression level. Regardless of the method, we decided that for all those users whose depression level was less or equal to 0, all the BDI questions were answered with 0. For the other users we applied di erent methods, depending on the run, as described below: { UNSLA: using the predicted depression level our model lled the questionnaires answering the expected number (b depression level c) on each question. If 21 this division had a remainder, the remainder points were randomly scattered so that the sum of all the answers always matched the predicted depression level given by SS3. { UNSLB : this time, only the predicted category, c, was used. Our model lled the questionnaire randomly in such a way that the nal depression level always matched the predicted category. Compared to the following three ones, these two models were the ones with the worst performance. { UNSLC : this model and the followings were more question-centered. Once again, as in UNSLA, our model lled the questionnaires answering the expected number derived from the predicted depression level (b depression level c). 21 But this time, answering this number only on questions for which a \textual hint" for a possible answer was found in the user's writings, and randomly and uniformly answered between 0 and d depress2i1on level e otherwise. To nd this \textual hint", our model split the user's writings into sentences and searched for the co-occurrence of the word \I" or \my" with at least one (a) If expected answer is 0 (b) If expected answer is 1 (c) If expected answer is 2 (d) If expected answer is 3 Fig. 6: Discrete probability distribution for each possible expected answer. word matching a regular expression specially crafted for each question.13 This method obtained the best AHR (41.43%) and the second-best DCHR (40%). { UNSLD : the same as the previous one, but not using the \textual hints", i.e. always answering every question randomly and uniformly between 0 and d depress2i1on level e. This model was mainly used only with the goal of measuring the actual impact of using these \textual hints" to decide which questions should be answered with the expected answer (b depression level c). 21 { UNSLE : the same as UNSLD, but this time not using a uniform distribution. More precisely, from the overall depression level predicted by SS3, once again the expected answer was computed (b depress2i1on level c) and, depending on the value of the expected answer, actual answers were given following the probability distributions shown in Figure 6. Note that, unlike uniform 13 e.g. \(sad)j(unhappy)" for question 1, \(future)j(work out)" for question 2, \failnw*" for question 3, \(pleasure)j(enjoy)" for question 4, etc. distribution (used in UNSLD), when using these probability distributions the expected answer is more likely to be selected over the other ones. This model obtained the best ACR (71.27%) and the second-best AHR (40.71%) and ADODL (80.48%, best was only 0.54% above).
The obtained results are shown in Table 7. As mentioned above, we obtained the best AHR (41.43%) and ACR (71.27%), and the second-best ADODL (80.48%) and DCHR (40%). However, since most of our models' answers are randomly generated, it implies that all of these measures are also stochastically generated.14 The natural question in cases like this is \How do we know these results properly represent our models' performance and we did not obtain them just by pure chance?". In order to clarify this, we run each model 1000 times and calculated the values for AHR, ACR, ADODL and DCHR each time15. After this process nished, we ended up with a sample of 1000 values for each measure and model, which we then used to produce the results shown in Table 8. Results have been replaced by intervals with 95% of con dence, which better represent our performance. One can notice that, in fact, when we participated we had a little bit of bad luck, especially for UNSLE's ADODL, because the actual value we obtained (80.84%) is almost a lower bound outlier. Another thing that we can notice, comparing UNSLC and UNSLD, is that the use of \textual hints" slightly improves the Average Hit Rate (AHR) but does not impact on the other measures. UNSLE is considerably the best method to estimate the overall depression level since it takes values within a range that is quite above the 14 Only ADODL and DCHR for UNSLA and DHR for UNSLB are deterministically determined by depressionlevel and c. 15 Just as if we had participated 1000 times in this task. others. Additionally, another important point is that taking into account these 95% con dence intervals, the obtained values would be among the best ones even in the worst cases. Finally, since all the methods we used are based on the depression level predicted by SS3, this shows us that SS3 is correctly inferring the depression level from the textual evidence accumulated while processing the user's writings, i.e. SS3 is correctly valuing words in relation to each category (depressed and non-depressed) which is consistent with the results obtained for the ranking-based measures for task T1 and T2. Additionally, this could also imply that could really be a relationship between how subjects write (what words they use) and the actual depression level they have. 6
In this article, we described the participation of our research group16 at the
CLEF eRisk 2019[