In this paper we describe the participation of the LIDIC Research Group of Universidad Nacional de San Luis (UNSL) - Argentina at CLEF eRisk 2018 Lab. The main goal of this Lab is considering early risk detection scenarios where the issue of getting timely predictions with a reasonable confidence level becomes critical. Two completely different approaches were used, that we will refer as flexible temporal variation of terms (FTVT) and sequential incremental classification (SIC). FTVT is a semantic representation of documents that explicitly considers the partial information that is made available in the different “chunks” to the early risk detection systems along the time. FTVT is an improvement on the TVT method [1] that allows varying the number of chunks considered in the representation according to the “level of urgency” required in the classification. SIC is a novel approach for text categorization that incrementally estimates the level of belonging of a piece of text to the different categories based on an accumulative process of evidence. In the test stage, FTVT obtained the lowest ERDE5 error in both pilot tasks and SIC achieved the highest precision for the anorexia detection task providing strong evidence that both approaches used by our team are interesting alternatives to deal with early risk detection tasks.
The increasing use of Internet, social networks and other computer technologies allows the extraction of valuable information to early prevent some risks. In this context, early risk detection (ERD) on the Internet is an important research area due to the impact it might have in areas like health when people suffer depression, anorexia or other disorders that can threaten life and safety when criminals and sex offenders try to attack using web technologies.
The same as other predictive tasks, ERD methods have been mainly based on supervised machine learning approaches. In those cases, the task is generally addressed as a standard binary classification problem with two unbalanced classes: a minority (risky) positive class and a majority (control) negative class. However, beyond the difficulty that the unbalanced classes present to the learning algorithms, ERD introduces an added problem that is not usually present in other classification tasks: the incremental classification of sequential data (ICSD).
To effectively support ICSD two important aspects need to be considered. First, we
must provide an adequate way to “remember” or “summarize” historical information
read up to specific points of time. The informativeness level of these partial models
will be critical to the effectiveness of the classifier in charge of detecting risky cases.
Second, these models need to also provide support to a very important aspect of ERD:
the decision of when (how soon) the system should stop reading from the input stream
and classify it with an acceptable level of accuracy. This aspect, that we will refer as the
supporting for early classification, is basically a multi-objective decision problem that
attempts balancing accurate and timely classifications [
The eRisk 2018 Lab presented two challenging tasks for ERD: early detection of signs of depression (task 1), and early detection of signs of anorexia (task 2). We participated in both tasks with two different approaches to deal with the ICSD issue: one that we will refer as flexible temporal variation of terms (FTVT) and the other named sequential incremental classification (SIC).
FTVT is a document representation that deals with the ICSD problem by keeping sequential information about the variation of terms occurring in the different chunks. The hypothesis behind this approach is that these variations can be informative to detect a risky case. SIC is a sequential approach that incrementally reads and estimates the evidence that words provide for both, positive and negative classes. SIC classifies a subject as risky as soon as the accumulated evidence of the risky (positive) class surpass the evidence of the negative one.
The experiments carried out on the training sets for both tasks were mainly aimed at determining adequate parameters for training the models (classifiers) for the test stage. Preliminary results reported by the Lab’s organizers showed that our systems obtained the best (lowest) ERDE5 error in both pilot tasks and SIC the highest precision for the anorexia detection task providing strong evidence that the used approaches are interesting alternatives to deal with early risk detection tasks.
The rest of the article is organized as follows: Section 2 gives general information of the data sets used in both pilot tasks and the methods used in our ERD systems. Next, in Section 3 the activities carried out in the training stage are described and the rationale behind the main design decisions made on our ERD systems, are presented. Section 4 shows the performance of our methods on the eRisk 2018 data sets released in the test stage. Finally, Section 5 depicts potential future works and the obtained conclusions.
The data sets supplied for the eRisk 2018 tasks4 are described in Losada et al. [
Each task was divided by their organizers into a training stage and a test stage. In the first one, the participating teams had access to the set of training users with ten chunks of all training users. They could therefore tune their systems with the training data. Then, in the test stage, the ten chunks from test set were gradually released by the organizers one by one until completing all the chunks that correspond to the complete writings of the considered individuals. Each time that a chunk chi was released, participants in the pilot tasks were asked to give their predictions on the users contained in the test set, based on the partial information read from chunks ch1 to chi. Once a class of an incoming stream is predicted, that decision is irreversible (it cannot be undone). 2.2
To deal with the problems posed in both pilot tasks we used two methods that previously referred as FTVT and HCI, which will describe below. An interesting aspect of those methods is that they are completely independent-domain. Thus, they do not require costly adaptation processes for each task, beyond the tuning of parameters that could depend of the used data set. In fact, due to limitations of time to carry out the experimental study, both methods were only evaluated on the data set of the task 1 (early depression detection) and the same parameters were used for task 2 (early anorexia detection).
Space constraints prevent us from giving detailed explanations of FTVT and HCI.
However, the interested reader can obtain in [
TVT method on which FTVT is based on. SIC is only introduced from an intuitive point of view because the method is currently under review in a scientific journal.5
Terms (FTVT) is an improvement of the temporal variation of terms (TVT) method [
As we previously said, TVT is based on the concise semantic analysis (CSA)
technique proposed in [
In this context, the underlying idea of TVT is that variations of the terms used in different sequential stages of the documents may have relevant information for the classification task. With this idea in mind, this method enriches the documents of the minority class with the partial documents read in the first 4 chunks. These chunks correspond to the minority (depressed or positive) class. Also TVT uses the complete documents (chunk 10). All this information is considered as a new concept space for a CSA method.
TVT naturally copes with the sequential caracteristics of ERD problems and also
gives a tool for dealing with unbalanced data sets. Preliminary results of this method
in comparison to CSA and BoW representations [
Sequential Incremental Classification (SIC) Sequential Incremental Classification
(SIC) is a very simple method. During the training phase a dictionary of words is built
for each category, in which frequency of each word is stored. Then, using those word
frequencies, and during classification stage, a value for each word was calculated using
a function gv(w; c) to value words in relation to categories. gv takes a word w and a
category c and outputs a number in the interval [
Additionally, g!v(w) = (gv(w; c0); gv(w; c1); : : : ; gv(w; ck)) is defined, where ci 2 C (the set of all the categories). That is, g!v is only applied to a word and it outputs a vector in which each component is the gv of that word for each category ci. For instance, following the above example, we have: gv(`sushi`) = (0:85; 0:09; 0:5; 0:02); gv(`the`) = (0; 0; 0; 0);
We have called the vector g!v(w), the “confidence vector of w”. Note that each category ci is assigned a fixed position, i, in g!v (for instance, in the example above (0; 0; 0; 0) is the confidence vector of “the” and the first position corresponds to f ood, the second to music, and so on).
Classification is finally carried out, for each subject, by means of the cumulative sum of all words g!v vectors, in symbols: !d =
X g!v(w) w2S where S is the subject’s writing history. Note that !d is a vector with two components, one for the positive class (depressed or anorexic) and one for the negative (control) class. The policy to classify a subject as positive was performed by analyzing how !d changed over time (i.e. over “chunks”), as shown with an example in Figure 1 for a depression case. Subjects were classified as depressed when the cumulated positive value exceeded the negative one, for instance the subject in the figure was classified as depressed after reading the 5th chunk.
It is worth mentioning that, to compute gv we used other two functions, lv and weight, as follows: gv(w; c) = lv (w; c) weight (w; c)
– lv (w; c) values a word based on the local frequency of w in c. As part of this process, the word distribution curve is smoothed by a factor controlled by the hyperparameter . – weight (w; c) decreases lv in relation to the lv value of w to the other categories.
The more categories ci whose lv (w; ci) is high, the smaller the weight (w; c) value. The hyperparameter controls how sensitive this sanction is. 3
As we mentioned above, this year there were two tasks: one for the early detection of depression cases (task 1) and the other one for the early detection of people with anorexia (task 2). We only used the T RDS data set for setting the parameters of our methods because it is the largest and, therefore, it would seem to be more appropriate to obtain more confident statistics.
In order to find the best values for the parameters of our methods we perform a five-fold cross validation on the depression training set. Hence, we divided the T RDS set into five folds (see Table 1). These folds maintain the same proportions of both kind of users and were randomly selected. Also each fold was divided into 10 chunks like they were provided by the organizers. We trained the classifiers with four folds and tested with the fifth fold. This process was repeated four times more, always choosing different folds, and later the results were averaged.
We used the Flexible Temporal Variation of Terms (FTVT) described previously to represent the documents. For this representation, a decision must be made related to the number n of chunks that will enrich the minority (positive) class. We considered different values for n, particularly we selected n from 0 to 5 for setting the initial chunks used.
FTVT was evaluated with different learning algorithms such as Logistic Regression (LR), Support Vector Machine (SVM) and Naïve Bayes (NB), among others. We used the implementation provided in the Scikit-learn package for Python 2.7 with the default parameters. That is, penalty = l2 and C = 1, for both SVM and LR.
We used the probability p assigned by the classifier to decide when to stop reading a document and giving its classification. Thus, our approach considered that when the probability p assigned to the positive class exceeds some particular threshold (p ) the instance/document is classified as positive. We used different thresholds : 0.9, 0.8, 0.7 and 0:6.
We evaluated the performance of our approaches with the early risk detection
error (ERDE) measure proposed in [
Due to space constraints, only combinations of n (parameter of FTVT), (probability threshold to classify an instance as positive) and the used classifier that allowed obtaining the best values of ERDE5 and ERDE50 metrics, are shown. These results are presented in Tables 2 and 3, respectively.
If we analyze Table 2, it can be seen that small values for n and a high threshold (more restrictive) generate a lower ERDE5. In particular, the best configuration for ERDE5 is n = 0 and p 0:8 with the SVM algorithm obtaining 13:58. A higher threshold means that it is necessary more confidence to classify a user as positive. This is because as the urgency level to decide is also high, what can be classified as positive has to be precise, otherwise the penalty is higher. On the other hand, with regards to the ERDE50 metric we can see in Table 3 that the best thresholds are a little lower than in Our five systems, three variants of FTVT (U N SLA, U N SLB and U N SLC) and two variants of SIC (U N SLD and U N SLE) were trained with the full training set of the pilot task 1 (T RDS ) and tested with the corresponding T E DS (see Table 5).In the same way, for task 2 the methods were trained with T RAX and tested with the corresponding T E AX (see Table 6). Both test sets were incrementally released during the testing phase of the pilot tasks.
In Table 5 we show the results of our 5 submissions and the results of those
systems that obtained the best ERDE5, ERDE50, F1, precision and recall in the eRisk
depression pilot task as reported in [
Table 6 shows similar results for the anorexia pilot task. As we can see, our system (U N SLB in this case) obtained the best ERDE5 again and U N SLD the best precision value. At this point it is important to note that we did not perform a parameter optimization of our methods for the anorexia task, such as we stated in the previous section. Then, it is not a minor aspect that our systems can perform well in a different domain from the used for setting the parameters. This independence of domain is such a really important aspect of the classifier systems for the optimization of real tasks.
With these results, we can conclude that our proposals are very reasonable and competitive alternatives for ERD tasks.
This article presented the participation of UNSL at eRisk 2018 Pilot tasks on Early Detection of Depression and Anorexia. We used two completely different approaches to deal with those tasks: one based on the FTVT representation and other on a simple method named SIC. Those approaches showed to be very effective on both types of tasks obtaining the best ERDE5 value over all participants in both tasks and the best precision value for the anorexia task. Besides, in the ERDE50 measure, although we did not achieve the best value, our results were very close to it. Thus, the performance of our systems seem to indicate that the used methods are very robust approaches for ERD tasks.
However, there are other aspects of our systems that we consider relevant. First of all, they are completely independent of the domain because they only relies on the terms present in the training set. That is to say, they do not require a costly process of feature engineering or very complex hand-crafted features specific of the problem under consideration. That independence was evident in this Lab where only a parameter setting was carried out on one of the data sets (depression) and the same configuration was used in the other one (anorexia). The excellent results obtained in both cases provide strong evidence of this independence and robustness. Another aspect that deserves special attention is that both approaches use very simple rules to decide when to stop reading and classify a user as positive. That contrasts with other approaches that require very complex and difficult to understand methods to make those decisions.
As future work we plan to extend the use of FTVT and SIC to other ERD problems such as the identification of sexual predators, people with suicide tendency and early rumour detection. In those cases, we consider that the ease and simplicity that our methods provide to be migrated from one domain to another make these applications a rather trivial process.