In this paper, we describe the third participation of the Information Management Systems (IMS) group at CLEF eHealth 2019 Task 1. In this task, participants are required to label with ICD-10 codes health-related documents with the focus on the German language and on non-technical summaries (NTPs) of animal experiments. We tackled this task by focusing on reproducibility aspects, as we did the previous years. This time, we tried three di erent probabilistic Nave Bayes classi ers that use di erent hypothesis on the distribution of terms in the documents and the collection. The experimental evaluation showed a signi cantly di erent behavior of the classi ers during the training phase and the test phase. We are currently investigating possible sources of biases introduced in the training phase as well as out-of-vocabulary issues and change in the terminology from the training set to the test set.
In this paper, we report the experimental results of the participation of the IMS
group to the CLEF eHealth Lab [
The main goal of our participation to the task this year was to test the e ectiveness of three simple Nave Bayes (NB) classi ers and provide the source code (as we did in the previous years) to promote failure analysis and comparison of results.3
The contribution of our experiments to this task can be summarized as
follows:
{ A study of a reproducibility framework to explain each step of the pipeline
from raw data to cleaned data;
{ An evaluation of three simple classi ers that use an optimization approach
based on the two-dimensional representation of probabilistic models [
We submitted 3 o cial runs, one for each classi er, and we will prepare a number of additional non-o cial runs that we will evaluate and compare in order to study the change in performance when adding more information in the pipeline. 2
In this section, we summarize the pipeline we used in [
In order to produce a clean dataset, we followed the same pipeline for data
ingestion and preparation for all the experiments. We used the tidytext [
We used three NB classi ers for the classi cation of documents. In particular,
the three classi ers di er in the model (the mathematical description) of the
distribution of documents and terms. We followed our previous work on the
visualization of classi ers for hyper-parameters optimization [
In the multivariate Bernoulli model, an object is a binary vector over the space of features: fk
where fkjc is the parameter of the Bernoulli variable of the k-th feature in the c class.
In the multinomial model we have one multinomial random variable which can take values over the set of features: oj (N1;j ; :::; Nm;j )
M ultinomial( fjc) : where Nk;j indicates the number of times feature fk appears in the object oj .
In the Poisson model, an object oj is generated by a multivariate Poisson random variable:
Ni;j
(1) (2) (3) 3
We submitted three o cial runs, one for each model. The goal of these experiments is to compare the e ectiveness of the three classi ers and study the di erence among them in a failure analysis (post experiments). 3.1
The dataset contains 8,793 documents: 7,544 documents for training, 842 for development, and 407 for testing. After we processed the training and development set, the number of features (words) after stemming and stopwords removal is 74,002. There are a total of 233 categories in the German Classi cation Diseases (ICD10) codes database. The training set contains 230 categories (categories H65-H75, R10-R19, and R20-R23 are missing), the development set contains 156 categories, while the test set 119 categories. Therefore, there are 112 categories that are in the training set but not in the test set and, surprisingly, one category, R10-R19, which is in the test set but not in the training set. Given this distribution of categories, we merged the training and the development set into one dataset that we used to train the classi ers with a k-fold cross validation. 3.2
In order to optimize the three models, we used the F1 measure for each binary classi er (one for each category). In this paper, we report the three measures used by the organizers, Recall, Precision and F1, both the macro-averaged measures (values averaged across all the categories) and the micro-averaged measures (as the sum of all the confusion matrices produced by each classi er). In the tables of the results, we use capital letters to indicate macro-averages (for example Recall) and small letters for micro-averages (for example recall). We report two macro-averaged F1 measures: one computed on the values of the macro-averaged Precision and Recall, the other (indicated with a `*' at superscript) computed as the average of the F1 measures. For those categories without positive documents (in the development or test set), by default we assign a recall of 1 and a precision of either 0 (when there is at least one false positive) or 1 (when no false positive is found for that category). 3.3
O
cial Runs We used a k-fold cross validation approach to train the models and optimize the hyper-parameters of the two-dimensional approach (more details in the nal version). We used all the training and development documents for the cross validation with k = 10, and we trained a binary classi er for each class in the corpus. Nine out of the ten folds have 838 documents while the latter has 844 documents; consequently, we have on average about 7,540 training documents and 840 validation documents.
The average results across the 233 classes on the ten validation folds are the ones shown in Table 1.
In Table 2, we show the results of the classi ers that use the training set to estimate the probabilities and the development set for the evaluation (with or without the optimization of the decision line). We can see that these results are in line with the one reported during the k-fold cross validation approach with a slightly di erence in the micro-averaged performance when the non-optimized version is used compared to the optimized one. This may indicate that there is some over tting for those categories with too few training documents; in fact, even with just one positive training documents the algorithm tries to nd the best tting line.
In Table 3, we report the results on the test set. Surprisingly, the behavior of the classi er is completely di erent from the one we observed during the training/development phase. Macro-averaged measures are still satisfactory, but we have to remind that we introduced a correction in the computation of the recall for those categories without positive documents that may have a ected (positively) the averages. 4
We are currently analyzing possible sources of error in the test phase. One thing
that seems evident from the analysis is that the distribution of probabilities of
terms has changed from the development to the test set. We can observe this
from a comparison of Figures 1 and 2. These gures show the two-dimensional
distribution ([
We are also studying whether this signi cant change in the position of the cloud of documents (corresponding to the probability of documents) is related to a di erent distribution of words in the test set or to a change in the vocabulary of terms in the test set.
There is also a chance in some bug in the source code that we were not able to nd until now. 5
In this work, we presented our participation to the CLEF eHealth Task 1 on the classi cation of medical documents. We presented the evaluation of three probabilistic classi ers based on di erent assumptions on the distribution of words, namely binary, multinomial and Poisson. We described a method to process the 0 0 5 − 0 0 0 1 − 0 0 5 1 − −1500 −1000 x −500 0 documents and optimize the classi ers according to the two-dimensional representation of probabilistic models. The results on the test set were very low despite a correct training/development phase that showed promising results. This opened new ideas about how to better control the training/development phase of the classi er and how to study possible sources of errors in the assumptions made during the training. 0 0 0 5 − 0 0 5 1 − −2000 Fig. 2. Poisson model test set. Distribution of positive (red) and negative (black) documents for the category II.