This report describes two methods implemented for the CLEF eHealth 2016 Task 1 challenge. They consist of: a) a feed forward neural network; and b) a random forest for classi cation and a feed forward neural net, applied to automatically ll in medical handover forms using synthetic medical records as inputs. Both approaches are interesting because they rely on word embeddings, are domain independent, and are feature engineering free. We discuss the complexity of the task, and the impact in our models, having too many output classes and a limited amount of training data. The performance of the methods are based on traditional classi cation metrics (e.g. precision, recall, and f1-score) on the macro and micro averaged level, and focus on two sets of labels: a) the "NA" tag, which recognize data that is irrelevant and therefore should be excluded from the form; and b) all other tags, which account for the di erent elds of the form. The neural network achieved an F1-score of 0.8 (for the "NA" tag) and a macro-averaged F1-score of 0.308 and a micro-averaged result of 0.514 (for the remaining categories), while the ensemble pipeline got 0.813 (for the "NA" tag) and 0.345 and 0.503 for the macro- and micro-averaged rates on the rest of the labels.
This year's CLEF eHealth challenge consists of 3 tasks[
This report describes two statistical approaches to solve the rst of these
tasks: Task 1: Handover information extraction.[
The outline of this report is set as follows: In section 2, a description and analysis of the datasets and the methods is presented; in section 3, the experiments results are shown and explained; section 4 includes conclusions drawn from this work, as well as encountered issues and future work; nally, there is an appendix section that includes additional information referenced in the report. 2 2.1
Three datasets were released for the purposes of this challenge[
Table 1 presents an overview of the datasets. These numbers account for tokens found in the data, with punctuation removal3 as the only pre-processing step applied. As it is shown in the table, the datasets are roughly the same in terms of size, namely: number of records included, number of tokens, and number of word types. But nearly half of the word types present in the validation and testing set are not seen in the training group (50.04% and 56.55%, respectively, when considering stopwords, and 56.62% and 62.65% excluding them). This is, de nitely, an obstacle to overcome; hopefully, the vector representations are going to capture enough semantic meaning to deal with it.
Note that, with no prior handling, constructions like: 'forty-eight', 'selfcaring', 'self-in ammatory' ; numbers, such as: '81', '220' ; symbols and punctuation: '@',',','.' ; misspellings and wrongly spaced words: 'bed2', '1pm', 'gout.', 'litres/nasal', 'urine.and', 'gastroscopy/colonoscopy' are treated as single tokens. This criterion follows the way in which the dataset was originally tokenized and listed with the given features.
Already at this stage it can be pointed out, one of the most important limitation to this report's approaches, and to other techniques of the same nature: there is not enough training data in order to e ectively train the neural networks. Considering the number of parameters to be learnt, these models might result too complex to train. 3 The following symbols were left, as there is a vector representation for them: ' ', '^', '@','=','>',' ','+','&','$','%','#' In this task, we are going to learn patterns from the data so as to ll in a xed handover form. The description of this form can be found in the dataset paper, for the purposes of this explanation, it is relevant to know that it consists of 36 tags/labels; one of these is the 'NA' label, which accounts for information that shouldn't be included in the form. Naturally, this tag covers the most number of tokens, and as we are dealing with a multiclass classi cation task, this di erence in label-group sizes will play a signi cant role at prediction time, speci cally when computing the resulting averaged metrics.
Looking at the training and validation sets, we can nd out that there is a mismatch between the tags that are included in one and the other. Namely, the training dataset includes 36 labels, 3 of which are not found in the validation set (tags: 'Appointment/ Procedure ClinicianGivenNames/ Initials', 'Appointment/ Procedure Ward', 'Appointment/ Procedure City' ); and the validation set includes 36 tags as well, 3 of which are not seen in the training set (tags: 'PatientIntroduction Title', 'Appointment/ Procedure ClinicianTitle', 'Appointment/ Procedure Hospital' ). Finally, the testing set is in agreement with the training label data. This di erence will have an impact on the validation scores, but not on the testing ones.
Conceptually, the statistical models to be trained will try to t the conditional probability of a label given certain word tokens. It is worth analysing the data at the tag level so to get an idea of how complex the task is. Table 5 and Table 6, included in the Appendix, show the empirical distribution shaped from the training data, and a word type count overlap breakdown. The overlap summary was done considering the validation and the training set (at the moment of writing this report, the testing set labels were not released), and it includes stopwords in the counts. As it can be seen in the empirical distribution, there is one category which concentrates the majority of the tokens (the "NA" tag, leaving a very low probability mass for the rest of the labels. Clearly, some categories are easier than others to predict; but, in the end, this is going to be dependant on: a) the number of samples; and b) the token overlap of the category. These two factors will a ect the quality of the trained word embeddings.
In the next subsections, the details of the two models are provided. 2.2
The rst approach consists of a one hidden layer context window feed forward neural network. Following the idea of constructing a pipeline that is domain independent, there are no features derived from medical data enrichment, and the neural network makes use of semantic features only, i.e. word embeddings.
From a Natural Language Processing point of view, we know that languages present a high level of ambiguity, and if we, furthermore, take into account the overlap schema presented in the previous section, it seems like a good idea to include a context window on the token to be predicted.
As mentioned before, even this simple architecture might have too many
parameters to be trained with the amount of data we have. As a way to help this
situation, the word embeddings are initialized using the pretrained Googlenews
Word2Vec representations4[
The network has two weight matrix and two bias parameters. The matrix shown as W1 corresponds to the word embeddings, initialised uniformly and intersected with the pretrained embeddings, while the matrix W2 and the bias vectors b1 and b2 are uniformly initialised. There is a non-linear behaviour in the net, introduced by the hyperbolic tangent function on the hidden layer, and the output layer is a classi cation step achieved by using a softmax activation function. The output layer has a dimension of 39, because of the 36 tags in the training/test set and the 3 added tags found in the validation set, as explained in the previous section.
The neural network is trained with Stochastic gradient descent, and
backpropagation[
2.3
In this second method, the pipeline is based on a layered prediction. First, a
random forest[
Again, both, the random forest as well as the neural network only make use of semantic features. It could be the case that the two models use di erent or extended types of features in order to increase their performance. As an advantage, these two methods could possibly learn di erent patterns from the data, helping the nal prediction quality. But, at the same time, the errors that are made in the rst step, are further passed to the second model; because falsepositive predictions by the random forest are taken out from the sample set to be predicted later by the neural net, and false-negatives are going to be included by the neural net in one of the remaining categories as false-positives values.
The random forest uses decision trees for classi cation, with 100 trees and a maximum depth of 5. The neural network is trained as described in method 1. 2.4
The neural networks described above utilize pretrained word embeddings as features. Given the small amount of data for training, and considering the novelty, with respect to word types, that are included in each dataset, if a word has not been seen in the training data and has no representation in the Googlenews pretrained embeddings, then it will be assigned a randomly generated vector. In these cases, there is no semantic information to use so as to assign a class.
Table 2 shows the statistics of the previously described cases for word tokens and word types in the validation and testing set.
Dataset Validation 231 0.03 160 Testing The performance of the methods measure the precision, recall, and F1-score using the conlleval evaluation script, as implemented in the CoNLL 2000 Shared Task on Chunking5.
In this section, the results for the two methods are presented and compared.
At the moment of submission, an error when writing the output le produced an alteration in the order of the predicted tags, completely mixing the results. The associated scores are not included in this report. 3.1
While training, the word embedding vectors increase their norms, this e ect is re ected in Figure 2 and Figure 3, where a PCA plot of the word representations before and after training is shown. In this graph, the 36 training tags are plotted (overlapped words appear as a separated colour). It does not seem visually clear in 2 dimensions how the data could be separated; but the further transformations of these representations and the non linearity added by the network are able to identify semantic regions, to some extent. In this case, the random forest uses a randomly initialized matrix intersected with the Googlenews pretrained embeddings as input features, and predicts whether the token belongs to the "NA" label or not. Later on, the neural network of the second step discriminates between the remaining labels. This latter model was trained using a context window of 7 for 50 epochs.
Figure 5 presents a PCA plot of the word embeddings corresponding to the tag "NA", the remaining tags, and also separates word types that belong to "NA" and, at least, some other label.
Table 4 presents a summary of the ensemble method results. A detailed, per tag, analysis can be found in Table 8. in their recall (lowering the false-negatives samples), and even though negatively a ecting the precision, the gain is big enough to push the F1-relation up. But for some categories this statement does not hold, and it is the sum of these misclassied tokens which causes the micro-averaged reduction. Examples of this negative behaviour can be found in labels: a) "MyShift Input/ Diet", where previously correct tokens are now assigned to other classes, while also wrongly predicting tokens as members of this tag; b) "MyShift Status", where 79 false-positives are added just for gaining 2 tokens in true predictions; and c) "PatientIntroduction AdmissionReason/ Diagnosis", case in which 90 false-positives are included, with an advantage of only 3 new true-positive samples. 4
In this report, a feed forward neural network, and a random forest - feed forward neural network ensemble method are presented as solutions for the CLEF eHealth Task 1 challenge. Both methods rely on semantic features and are domain independent (no medical features are used). While the neural network alone achieves a macro-averaged F1-score of 0.308 on the test set, considering 35 categories (all but "NA"), and an F1-score of 0.8 for the "NA" tag; the ensemble method produces better results with a macro F1-score of 0.345 on the 35 tags of the test dataset, and an F1-score of 0.813 for the "NA" label, increasing the precision but a ecting the recall metric. This gain in F1-results suggests the second method performs better when considering the entire set of labels. On the other hand, from the micro-averaged perspective, the former method gets an F1-score of 0.514 for the 35 tags, and the latter an F1-score of 0.503, which can be explained from the relation between the general raise of the recall and the decrement in the precision.
As explained in the analysis section, the handover form presents a large number of tags, while some appear to be easy to learn, some others are clearly a complex task. One of the main problems limitations, given the Machine Learning methods implemented, is the amount of data available for training.
In this report's pipelines no pre-processing steps were applied, and the negative e ects of this decision were explicitly pointed out in the analysis. There should be a prior stage in which abbreviations, misspellings, and errors during tokenization are treated.
While this work shows that semantic representations are able to help in this task, most likely, a higher performance could be achieved by incorporating features of other types; lexical features such as part-of-speech tags, or dependency parsing, as well as features resulting from external taggers. Moreover, a potentially useful characteristic, not exploited in this work, is the natural structure of the medical records, for instance, word or sentence locations, and tag-precedence.
41 29 18 168 43 122 171 63 0.931 0.927