We present concept detection and normalization experiments on the French and English CLEF eHealth 2017 death certificate datasets. For this purpose, we start from our last published system, which relied upon dictionary projection and supervised multi-class, mono-label text classification using simple features. We extend this system in several dimensions with multi-label classification and new features, including an additional combination of dictionary and classifier. Because it only relies on the material provided by the task organizers, we could apply the same system to both the French and English datasets. Its results, registered as unofficial runs, equal or exceed those of the best submitted systems.
Most uses of medical information require its representation in a standardized,
normalized form: this form is abstracted away from the natural language
utterances that are generally used by health care professionals to record their
observations. For instance, diagnoses for diseases in hospitals or for causes of
deaths in death certificates are represented according to the International
Classification of Diseases (ICD-10), a large classification maintained by the World
Health Organization. Actually, causes of death in death certificates are initially
described and recorded in natural language. National and international
statistics, however, require their normalization according to ICD-10 classes, following
specific rules established by the WHO. Efforts at automating this normalization
process (also called coding ) have been made to lighten the burden of human
coders [
ICD-10 coding of death certificates is an instance of concept detection and
normalization in very short texts. It was proposed as a shared task in CLEF
eHealth 2016 [
The CLEF eHealth 2017 Shared Task [
Additionally, the 2016 task was continued for the French data: a line-level gold standard was provided for the French training data (which was extended with respect to the 2016 dataset), and line-level ICD codes were expected to be produced in that sub-task.
We built on these datasets to investigate the following points:
1. To examine the differences between the line-level sub-task and the
certificatelevel sub-task. A natural hypothesis is that training a classifier on
certificatelevel annotations will lead to a loss in classification quality. The question is
how much. Besides, for longer (certificate-size) texts, the optimal
combination of features might be different from that found for shorter (line-size)
texts.
2. To examine the differences between the French certificate-level sub-task and
the English certificate-level sub-task. Various reasons might make one more
accurate than the other, including differences in the size of training data or
dictionary, differences in intrinsic properties of each language such as
inflection, differences in coding conventions, and differences in the distribution of
ICD codes in the provided datasets.
3. In our previous work, as in [
Being members of the organizing team, we prepared a system in parallel with the participants and submitted unofficial runs that we report in the results section. 2
2.1
– Lines (aligned data): training data, test data and evaluation are performed at the level of each line of a certificate. – Certificates (unaligned data): training data, test data and evaluation are performed at the level of each full certificate.
– Training: 2016 training data: certificates of 2006–2012 – Development: 2016 test data: certificates of 2013 – Test: 2017 test data: certificates of 2014
We tuned our methods on the training data. Then we retrained them using the tuned parameters on the training+development data and applied them to the test data. 2.2
– Certificates (unaligned data): training data, test data and evaluation are performed at the level of each full certificate.
– Training: training data except the last 666 certificates. This number was designed to obtain a similar ratio of examples in the training vs. development splits as in the French dataset. – Development set: the last 666 certificates of the training data.
We used the training and development sets as for the French data to tune and train the system.
Our main methods were presented in [
Dictionary calibration: Because dictionary entries are sometimes ambiguous, projection sometimes leads to false positives. We train a classifier to determine whether an ICD code assigned by the dictionary is likely to be correct or incorrect. Then we apply this trained classifier to the ICD codes produced for the test split: if it yields a negative answer for a code, we discard this code from the output of the dictionary. We call this a calibrated dictionary. The features given to the classifier are the produced ICD code and the bag of words obtained for the input text (line or certificate, depending on the dataset).
Supervised classification (linear SVM), with bag-of-word features.
The list of ICD codes and coding rules has evolved over the years; it is therefore useful to take into account the year coding was performed (coding year), which we encode as a set of interval features. For instance, a certificate with coding year 2011 receives features >2007, >2008, >2009, >2010, >2011, <2012, <2013, <2014, <2015, <2016. We observed that coding year is not relevant in the English dataset of CLEF eHealth 2017, because all its certificates belong to the same year (2015). It is therefore not included in the features for that dataset.
In our previous work [
In the present work, we use a multi-label classifier, whose precision and recall are more balanced. In this context, character n-grams decrease precision more than they increase recall, and are thus a less useful feature to optimize F1-score. Instead we used word bigrams, with the aim of increasing precision. We thus start from the following features: bag of word unigrams (noted u), bag of word bigrams (b), coding year (y ).
Union (and intersection) of the labels predicted by (calibrated) dictionary
projection and classification. This is a crude way of combining these two
methods, but it proved effective in our former experiments [
In [
In addition to [
Multi-label classification, with unigrams and bigrams, plus coding year for the French datasets (u b y, shortened as uby ), was used to submit unofficial runs LIMSI 1 for each dataset. Its union with dictionary projection was submitted as unofficial runs LIMSI 2 for each dataset. 4
Here we test the above methods on the CLEF eHealth 2017 training datasets to choose the best performing ones: – French, lines (in CLEF terms, aligned ) – French, certificates (in CLEF terms, raw ) – English, certificates (in CLEF terms, raw ) As mentioned above, we first submitted for each dataset one basic run with a multi-label, supervised classifier (run1) and one with the union of this classifier and calibrated dictionary projection output (run2). We also submitted runs with variant features. 4.1
We compared various configurations on the training data. In this purpose, we
trained the system on the training set (the CLEF eHealth 2016 training set)
and tested it on the development set (the CLEF eHealth 2016 test set). The
presented results are thus comparable to those published by CLEF eHealth 2016
participants (who indeed had less time to prepare their systems). For want of
time, we did not use cross-validation on the training set, which would be more
appropriate. Evaluation was performed internally using scikit-learn functions,
which produce approximately the same results as the official CLEF eHealth
evaluation program. The results are presented in Table 1, upper pane (FR, line).
– Dictionary with calibration (column Cal Dict ) obtains 78.1% F-score: this
is a basis for comparison. Note that this is still much below the best CLEF
eHealth 2016 system (F=84.8), which was dictionary-based, but also used
post-processing
– Multi-label classifier (column Sup): with the baseline features (uby ), the
supervised classifier is ten points above the calibrated dictionary and above
the best CLEF eHealth 2016 system and our previously published results
(F=85.9) which used the union of the mono-label classifier and of the
calibrated dictionary [
Not displayed in the tables, we noticed that quite a few dictionary entries have more specific ICD-10 codes with an additional digit, whereas the training datasets never had such codes. We examined the impact of removing this extra digit from the dictionary. Indeed, this drastically improves dictionary projection by 6pt F-score. However, this has nearly no impact on F-score in supervised and union configurations, where it marginally decreases precision and increases recall; this shows that on the one hand, the supervised classifier independently learns to recognize the involved expressions (compensates for recall) and filters out the longer codes, because they are not seen in the training set (precision).
On the development set, the best configurations are close to each other. Nevertheless, we kept the one with the fixed dictionary and all features for running on the test set: unigrams, bigrams, coding year, age, dictionary projection results as features (ubyafst, run 3), and union with dictionary projection results (run 4). We compared the same configurations as for the line-oriented dataset, using the same training and development sets, in their line versions. The results are shown in Table 1, middle pane (FR, certif.).
– Dictionary with calibration obtains an F1-score comparable to that on the line-level dataset. – Surprisingly, the multi-label classifier looses very little (or even gains a little) in F1-score when applied to the certificate-level dataset. A closer inspection shows that its precision increases whereas its recall decreases. We assume its increase in precision is related to its better handling of contextual codes. We return to this point below. – Union of dictionary and classifier (multi-label) increases the F-score by between 0.3pt (ubfst ) and 1.5pt (uba). As can be expected, it is less useful when dictionary projection results are already provided as features to the classifier. – Age contributes 0.7pt F-score. – Introducing dictionary projection results as features contributes about 1.2– 1.3pt F-score to the classifier alone. They contribute 0.5pt F-score in union configurations, where dictionary projection results are also added as predicted labels directly.
Fixing overlong codes in the dictionary leads to the same observations as in the line-level dataset.
The same overall observations hold as on the line-oriented dataset, therefore we selected the same configuration to run on the test set (ubyafst ). Comparison with line-level dataset Training with gold labels at the level of full certificates is a more difficult condition than training with gold labels at the level of individual lines because the identification of specific features in longer texts is made more difficult. However, training with gold labels at the level of full certificates is likely to help identify labels that depend on a larger context than a single line. For instance, this is the case for diagnoses that are coded differently depending on whether or not they are caused by a trauma, such as those in Chapter XIX (S00–T989, Injury, poisoning and certain other consequences of external causes ) of ICD-10. Specifically, error analysis revealed that statements mentioning a hemorrhagic shock (choc hémorragique) should be coded T794 (traumatic shock ) if a trauma is mentioned elsewhere in the death certificate, but are often confused with R571 (hypovolemic shock ), which applies in the absence of a trauma. Certificate-level analysis can thus be beneficial for such codes: the best line-level classifier (ubyafst ) over-predicts R571 and underpredicts T794, whereas the certificate-level classifier (ubyafst ) is much closer to the true distribution of these two codes.
Besides, evaluating with gold labels at the level of full certificates is a more
lenient condition than evaluating with gold labels at the level of individual lines:
a label may be incorrectly attributed to a given line (false positive for line-level
evaluation) but be present elsewhere in the same certificate (true positive for
certificate-level evaluation). Additionally, since the line-level alignments were
performed automatically [
The only differences from French when applying the system to English data are the handling of apostrophes in word segmentation, the choices of stop word lists (based on those in NLTK) and stemmers (FrenchStemmer and EnglishStemmer from nltk.stem.snowball).
We compared various configurations on the training set. In this purpose, we split it into a test split (the last 666 certificates, ordered by DocID number) and a training split (the other certificates). For want of time, we did not use cross-validation on the training set, which would be more appropriate.
We recall that coding year is not relevant in the English dataset and is therefore not included in the features for this dataset.
– Dictionary with calibration obtains 4pt F1-score less than for French. This is not directly linked to their relative sizes, which is larger for English (170,282 lines) than for French (147,342 lines). – The multi-label classifier obtains better results than on the French dataset: we return to this point below. – Union of dictionary and classifier (multi-label) increases the F-score by between 0.4pt (ubfst) and 1.4pt (uba). As can be expected, it is less useful when dictionary projection results are already provided as features to the classifier. – Age decreases F-score by 0.5pt. The reason why this is so in the English dataset remains to be investigated. – Introducing dictionary projection results as features contributes about 1.1– 1.5pt F-score to the classifier alone. They contribute 0.6–0.7pt F-score in union configurations, where dictionary projection results are also added as predicted labels directly.
Not displayed in the tables, fixing overlong codes in the dictionary improves dictionary projection by only 1pt F-score. However, it has nearly no impact on F-score in supervised and union configurations.
On the development set, the best configuration is that with the fixed dictionary, unigrams, bigrams, dictionary projection results as features (but not the age feature), and union with dictionary projection results. We retain it for running on the test set (ubfst, runs 3 and 4).
Comparison with the French certificate-level dataset Although the English dataset is smaller than the French certificate-level dataset, the results on the development set are better on the English dataset. This may be explained by the smaller number of codes and their different distribution in the English dataset. 4.4
We show the results obtained for our four unofficial runs in Table 2. For the French datasets, the line-level classifier loses about 2pt F1-score from the development set to the test set in each of the four tested configurations. This shows that it did not overfit the training set.
The certificate-level classifier loses about 7pt F1-score, which is a much higher loss. This comes from a loss of 10pt in recall, whereas precision is maintained overall or even increased. This shows that this classifier overfits the training set.
Therefore, compared to the line-level classifier, the certificate-level classifier loses 10pt in recall and 5pt in F1-score. A similar loss was observed in the results of the best CLEF eHealth 2017 participant (SIBM: –10pt recall, –4pt F1-score).
On the English test set, the certificate-level classifier loses about 5–6pt in precision, recall and F1-score compared to the development set. Cross-validation tests on the training set should now be performed to check whether this is a general property of the training set.
Table 2 reproduces the best results obtained by CLEF eHealth 2017 participants. It shows that the methods presented here obtain better results on the French datasets and comparable results on the English dataset. 5
We presented dictionary-based and supervised classification methods for ICD10 coding of French and English death certificates. These methods use various combinations of dictionary and other features and obtain state-of-the-art results on the CLEF eHealth 2017 datasets.
We saw that certificate-level training and evaluation obtained similar results as line-level training and evaluation on the development set, and even improved for some context-dependent codes. However, on the test set, the certificate-level classifier proved less robust than the line-level classifier. This is an encouragement to study methods that can align the gold-standard codes with the input lines of the certificates in the training data, as was done in the CLEF eHealth 2016 dataset.
The results obtained on the English dataset are higher than those for the French dataset. This is likely due to its smaller set of codes (about one third). The addition of dictionary output as features further increased the performance of the classifier, while reducing the contribution of the union with dictionary output. The addition of the age of death helped in the French dataset, but not in the English dataset.
Perspectives for further work include, among others, the exploration of word embeddings and other neural methods, the introduction of other dictionary sources, better combination of dictionary output and supervised classifier beyond simple union, context-dependent coding of those ICD codes that require it, and automatic line-level alignment of input codes in certificates before training. The differences between the French and English datasets remain to be investigated further, as well as the potential for their joint usage.