This paper describes the participation of ECSTRA-INSERM team at CLEF eHealth 2016, task 2.C. The task involves extracting ICD10 codes from death certificates, mainly described with short plain texts. We cast the task as a machine learning problem involving the prediction of the ICD10 codes (categorical variable) from the raw text transformed into a bag-of-words matrix. We rely on probabilistic topic models that we evaluate against classical classifiers such as SVM and Naive Bayes. We demonstrate the effectiveness of topic models for this task in terms of prediction accuracy and result interpretation.
Completing death certificates is a routine task in hospitals and healthcare institutions. In France, the death certificates are produced by physicians and transmitted to the French Epidemiological Center for the Causes of Death (C´epiDC)6. Beyond the administrative and personal information, the death certificates usually contain a free-text description of the cause(s) of death. To monitor population’s health, free texts are converted by the C´epiDC into formal standardized codes, usually derived from the International Classification of Diseases (ICD) taxonomy. These codes also serve as a basis for mortality and epidemiology studies.
The ICD taxonomy covers a wide range of diseases, symptoms, signs, procedures, and other content related to diseases7. The World Health Organization
issues separate versions of ICD per language/country. In this paper, we use the French release of ICD, which is now at its 10th revision (called ICD10). It covers more than 20,000 codes including diagnoses and procedures, but only a subset of theses codes can be causes of death. An example is provided in Table 1.
Requiring manual work and expertise, the task of ICD10 code extraction
from text is quite time-consuming because the ICD10 taxonomy contains
thousands of possible causes of death. Within the CLEF eHealth 2016, the task 2.C
focuses on the problem of automatic extraction of the causes of death from the
textual description [
In this paper, we describe our system to automatic cause-of-death extraction
following the first approach. Concerning the used methods, we mainly focus on
probabilistic topic models [
Topic models are probabilistic approaches to discovering hidden structures
(commonly called topics) from text. Topic models have shown significant efficiency
over baseline models for various language modeling and text mining tasks, like
topic discovery [
Latent Dirichlet Allocation (LDA) [
Relying solely on word co-occurrence, LDA model is fully unsupervised. In
this work, we rely on a supervised extension of LDA called LabeledLDA [
To better explain how LabeledLDA works, we rely on the plate notation given in Figure 1.b. In the traditional machine learning methods, the words (variable w) directly influence the prediction decision. For example, in Naive Bayes classifier, the decision depends on the word frequencies per class. In LabeledLDA, The prediction decision depends on both document’s words and topics. That is, the topic’s proportions of the document to be classified are taken into account when computing the document-code probabilities. On the other hand, the topic extraction with LabeledLDA is influenced by the document’s classes. Thus, the documents from the same class are likely to be linked to the same topics. This feature allows LabeledLDA to take advantage of the hidden structures of documents as well as the topic-code relations. For example, the code N189 (chronic kidney disease) is likely to describe a topic about loss in kidney function, that in turn can be described using the words “kidney”, “renal”, “failure”, etc.
The efficiency of LabeledLDA for text classification has been proved in many
tasks, including diagnosis code assignment to medical summaries [
The parameters of LabeledLDA are fixed empirically in such a way to maximize the predictive scores on a held-out dataset (20% documents randomly sampled from the whole dataset). As such, the number of topics K is equal to the number of codes (3,231), = 0:005; = 0:07. The remaining hyperparameters are learnt from data. In addition, we perform two experiments by setting the number of iterations to 20,000 and 50,000 respectively.
The two other classifiers used in this work are SVM and Naive Bayes. SVM is a non-probabilistic binary classifier that maps the documents in such a way to maximize the gap between those from opposite classes. For our non-binary problem, we use the one-vs-one technique that consists of learning a separate classifier of each pair of classes then taking the class with the largest weight. Naive Bayes is a simple and widely-used probabilistic classifier based on the calculation of conditional probabilities (probability of words given the outcome class). The process is then easily inverted using Bayes theorem and word independence assumption. To run these methods, we rely on Python and “scikit-learn” package8, specifically “BernoulliNB” and “LinearSVC” implementations. For SVM, we set the penalty parameter c to 1.0. The rest of parameters are left to default values. 3
The C´epiDC corpus has been created by the French Center for Epidemiology
and Medical Causes of Death (C´epiDC) specifically for the CLEF eHealth 2016
contest [
Also, in order to reduce dimensionality, we filter out frequent words (contained in more than 50% of documents) and rare words (contained in less than 3 documents). We also remove stopwords and numerics. The preprocessed text documents are then mapped into a bag-of-words representation where the words are weighted according to their presence/absence in the document (binary values). Table 2 gives an overview of the preprocessed C´epiDC sample used for training. 4
The methods are evaluated and ranked based on micro-averaged F-score (harmonic mean of precision and recall weighted by the class size). LabeledLDA
model is put against other traditional machine learning methods: Naive Bayes and SVM. We have chosen these methods among others because they gave the best scores.
The obtained results are given in Table 3. For the official evaluation, we have submitted a run of LabeledLDA with 20,000 iterations because the running time was too long. In Table 3, we also evaluate a run of LabeledLDA with 50,000 iterations. Our two official submissions are given in italic whereas the best scores are given in bold.
As can be seen, SVM achieves the best F-score compared to other
methods, followed by LabeledLDA model. In terms of micro-averaged F-score, SVM
achieves 75.19% while LabeledLDA arrives second with 73.53%. Compared to the
other 6 participant systems, our LabeledLDA-based system was ranked fourth
(based on 20,000 iterations). When rising the number of iterations to 50,000,
LabeledLDA would be ranked second. Based on the official results from [
Although the superiority of SVM over LabeledLDA in terms of precision
score, LabeledLDA’s most interesting advantage resides in offering more
easilyinterpreted results. In Table 4, we show the top-10 words characterizing topics
extracted with LabeledLDA model. To complete these results, we have asked a
physician to decide of the significance of topics by looking at the top-10 words
globally. The physician has agreed that topics were extremely informative. Most
of the causes of death represented here could easily be recognized from the
associated topical words.
pleural (pleural), m´esoth´eliome (mesothelioma), malin
(malignant), droit (right), m´esoth´elium (mesothelium), gauche (left),
m´etastatique (metastatic), terminal (terminal), pl`evre (pleura),
´epithelioide (epithelioid)
sarcome (sarcoma), thoracique (thoracic), pulmonaire
(pulmonary), multim´etastatique (multi metastatic), art`ere (artery),
r´ecidive (recurrence), ´evolutif (evolutive), thorax (thorax),
pari´etal (parietal), paroi (lining)
insuffisance (failure), surr´enalienne (adrenal), aigu¨e (acute),
chronique (chronic), diab`ete (diabetes), surr´enale (adrenal),
r´enal (renal), insulino (insulin), requ´erant (petitioner),
hypophysaire (hypophyseal)
´epilepsie (epilepsy), AVC (stroke), ´epileptique (epileptic),
enc´ephalopathie (encephalopathy), alzheimer (alzheimer),
evolu´ee (evolved), syndrˆome (syndrome), s´equelles (aftermath),
post (after), maladie (disease)
The problem of ICD code extraction has been investigated from a larger
perspective involving code assignment to various types of medical documents. The cause
of death may be considered as a specific task. The majority of these works have
focused on English documents. Number of them have been published with the
Computational Medicine Center’s 2007 medical NLP contest involving ICD code
assignment to radiology reports [
Apart from this contest, in [
In [
In this paper, we have described our system used to extract ICD10 codes from death certificates within the CLEF 2016 eHealth, task 2.C. Our system is based on LabeledLDA model: a supervised topic model for topic discovery and document classification. Even if LabeledLDA does not outperform SVM classifier, LabeledLDA provides an explanation of the results (why such code for such document?) and allows a more in-depth understanding of the classification mechanisms. This feature is obviously a clear advantage over machine learning methods, like SVM that are usually based on complex mechanisms and consequently less suitable for human interaction.
As a promising future direction, we believe that the performance of
LabeledLDA can be improved by integrating an “active learning” component. That
is, the active learning allows capturing and integrating user’s feedback into the
learning process [