English. In this paper, we report the ongoing developments of our first participation to the Cross-Language Evaluation Forum (CLEF) eHealth Task 1: “Multilingual Information Extraction - ICD10 coding” (Ne´ve´ol et al., 2017). The task consists in labelling death certificates, in French with international standard codes. In particular, we wanted to accomplish the goal of the 'Replication track' of this Task which promotes the sharing of tools and the dissemination of solid, reproducible results.
include experimental data, parameter values, and
the source code of the program. Those readers
who wish to use the same approach of the paper,
hence reproduce the results, must reimplement the
whole process, which sometimes may be an
unfeasible task. The extreme of reproducibility is
‘replicability’, i.e. a perfect replica of a scientific
experiment. The discussion of the difference
between replicability and reproducibility is beyond
the scope of this paper
Reproducibility matters because the lack of reproducibility in science causes significant issues for science itself, for other researchers in the community, and for public policy. For example, Nature published a special issue about “Challenges in Irreproducible Research”1 where the examined cases showed that there is [ . . . ] a growing alarm about results that cannot be reproduced. Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions and funders all
have a part in tackling reproducibility.
Among many other problems, the article showed
that most of the drug validation studies (43 out of
67 studies) failed to reproduce. Another important
case concerned Science, where the Editor-in-Chief
retracted in 2015 a study of how canvassers can
sway people’s opinions about gay marriage
because: “ (i) Survey incentives were misrepresented
[ . . . ], (ii) The statement on sponsorship was false.
[ . . . ]” 2 There are also cases of papers retracted
by authors themselves because “After carefully
reexamining the data presented in the article, they
identified that data of two different hospitals got
terribly mixed. The published results cannot be
reproduced in accordance with scientific and clinical
correctness.” as declared in the note of retraction
of the paper “Low Dose Lidocaine for Refractory
Seizures in Preterm Neonates”
The problem of reproducibility in Information
Retrieval (IR) has been addressed by many
researchers in the field in the last years
The Natural Language Processing (NLP) community has witnessed the same problem. In 2016, the workshop “Workshop on Research Results Re
producibility and Resources Citation in Science and Technology of Language” at the Language Resources and Evaluation Conference (LREC) encouraged the discussion and the advancement on the reproducibility of research results and the citation of resources, and its impact on research integrity in the research area of language processing tools and resources. The workshop gathered authors interested in discussing the challenges, the risk factors, the procedures that should be adopted including the new risks raised by the replication articles themselves and their own integrity, in view of the preservation of the reputation of colleagues. 1.2
In this paper, we report the current developments
of our first participation to the CLEF eHealth
Lab
Therefore, our main objective was to build a modular system that can be easily enhanced in order to make use of the cleaned training data available and to build a reproducible set of experiments of a system that i) converts raw data containing death certificates into a cleaned dataset, ii) implements a set of semi-manual rules to split sentences and translate medical acronyms, and iii) implements a lexicon based classification approach with the aim of building a sufficiently strong baseline (our initial objective was to achieve a classifier performance close to 50%). For this purpose, we devised a pipeline for processing each death certificate and producing a ‘normalized’ version of the text that will be presented in the following sections.
A Tutorial given during the UseR! 2017
conference entitled “Data Carpentry: Open and
Reproducible Research with R”6 presented an overview
of the problems related to (the lack of)
reproducible research and the possible solutions in
particular when programming with the R Language.
In the field of Data Science, the R Markdown
framework7 is considered one of the possible
solutions to document the results of an experiment
and, at the same time, reproduce each step of the
experiment itself. Following the indications given
by
The Ce`piDc corpus was provided by the French
institute for health and medical research (INSERM)
for the task of ICD10 coding in CLEF eHealth
2017 (Task 1). It consists of free text death
certificates collected from physicians and hospitals
in France over the period of 2006-2014
the word ‘atrial’ has been written as ‘atrail’, as well as many diacritics missing in the French raw text (hemorragie instead of he´morragie). 2.2
In order to process the raw death certificate and produce a clean dataset, we implemented the following pipeline for data ingestion: read a line of a death certificate, split the line according to a list of expressions (i.e. “dans un contexte de”, suite a` un[e]”, etc.); remove extra white space (leading, trailing, internal); transform letters to lower case; remove diacritics (optional); remove punctuation; expand acronyms (if any); correct common patterns (if any).
The removal of diacritics was surprisingly
effective for the French dataset, as discussed in the
preliminary experiments
In this paper, we use a simple semi-automatic step to correct misspellings based on the dictionary of ICD10 codes that was not present in the original experiment. In particular, after cleaning the data and expanding the acronyms, we computed the generalized Levenshtein distance10 between each token of the death certificate and each token of the dictionary. At the end of this process, we found 4,142 tokens having no match (distance greater than zero) with the ICD10 vocabulary. The terms having more than 10 occurrences in the certificates were hard-coded in the source code, while all the others were automatically substituted onthe-fly.
The vocabulary has 6,295 unique entries, and there are 91,953 lines of 31,682 death certificates to classify.
10Given a strings s and t, the Levenshtein distance is the minimal possibly weighted number of insertions, deletions and substitutions needed to transform s into t (so that the transformation exactly matches t).
YearCoded 2015 2015 2015
LineID 1 2 6
RawText
PNUEMONIA
ATRAIL FIBRILLATION
CVA PARKINSONS DISEASE
(a) Example of death certificate.
(b) Example of ICD10 codes for death certificate.
The classification of each line of a death certificate
uses the approach, proposed by
X wj tj (1) which the sum of the weights wj of each term tj using binary weighting (one if term present, zero if absent). In those cases where two or more classes have the same score, the first class in the list is assigned by default. 3
For the experiments of this paper, we used the
‘raw’ dataset, that is the portion of dataset where
a file records the native text entered in the death
certificates (referred to as ‘raw causes’ thereafter).
System performance was assessed by means of a
script provided by the organizers of the Lab; the
script computes micro-Precision (the fraction of
correct instances among the retrieved instances),
micro-Recall (the fraction of relevant instances
that have been retrieved over total relevant
instances), and micro-F1 measure (the harmonic
mean between micro-Precision and micro-Recall).
As requested by the task, these measures were
computed for all causes (FR-ALL) in the datasets
and for external causes (FR-EXT), where the
evaluation is limited to ICD codes addressing a
particular type of deaths, called external causes or
violent deaths (see the Task overview for more
information
In Table3, we compare the preliminary results
of the non-official French experiments
submitted in
In the new experiment, we tried to vary the approach of splitting the sentences of a death certificate by: non-splitting the sentence (no-split), using only punctuation characters to split like commas, semi-colon, etc. (simplesplit), and using the same strategy of the original experiment (allsplit). We also tried to use the semi-automatic checkspelling (exp) that uses a mix of manual checking for the most common misspelled words (a misspell that occurs more than 10 times in the dataset) and an automatic substitution for all the remaining misspelled words (partialexp).
The experimental results showed that in all
cases we could achieve our initial goal that was a
classification performance around 0.50 for the F1
measure; moreover, our approach performed
better than the average and the median score of all
the experiments that were submitted to the CLEF
eHealth Task 1. This was a bit of a surprise
considering that our classification approach does not use
any machine learning approach, but it just cleans
the data and assigns the most frequent ICD10
code. This is an encouraging result that sets a solid
basis of cleaned data on which we can apply more
sophisticated NLP techniques, like those used by
the best systems like LIMSI (see
The aim of this work was to continue the work on the reproducible research approach that can be used as a baseline for further experiments. The performance of the system that uses a semimanual spell-checking approach improved the baseline set by the original paper. The documentation produced for the reproducibility approach helped us to spot bugs during the implementation phase and we strongly believe that this type of actions should be supported more and more because, as reported by the analysis who tested the systems at CLEF eHealth “[ . . . ] still experienced varying degrees of difficulty to install and run the systems. [ . . . ] Analysts also report that additional information on system requirements, installation procedure and practical use would be useful for all the systems submitted, although documentation was overall more abundant and detailed compared to last year’s experiments. [. . . ] The results of the experiments suggest that replication is achievable. However, it continues to be more of a challenge than one would hope.”