This paper reports on Task 1 of the 2018 CLEF eHealth evaluation lab which extended the previous information extraction tasks of ShARe/CLEF eHealth evaluation labs. The task continued with coding of death certi cates, as introduced in CLEF eHealth 2016. This largescale classi cation task consisted of extracting causes of death as coded in the International Classi cation of Diseases, tenth revision (ICD10). The languages o ered for the task this year were French, Hungarian and Italian. Participant systems were evaluated against a blind reference standard of 11,932 death certi cates in the French dataset 21,176 certi cates in the Hungarian dataset and 3,618 certi cates in the Italian dataset using Precision, Recall and F-measure. In total, fourteen teams participated: 14 teams submitted runs for the French dataset, 5 submitted runs for the Hungarian dataset and 6 for the Italian dataset. For death certi cate coding, the highest performance was 0.838 F-measure for French, 0.9627 for Hungarian and 0.9524 for Italian.
This paper describes an investigation of information extraction and
normalization (also called \entity linking") from French, Hungarian and Italian-language
health documents conducted as part of the CLEF eHealth 2018 lab [
In continuity with previous years, the methodology applied is the shared task
model[
Over the past ve years, CLEF eHealth o ered challenges addressing several
aspects of clinical information extraction (IE) including named entity
recognition, normalization [4{7] and attribute extraction [
The signi cance of this work comes from the observation that challenges and
shared tasks have had a signi cant role in advancing Natural Language
Processing (NLP) research in the clinical and biomedical domains [
One of the goals for this shared task is to foster research addressing multiple languages for the same task in order to encourage the development of multilingual and language adaption methods.
This year's lab suggests that the task of coding can be addressed reproducibly
with comparable performance in several European languages without relying on
translation. Furthermore, a global method addressing three languages at once
opened interesting perspective for multi-lingual clinical NLP [
In the CLEF eHealth 2018 Evaluation Lab Task 1, three datasets were used. The French dataset was supplied by the CepiDc1, the Hungarian dataset was supplied by KSH2 and the Italian dataset was supplied by ISTAT3. All three datasets refer to the International Classi cation of Diseases, tenth revision (ICD10),a reference classi cation of about 14,000 diseases and related concepts managed by the World Health Organization and used worldwide, to register causes of death and reasons for hospital admissions. Further details on the datasets, tasks and evaluation metrics are given below. 2.1
The CepiDc corpus was provided by the French institute for health and
medical research (INSERM) for the task of ICD10 coding in CLEF eHealth
1 Centre d'epidemiologie sur les causes medicales de deces, Unite Inserm US10, http:
//www.cepidc.inserm.fr/.
2 Kozponti Statisztikai Hivatal, https//www.ksh.hu/.
3 Istituto nazionale di statistica, http://www.istat.it/.
2018 (Task 1). It consists of free text death certi cates collected electronically
from physicians and hospitals in France over the period of 2006{2015 [
The ISTAT-IT corpus was provided by the Italian national institute of statistics (ISTAT). To better preserve con dentiality, the corpus was fabricated based on real data. Indeed, the fake certi cates were created from authentic death certi cates corresponding to di erent years of coding. The lines of a synthetic document each came from a di erent certi cate, while ensuring topical coherance and preserving the chain of causes of death (line 1 of a synthetic certi cate was created using line 1 of a real certi cate). The coherence of age, sex and causes referred were also preserved. The synthetic certi cates were then coded as if they reported a real death for 2016. To summarize, this synthetic corpus provides a realistic simulation of language and terminology found in Italian death certi cates, together with o cial coding. Up to 90 percent of the corpus contains terminology completely recognized by the Italian dictionary but it also o ers examples of language that cannot be automatically recognized by the Italian system : linguistics variants, new expressions and spelling mistakes in the text for instance. A characteristic of the Italian dictionary is the poverty of labels associated with the ICD-10 codes for external causes (including certi cates reporting surgery), which must be reviewed manually by the coding team.
Dataset excerpts. Death certi cates are standardized documents lled by
physicians to report the death of a patient. The content of the medical
information reported in a death certi cate and subsequent coding for public health
statistics follows complex rules described in a document that was supplied to
participants [
Descriptive statistics. Table 4 present statistics for the speci c data sets provided to participants. For two of the languages, the dataset construction was time-oriented in order to re ect the practical use case of coding death certi cates, line text normalized text IcCodDes 1 choc septique 2 peritonite stercorale sur perforation colique aw3 Syndrome de detresse respiratoire aigue R4 defaillance multivicerale 5 HTA 1 defaillance multivicerale R57.9 ted2 rsaytnodirroemaiegudeetresse respi- J80.0 up3 choc septique A41.9 m4 peritonite stercorale K65.9 oC5 perforation colique K63.1 6 hta I10.0 1 choc septique choc septique A41.9 2 peritonite stercorale sur perforation colique peritonite stercorale K65.9 ed2 peritonite stercorale sur perforation colique perforation colique K63.1 ilgn3 Syndrome de detresse respiratoire aigue rsaytnodirroemaiegudeetresse respi- J80.0 A4 defaillance multivicerale defaillance multiviscerale R57.9 5 HTA hta I10.0 where historical data is available to train systems that can then be applied to current data to assist with new document curation. For French, the training set covered the 2006{2014 period, and the test set from 2015. For Hungarian,
type line text 1 CACHESSIA NEOPLASTICA 2 FA AD ELEVATA RISPOSTA VENTRICOLARE aw 3 SCOMPENSO CARDIOCIRCOLATORIO, SCOMPENSO R RESPIRATORIO 4 NEOPLASIA POLMONARE 6 RESEZIONE DEL SIGMA PER NEOPLASIA , BPCO ,
IPOTIROIDISMO
data was only available for the year 2016, but the training and test sets were
nonetheless divided chronologically during that year. While the French dataset
o ers more documents spread over a nine year period, it also re ects changes in
the coding rules and practices over the period. In contrast, the Hungarian dataset
is smaller but more homogeneous. The Italian dataset was fabricated from
deidenti ed original death certi cates to further preserve patient con dentiality.
Dataset format. In compliance with the World Health Organization (WHO)
international standards, death certi cates comprise two parts: Part I is dedicated
to the reporting of diseases related to the main train of events leading directly to
death, and Part II is dedicated to the reporting of contributory conditions not
directly involved in the main death process.4 According to WHO
recommendations, the completion of both parts is free of any automatic assistance that might
in uence the certifying physician. The processing of death certi cates,
including ICD10 coding, is performed independently of physician reporting. In France,
Hungary and Italy, coding of death certi cates is performed within 18 months of
reporting using the IRIS system [
The coding task consisted of mapping lines in the death certi cates to one or more relevant codes from the International Classi cation of Diseases, tenth revision (ICD10). For the raw datasets, codes were assessed at the certi cate level. For the aligned dataset, codes were assessed at the line level. 4 As can be seen in the sample documents, the line numbering in the raw causes le may (Table 2) or may not (Table 1) be the same in the computed causes le. In some cases, the ordering in the computed causes le was changed to follow the causal chain of events leading to death. System performance was assessed by the usual metrics of information extraction: precision (Formula 1), recall (Formula 2) and F-measure (Formula 3; speci cally, we used =1.).
Precision =
true positives true positives + false positives (1) Recall =
true positives true positives + false negatives (1 + 2)
precision recall
F-measure = 2 precision + recall (3)
Results were computed using two perl scripts, one for the raw datasets (in French, Hungarian and Italian) and one for the aligned dataset (in French only). The evaluation tools were supplied to task participants along with the training data. Measures were computed for all causes in the datasets, i.e. the evaluation covered all ICD codes in the test datasets.
For the raw datasets, matches (true positives) were counted for each ICD10 full code supplied that matched the reference for the associated document.
For the aligned dataset, matches (true positives) were counted for each ICD10 full code supplied that matched the reference for the associated document line.
This year, we also experimented with a secondary metric, which consisted in computing recall over the primary causes of death. In death certi cate coding, once all the relevant causes of death have been identi ed in all certi cate lines, the chain of events leading to the dealth is analyzed to yield one single primary cause of death, which is central to national statistics reporting. This primary cause was available to us for the French and Italian datasets. Primary recall was therefore computed as the number of certi cates where the primary cause was retrieved by systems over the total number of certi cates. (2) 3
Participating teams included between one and nine team members and resided in Algeria (team techno), Canada (team TorontoCL), China (teams ECNU and WebIntelligentLab), France (teams APHP, IAM, ISPED), Germany (team WBI), Italy (Team UNIPD), Spain (teams IxaMed, SINAI and UNED), Switzerland (team SIB) and the United Kingdom (team KCL).
For the Hungarian raw dataset, we received 9 o cial runs from 5 teams. For the Italian raw dataset, we received 12 o cial runs from 7 teams. For the French raw dataset, we received 18 o cial runs from 12 teams. We also received three additional non-o cial runs from 2 teams, including one run implementing corrections for a faulty o cial run. For the French aligned dataset, we received 16 o cial runs from 8 teams. We also received three additional non-o cial runs from 2 teams, including one run implementing corrections for a faulty o cial run. 3.1
Participants relied on a diverse range approaches including classi cation methods (often leveraging neural networks), information retrieval techniques and dictionary matching accommodating for di erent levels of lexical variation. Most participants (12 teams out of 14) used the dictionaries that were supplied as part of the training data as well as other medical terminologies and ontologies (at least one team).
ECNUica. The methods implemented by the ECNUica team [
IAM-ISPED The method used by the IAM ISPED team [
IxaMed. The IxaMed group [
KCL-Health-NLP. The KCL-Health-NLP team [20] employed a document-level encoder-decoder neural approach. The convolutional encoder operates at the character level. The decoder is recurrent. For French, they contrast the usage of only Raw Text, as well as this text combined with string matched ICD codes. The string matching approach relies on the dictionaries provided, and uses a word n-gram (1-5) representation (ignoring diacritics, including stemming and removal of stopwords) to search for matches. For Italian, they take advantage of language-independent character-level characteristics and contrast results with and without pre-training using the French data. External resources are not used. LSI-UNED. The LSI-UNED team [21] submitted two runs for each raw dataset. A supervised learning system (run 2) has been implemented using multilayer perceptrons and an One-vs-Rest (OVR) strategy. The training of models was carried out with the training data and dictionaries of CepiDC, estimating the frequency of terms weighted with Bi-Normal Separation (BNS). Additionally, this approach has been supplemented with IR methods in a second system (run 1). To this end, the bias has been limited, generating learning models for the ICD-10 codes that appear more than 100 times in the training dataset. The unclassi ed diseases by these models are used to build queries and apply them to search engines with code descriptions.
SIB-BITEM. The BITEM-SIB [22] leveraged the large size and textual nature of the training data by investigating an instance-based learning approach. The 360,000 annotated sentences contained in the training data were indexed with a standard search engine. Then, the k-Nearest Neighbors of an input sentence were exploited in order to infer potential codes, thanks to majority voting. A dictionary-based approach was also used for directly mapping codes in sentences, and both approaches were linearly combined.
SINAI. The SINAI team [23] made a system based on Natural Language Processing (NLP) techniques to detect International Classi cation Diseases (ICD10) codes using di erent machine learning algorithms. First, their system found all the possibles ICD10 codes looking for how many words of each code exist in the text. Next, several measures of quality of these codes were calculated. With these metrics, di erent machine learning algorithms were trained and nally the best model was selected to use in the system. Most of the techniques used are independent of the language, therefore the system is easily adaptable to other languages.
KR-ISPED. The SITIS-ISPED team [24] used a deep learning approach and relied on the training data supplied: they used OpenNMT-py, an open source framework for Neural Machine Translation (seq2seq), implemented in PyTorch. To transform diagnostics into ICD10 codes they utilize an encoder-decoder architecture, consisting of two recurrent neural networks combined together with an attention mechanism. First, the diagnostics and their ICD10 codes are extracted from the csv les and then respectively split into a source text le and a target text le. This extraction is made by a simple bash program. In this way the data consists of parallel source (diagnosis) and target (ICD10 codes) data containing one sentence per line with words separated by a space. Then those data are split into two groups: one for training and one for validation. Validation les are used to evaluate the convergence of the training process. For source les, a rst preprocessing step converts upper cases into lower cases. A tokenization process is applied on sources les and on target les which are used as input for the neural network The used encoder/decoder model consists of a 2 layers LSTM with 500 hidden units on both the encoder and decoder. The encoder encodes the input sequence into a context vector which is used by the decoder to generate the output sequence. The training process goes on for 13 epochs and provide a model. From the test data provided by the CLEF organization, we extracted the diagnostics, preprocessed them and used the model we created to "translate" them into their respective ICD10 codes.
Techno. The techno team [25] developed Naive Bayes (NB) classi er for text
classi cation to information extraction from written text at CLEF eHealth 2018
challenge, task1. We used a NB classi er to generate a classi cation model. The
evaluation of the proposed approach does not show good performance.
TorontoCL. The TorontoCL team [26] assigned ICD-10 codes to cause-of-death
phrases in multiple languages by creating rich and relevant word embedding
models. They train 100-dimensional word embeddings on the training data provided,
as well as on language-speci c Wikipedia corpora. they then use an ensemble
model for ICD coding prediction which includes n-gram matching of the raw
text to the provided ICD dictionary followed by an ensemble of a convolutional
neural network and a recurrent neural network encoder-decoder.
WBI. The contribution of the WBI team [
WebIntelligentLab. The WebIntelligentLab team used a deep learning method viz. lstm with fully connected layers that uses only training data, no dictionary, and other external data.
Baseline. To provide a better assessment of the task di culty and system performance, this year we o ered results from a so-called frequency baseline, which consisted in assigning to a certi cate line from the test set the top 2 most frequently associated ICD10 codes in the training and development sets, using case and diacritic insensitive line matching.
Tables 6 to 9 present system performance on the ICD10 coding task for each dataset. Team IxaMed obtained the best performance in terms of F-measure for all datasets. However, we can note that the overall recall perfromance did not always align with the recall computed over primary causes of death (for French and Italian only). 4
In this section, we discuss system performance as well as dataset composition and we highlight directions for future work. 4.1
Natural Language Processing for assisting death certi cates coding System performance generally far exceeded the baseline for all three languages. The best systems achieved high precision (.846 F-measure and above) as well as high recall (.597 for French, .955 for Hungarian and .945 for Italian). Similarly to last year, we observe a gap in recall performance between the raw and aligned version of the French dataset, which suggests that there is value in performing the line alignment of the training data. We also note that the primary cause of death recall is higher on the aligned vs. raw format. Many systems o ered higher primary cause of death recall than overall recall on the aligned dataset.
Although no direct comparison is possible because the test sets were di erent, we can notice that the best performance from last year (.825 F-mesure for French raw, .867 F-mesure for French aligned by the LIMSI team [27]) remains ahead of this year's achievements.
The results of the submitting systems show consistent performance across
languages for those that addressed more than one language. Of note, all systems
but one set up a common architecture for the di erent languages, that then
independently leveraged the resources available in each language (i.e. pre-processing,
training corpus, dictionaries, external corpora used to create word embeddings...)
Only one team [
Overall, the level of performance achieved by participants this year shows great potential for assisting death certi cate coders throughout Europe in their daily task. 4.2
Size of the French test set. The French test set initially distributed this year comprised 24,375 death certi cates in the raw and aligned format. Owing to a bug in the selection process, only 11,931 certi cates were present in both raw and aligned format. In order to make the results directly comparable between formats, system performance was eventually computed on the subset of 11,931 common certi cates. Even though the nal size of the test is smaller than initially planned, we believe that the test set is still large enough to provide interesting insight on system performance for death certi cate coding in French. Comparability across languages. Overall system performance seem to be higher on the Hungarian (average F-measure .80) and Italian (average F-measure .799) datasets, compared to French (raw average F-measure .507). However, the question of strict comparability across languages remains open because of the di erences in nature between the datasets. The Italian dataset is a synthetic dataset fabricated using selected real data. It is possible that the selection process yielded somewhat content that was more standard and more easy to analyze in order to reach the consistency goals for the nal synthetic certi cates. The Hungarian dataset was obtained from transcribed paper certi cates. It is possible that some of the natural language di culties present in the original paper certi cates (such as typos) were smoothed out during the transcription process, which was performed manually by contractors. The French dataset was obtained directly from electronic certi cation, which means that it contains the original text exactly as entered by doctors without any ltering of di culties. The practice of writing death certi cates in the three di erent countries may also generate notable di erences in the writing style or depth of descriptions that impact the analysis. A further exploration of dataset characteristics in terms of number of typos, acronyms or token/type ratios could yield interesting insight on the comparability of data across languages.
We released a new set of death certi cates to evaluate systems on the task of ICD10 coding in multiple languages. This is the fourth edition of a biomedical NLP challenge that provides large gold-standard annotated corpora in a language other than English. Results show that high performance can be achieved by NLP systems on the task of coding for death certi cates in French, Hungarian and Italian. The level of performance observed shows that there is potential for integrating automated assistance in the death certi cate coding work ow in all three languages. The corpus used and the participating team system results are an important contribution to the research community. The comparable corpora could be used for studies that go beyond the scope of the challenge, including a cross-country analysis of death certi cate contents. In addition, the focus on three languages other than English (French, Hungarian and Italian) remains a rare initiative in the biomedical NLP community.
We want to thank all participating teams for their e ort in addressing new and challenging tasks. The organization work for CLEF eHealth 2018 task 1 was supported by the Agence Nationale pour la Recherche (French National Research Agency) under grant number ANR-13-JCJC-SIMI2-CABeRneT. The CLEF eHealth 2018 evaluation lab has been supported in part by the CLEF Initiative and Data61. 20. Ive J, Viani N, Chandran D, Bittar A, and Velupillai S (2018). KCL-HealthNLP@CLEF eHealth 2018 Task 1: ICD-10 Coding of French and Italian Death Certi cates with Character-Level Convolutional Neural Networks CLEF 2018 Online Working Notes. CEUR-WS 21. Almagro M, Montalvo S, Diaz de Ilarraza A, and Perez A (2018). LSI UNED at CLEF eHealth 2018: A Combination of Information Retrieval Techniques and Neural Networks for ICD-10 Coding of Death Certi cates. CLEF 2018 Online Working Notes. CEUR-WS 22. Gobeill J and Ruch P (2018). Instance-based learning for ICD10 categorization.
CLEF 2018 Online Working Notes. CEUR-WS 23. Lopez-Ubeda P, Diaz-Galiano MC, Martin-Valdivia MT, and Uren~a-Lopez LA (2018). Machine learning to detect ICD10 codes in causes of death. CLEF 2018 Online Working Notes. CEUR-WS 24. Reby K, Cossin S, Bordea G, and Diallo G (2018). SITIS-ISPED in CLEF eHealth 2018 Task 1: ICD10 coding using Deep Learning. CLEF 2018 Online Working Notes. CEUR-WS 25. Bounaama R and El Amine Abderrahim M (2018). Tlemcen University at CELF eHealth 2018 Team techno: Multilingual Information Extraction - ICD10 coding.
CLEF 2018 Online Working Notes. CEUR-WS 26. Jeblee S, Budhkar A, Milic S, Pinto J, Pou-Prom C, Vishnubhotla K, Hirst G, and Rudzicz F (2018). TorontoCL at the CLEF 2018 eHealth Challenge Task 1. CLEF 2018 Online Working Notes. CEUR-WS 27. Zweigenbaum P and Lavergne T (2017). Multiple methods for multi-class, multilabel ICD-10 coding of multi-granularity, multilingual death certi cates. CLEF 2017 Online Working Notes. CEUR-WS