Introduction

This paper describes an investigation of information extraction and normalization (also called "entity linking") from French and English-language health documents conducted as part of the CLEF eHealth 2017 lab [1]. The task addressed is the automatic coding of death certificates using the International Classification of Diseases, 10th revision (ICD10) [2]. This is an essential task in epidemiology, as the determination and analysis of causes of death at a global level informs public health policies.

The methodology applied is the shared task model. In shared tasks, multiple groups agree on a "shared" task definition, a shared data set, and a shared evaluation metric. The idea is to allow evaluation of multiple approaches to a problem while minimizing avoidable differences related to the task definition, the data used, and the figure of merit applied [3,4].

Over the past four years, CLEF eHealth offered challenges addressing several aspects of clinical information extraction (IE) including named entity recognition, normalization [5][6][7] and attribute extraction [8]. Initially, the focus was on a widely studied type of corpus, namely written English clinical text [5,8]. Starting in 2015, the lab's IE challenge evolved to address lesser studied corpora, including biomedical texts in a language other than English i.e., French [6]. This year, we continue to offer a shared task based on a large set of gold standard annotated corpora in French with a coding task that required normalized entity extraction at the sentence level. We also provided an equivalent dataset in English.

The significance of this work comes from the observation that challenges and shared tasks have had a significant role in advancing Natural Language Processing (NLP) research in the clinical and biomedical domains [9,10], especially for the extraction of named entities of clinical interest and entity normalization.

One of the goals for this shared task is to foster research addressing multiple languages for the same task in order to encourage the development of multilingual and language adaption methods. This year's lab suggests that the task of coding can be addressed reproducibly with comparable performance in French and in English without relying on translation.

Material and Methods

In the CLEF eHealth 2017 Evaluation Lab Task 1, two datasets were used. The French dataset was supplied by the French CépiDc1 and the English dataset was supplied by the American CDC 2 . Both datasets refer to the International Classification of Diseases, tenth revision (ICD10),a reference classification of about 14,000 diseases and related concepts managed by the World Health Organization and used worldwide, to register causes of death and reasons for hospital admissions. Further details on the datasets, tasks and evaluation metrics are given below.

Datasets

The CépiDc corpus was provided by the French institute for health and medical research (INSERM) for the task of ICD10 coding in CLEF eHealth 2017 (Task 1). It consists of free text death certificates collected from physicians and hospitals in France over the period of 2006-2014 [11].

The CDC corpus was provided by the American Center for Disease Control (CDC). It consists of free text death certificates collected electronically in the United States during the year 2015. These are all records due to natural causes, i.e., there are no injury-related deaths included.

Dataset excerpts. Death certificates are standardized documents filled by physicians to report the death of a patient. The content of the medical information reported in a death certificate and subsequent coding for public health statistics follows complex rules described in a document that was supplied to participants [11]. Tables 1 and 2 present excerpts of the CépiDC and CDC corpora that illustrate the heterogeneity of the data that participants had to deal with. While some of the text lines were short and contained a term that could be directly linked to a single ICD10 code (e.g., "choc septique"), other lines could contain non-diacritized text (e.g., "peritonite..." missing the diacritic on the first "e"), abbreviations (e.g., "DM II" instead of "diabetes mellitus, type 2"). Other challenges included run-on narratives or mixed text alternating between upper case non-diacritized text and lower-case diacritized text. Descriptive statistics. Tables 3 and 4 present statistics for the specific sets provided to participants. For both languages, the dataset construction was timeoriented in order to reflect the practical use case of coding death certificates, where historical data is available to train systems that can then be applied to current data to assist with new document curation. For French, the training set covered the 2006-2012 period, and the development set contained death certificates from 2013 and the test set from 2014. For English, data was only available for the year 2015, but the training and test sets were nonetheless divided chronologically during that year. While the French dataset offers more documents spread over an eight year period, it also reflects changes in the coding rules and practices over the period. In contrast, the English dataset is smaller but more homogeneous. Dataset format. In compliance with the World Health Organization (WHO) international standards, death certificates comprise two parts: Part I is dedicated to the reporting of diseases related to the main train of events leading directly to death, and Part II is dedicated to the reporting of contributory conditions not directly involved in the main death process. 10 According to WHO recommenda-tions, the completion of both parts is free of any automatic assistance that might influence the certifying physician. The processing of death certificates, including ICD10 coding, is performed independently of physician reporting. In France and in the United States, coding of death certificates is performed within 18 months of reporting using the IRIS system [12]. In the course of coding practice, the data is stored in different files: a file that records the native text entered in the death certificates (referred as 'raw causes' thereafter) and a file containing the result of ICD code assignment (referred as 'computed causes' thereafter). The 'computed causes' file may contain normalized text that supports the coding decision and can be used in the creation of dictionaries for the purpose of coding assistance. We found that the formatting of the data into raw and computed causes made it difficult to directly relate the codes assigned to original death certificate texts. This makes the datasets more suitable for approaching the coding problem as a text classification task at the document level rather than a named entity recognition and normalization task. We have reported separately on the challenges presented by the separation of data into raw and computed causes, and proposed solutions to merge the French data into a single 'aligned' format, relying on the normalized text supplied with the French raw causes [13]. Table 1 presents a sample of French death certificate in 'raw' and 'aligned' format. It illustrates the challenge of alignment with the line 2 in the raw file "péritonite stercorale sur perforation colique" which has to be mapped to line 4 "peritonite stercorale" (code K65.9) and line 5 "perforation colique" (code K63.1) in the computed file.

As can be seen in Table 2 similar alignment challenges can be encountered in the English dataset. In Sample certificate 2, line 1 in the raw file "STROKE IN SEPTEMBER LEFT HEMIPARESIS" has to be mapped to line 1 (code I64, "Stroke, not specified") and line 2 (code G819, "Hemiplegia, unspecified") in the computed file. However, no normalized text was available for English and we were not able to offer an aligned version of the raw and computed files for the American dataset in this edition of the shared task. Data files. Table 5 presents a description of the files that were provided to the participants: training (train) and development (dev, French only) files were distributed early in the challenge (in January 2017) ; test files (test, with no gold standard) were distributed at test time (at the end of April 2017); and the gold standard for test files (test+g in aligned format, test, computed in raw format) were disclosed to the participants after the text phase (in May 2017) just before the submission of their workshop papers, so that participants could reproduce the performance measures announced by the organizers.

Tasks ICD10 coding

The coding task consisted of mapping lines in the death certificates to one or more relevant codes from the International Classification of Diseases, tenth revision (ICD10). For the raw datasets, codes were assessed at the certificate level. For the aligned dataset, codes were assessed at the line level. Replication. The replication task invited lab participants to submit a system used to generate one or more of their submitted runs, along with instructions to install and use the system. Then, two of the organizers independently worked with the submitted material to replicate the results submitted by the teams as their official runs.

Evaluation metrics

System performance was assessed by the usual metrics of information extraction: precision (Formula 1), recall (Formula 2) and F-measure (Formula 3; specifically, we used β=1.).

Precision = true positives true positives + false positives (1)

Recall = true positives true positives + false negatives

(2)

F-measure = (1 + β 2 ) × precision × recall β 2 × precision + recall(3)

Results were computed using two perl scripts, one for the raw datasets (in English and in French) and one for the aligned dataset (in French only). The evaluation tools were supplied to task participants along with the training data.

Measures were computed for "ALL" causes in the datasets as our main evaluation reference for the task. In this case the evaluation is performed for all ICD codes. Measures were also computed for "EXTERNAL" causes as our secondary reference for the task. In this case, the evaluation is limited to ICD codes addressing a particular type of deaths, called "external causes" or violent deaths. These causes are of particular interest for two reasons: first, they are considered as "avoidable" and public health policies can target them specifically, e.g., suicide prevention. Second, the context associated with these deaths is often quite different from other deaths in terms of comorbidity, population affected and terminology used to describe the event. In practice, external causes are characterized by codes V01 to Y98.

For the raw datasets, matches (true positives) were counted for each ICD10 full code supplied that matched the reference for the associated document.

For the aligned dataset, matches (true positives) were counted for each ICD10 full code supplied that matched the reference for the associated document line.

The evaluation of the submissions to the replication task was essentially qualitative: we used a scoring grid to record the ease of installing and running the systems, the time spent to obtain results with the systems (analysts were committed to spend at most one working day-or 8 hours-to work with each system), and whether we managed to obtain the exact same results submitted as official runs.

Results and Discussion

Participating teams included between one and twelve team members and resided in Australia (team UNSW), France (teams LIMSI, LIRMM, LITL, Mondeca and SIBM), Germany (teams TUC and WBI), Italy (Team UNIPD) and Russia (team KFU). Teams often comprised members with a variety of backgrounds and drew from computer science, informatics, statistics, information and library science, clinical practice. It can be noted that one team (LITL) participated in the challenge as a master-level class project. One team (LIMSI) was composed of members of the organization team and submitted unofficial runs due to conflict of interest. One team submitted baseline runs.

For the English raw dataset, we received 15 official runs from 9 teams, including one baseline run and one invalid run (due to formatting issues). For the French raw dataset, we received 7 official runs from 4 teams. For the French aligned dataset, we received 9 official runs from 6 teams, including one baseline run.

Five systems were submitted to the replication track, allowing us to attempt replicating a total of 22 system runs.

Methods implemented in the participants' systems

Participants used a variety of methods, many of which relied on lexical sources including the dictionaries supplied as part of the training data as well as other medical terminologies and ontologies. Some of these knowledge-based methods exploited the gold standard training data as an additional knowledge source.

IMS-UNIPD.

The UNIPD team submitted official runs for the English dataset and later submitted unofficial runs for the French datasets as well [14]. This team implemented a minimal expert system based on rules to translate acronyms together with a binary weighting approach (run 1) and a tf-idf approach (run 2) to retrieve the items in the dictionary most similar to the portion of the certificate of death. For both configurations, a basic approach was used to select the class with the highest weight.

KFU. The KFU team submitted two runs for the English dataset [15]. They used sequence to sequence deep learning models based on recurrent neural networks. As input sequence, the method takes the raw text and outputs sequence of ICD10 codes. Both the supplied corpus and dictionary were used for training, exclusive of any additional data.

LITL. The LITL team submitted runs for the French dataset in the raw and aligned formats [16]. The LITL team system was specifically designed by master's students (LITL programme, university of Toulouse) and their teachers for the challenge. The system is based on the search platform SOLR. Training data was indexed using the SolrXML format. The core is organized into ICD codes associated with the corresponding "raw Texts", "diagnostic Texts", ICD headings and SNOMED labels. The raw Texts from the test dataset were automatically transformed into queries and submitted to SOLR. The two runs submitted are based on the same collection and SOLR configuration. For Run 1, raw texts were automatically split into several queries when different causes were detected by using a custom-made rule-based system. For Run 2, each query corresponds to the entire raw text of each CépiDC line.

LIMSI. The LIMSI team submitted unofficial runs for all datasets [17]. The starting point for these submissions is their last published system [18], which relied upon dictionary projection and supervised multi-class, mono-label text classification using simple features (bag of normalized tokens, character trigrams, and coding year). They extended this system to multi-label classification and the use of dictionary and token bigram features in the classifier. Character n-grams did not improve the F1-score on the training set and were discarded. Coding year was kept for the French data, but not for the English data, because it only spans year 2015. Because it only relies on the material provided by the task organizers, the same system could be applied to both the French and English datasets. In each case, Run 1 used a supervised machine learning method (multi-label SVM, with unigrams, bigrams and [for French] coding year), and Run 2 used a hybrid method: union of calibrated dictionary and multi-label SVM.

LIRMM. The LIRMM team submitted runs for all datasets [19]. They annotated death certificate text through the SIFR Bioportal Annotator (http: //bioportal.lirmm.fr/annotator) using different configurations of the web service. For French, Simple Knowledge Organization System (SKOS) was built using ICD10 content from the CISMeF portal, the set of dictionaries provided in the challenge, as well as the training corpus. For the first run, the ontology was generated with a heuristic, where labels that correspond to multiple codes are assigned to the most frequent code only. For the second run, a fall back strategy relaxes the most frequent code heuristic for lines that were not assigned any codes initially. For English, in the first run, the SKOS was built using the American dictionary supplied with training data. In the second run the dictionary was combined with an owl version of ICD10 and ICD10CM (extracted from the Unified Medical Language System).

Mondeca. The Mondeca team submitted unofficial runs11 for all datasets [20]. They approached multilingual extraction of IC10 codes by combining semantic web technology and NLP concepts in four steps: (i) transform all the datasets into RDF for a graph-based manipulation; (ii) transform the dictionaries for all the years into SKOS for better enrichment across the knowledge-bases; (iii) design a GATE workflow to annotate the RDF datasets based on gazetteers extracted from the dictionaries; and (iv) work on both French (raw data) and English corpus within a unique workflow, in a multilingual approach thus enabling simultaneous processing of multiple languages.

SIBM. The SIBM team submitted runs for all datasets [21]. Their approach of term extraction is performed at the phrase level using natural language processing. The system is built using Python and Python/C extensions and produces the following output for each identified concept: (i) the entry text, (ii) the offset of the first and the final word contained in the health concept, (iii) the ICD10 identifier and (iv) the ICD10 term. Three main steps lead to the identification of ICD10 concepts for a given text: During tokenization, the input text is sliced into phrases, then words. Stop words are filtered and spell checking is performed using the Enchant library. Next, during ICD10 candidate selection, a method based on the phonetic encoding algorithm Double Metaphone (DM) is used for approximate term search. This system relies on a database storing pre-computed DM codes for each word available in the ICD10 dictionaries. Finally, during candidate ranking, a combination of the longest common substring and fuzzy match algorithms provides the candidate ranking. The most likely term having the highest score is retained as the matching ICD10 code for the phrase.

TUC. The TUC team submitted runs for all datasets [22]. Their approach is focused on the exploration of relevant feature groups for multilingual text classification regarding ICD10 codes. First, a large scale brute-force feature set is constructed using the groups bag of words, bag of bigrams, bag of trigrams, latent Dirichlet allocation, and the ontologies of WordNet and UMLS. In the development phase, three different strategies were evaluated in conjunction with support vector machines for the English and French corpus: each feature group separately, early fusion of all feature groups, and late fusion. For English, early fusion (run 1) and the feature group bag of bigrams (run 2) achieved the best results. For French, average late fusion concerning bag of words and bag of bigrams (run 1), and the feature group bag of bigrams (run 2) performed best.

UNSW. The UNSW team submitted runs for the American dataset [23]. They deployed a knowledge-based approach to tackle the task by solely using dictionary lookup. The first step is to index manually coded ICD10 lexicon followed by dictionary matching. Priority rules are applied to retrieve the relevant entity/entities and their corresponding ICD10 code(s) given free text cause of death description. Two priority methods were implemented in the submitted runs: the first one relied on BM25 and the second one on direct term match. The advantages of a knowledge-based method include speed and no need for training data.

WBI. The WBI team submitted runs for the English raw dataset and for the French aligned dataset [24]. They combined standard rule-based methods for Named Entity Recognition (NER) with machine-learning approaches for candidate ranking. For NER rule-based dictionary lookup and fuzzy matching using Lucene Sorl was applied. Preference was on generating potential candidates for each match to increase recall. Candidates were then ranked using a machinelearning approach. Based on the hierarchy of the ICD10 terminology (chapters, blocks, sub-chapters) combined with ICD10-Codes and Text available from the provided dictionaries a classifier was developed for ranking candidates.

Baselines. To provide a better assessment of the task difficulty and system performance, this year we offer baseline results using two methods: 1/ the ICD baseline consisted of exact string matching between the terms in the ICD and the death certificate text. 2/ the frequency baseline consisted in assigning to a certificate line from the test set the top 2 most frequently associated ICD10 codes in the training and development sets, using case and diacritic insensitive line matching.

System performance on death certificate coding

Tables 6 to 8 present system performance on the ICD10 coding task for each dataset. Team KFU obtained the best performance in terms of F-measure both overall and for the external causes of death on the English dataset. Team SIBM obtained the best official performance in terms of F-measure both overall and for the external causes of death on the French datasets. It is interesting to note that the participants who obtained the best scores on the French datasets (SIBM and LIMSI) are returning teams who also participated in the coding task in 2016. Team SIBM's performance improved from an F-measure of .680 in 2016 to an F-measure of .804 this year while team LIMSI's performance improved from an Table 6. System performance for ICD10 coding on the English raw test corpus in terms of Precision (P), recall (R) and F-measure (F). The top part of the table displays official runs, while the bottom part displays non-official and baseline runs. F-measure of .652 in 2016 to an F-measure of .867 this year, which also exceeds the best performance of 2016 obtained by team Erasmus with F-measure of .848. 12 This suggests that there is room for improvement on this task, and that iterations of the task are useful to help identify the best ideas and methods to address the task.

ALL EXTERNAL Team P R F Team P R F

To provide a more in-depth analysis of results, this year we also introduced a measure of system performance on the external causes of death, which are of specific interest to public-health specialists, and are also thought to be more difficult to code. This hypothesis was confirmed by the results, as system performance was much lower on the external causes vs. all causes for all systems, both for the English and French datasets. Interestingly, some systems offered very good performance overall, but comparatively quite low performance on external causes, and vice-versa. We also note that the performance of the frequency baseline was much higher on the French aligned dataset, compared to the French raw dataset and English dataset. This suggests that there is value to the alignment Table 7. System performance for ICD10 coding on the French raw test corpus in terms of Precision (P), recall (R) and F-measure (F). A horizontal dash line places the frequency baseline performance. The top part of the table displays official runs, while the bottom part displays non-official and baseline runs. step of data preparation, and to the size of the dataset (the French dataset was significantly larger than the English dataset).

ALL

The results show that both knowledge-based and statistical methods can perform well on the task. For English the best performance is obtained from a statistical neural method (team KFU) and the second best is obtained by a machine learning method relying on knowledge based-sources (team LIMSI). For French, the best performance is obtained from a machine learning method relying on knowledge based-sources (team LIMSI), while the second best is obtained with a combination of knowledge based and Natural Language processing methods (Team SIBM). In addition, many teams relied on a system architecture that was the same for both languages and utilized language specific features or knowledge sources, requiring little language adaptation. The results are very encouraging from a practical perspective and indicate that a coding assistance system could prove very useful for the effective processing of death certificates in multiple languages.

Replication track and replicability of the results

Five teams submitted systems to our replication track. Only one of these teams had also participated in the replication track last year. Four systems covered both French and English, and one system only processed English. In addition, the replication track also used the simple scripts used to produce baseline runs.

Most of the baseline and system runs could be replicated by at least one analyst. However, the analysts still experienced varying degrees of difficulty to install and run the systems. Differences were mainly due to the technical set-up of the computers used to replicate the experiments. Analysts also report that additional information on system requirements, installation procedure and practical use would be useful for all the systems submitted, although documentation was overall more abundant and detailed compared to last year's experiments. In some cases, system authors were contacted for help. They were responsive and contributed to facilitate the use of their system. The results of the experiments suggest that replication is achievable. However, it continues to be more of a challenge than one would hope.

Limitations

Formatting issues. In the French dataset, a formatting issue affected the certificates whose narratives contained a semicolon. The data export from IRIS to csv failed to adequately protect the text field with quotes, so that some of the data instances were made difficult to parse. Nonetheless, this problem affected less than 1% of the lines so we believe it had limited impact on the results. The export format will be corrected in future releases of the dataset. However, we would like to note that this type of issue fits within the practical 'real life' element of this challenge. While it certainly may have made system development more difficult, it also advocated for systems with strategies for dealing with potentially less-than-perfect data. While unintended, we believe this situation in fact makes for a robust evaluation because this kind of data would also be present in a practical workflow.

Did smoking contribute to the death? In the American dataset, the assignment of code F179 "Mental and behavioral disorders due to use of tobacco, unspecified" may be supported by information supplied by the reporting physician either in certificate narrative or in a structured data form. As a result, the gold standard assignment of F179 is sometimes unsupported by text. The prevalence of F179 due to form filling vs. text report is unknown and the two cases are currently indistinguishable in the dataset. The sample document shown in Table 2 illustrates the case of F179 assignment supported by data form and not by text. The prevalence of the code is 4.7% in the training set and 3.9% in the test set, which creates a bias for all evaluated systems. We estimate that the bias could create differences of up to 2% in the overall F-measure. However, we note that the external causes evaluation is not impacted because F179 does not belong to the external cause of death category.

Conclusion

We released a new set of death certificates to evaluate systems on the task of ICD10 coding in multiple languages. This is the third edition of a biomedical NLP challenge that provides large gold-standard annotated corpora in French. Results show that high performance can be achieved by NLP systems on the task of coding for death certificates in French and in English. The level of performance observed shows that there is potential for integrating automated assistance in the death certificate coding workflow in both languages. We hope that continued efforts towards reproducibility will support the shift from research prototypes to operational production systems. The corpus used and the participating team system results are an important contribution to the research community. In addition, the focus on a language other than English (French) remains a rare initiative in the biomedical NLP community.