This paper presents SIBM's participation in the Multilingual Information Extraction task 2 of the CLEF eHealth 2016 evaluation initiative which focuses on named entity recognition in French written text. We report on the indexing of the provided QUAERO dataset with multiple knowledge organization systems (KOS) partially or totally translated in French. The extraction method is available online in the form a webbased service that requests the KOS to extract clinical concepts from Electronic Health Records. It is also available via a user-friendly interface developed for clinicians. We addressed the identi cation of relevant clinical entities within the International Classi cation of Diseases version 10 in the CepiDC dataset with a system based on natural language processing and approximate string matching methods. The results obtained this year were rather satisfactory and attested signi cant progress, particularly in exact match recognition, since our last year's participation.
Since the amount of digital medical documents has widely expanded in the last
twenty years, the information retrieval from such heterogeneous documents has
become a signi cant challenge to address a large variety of tasks in clinical
and biomedical research as well as personalized medicine. Since 1995, the
department of BioMedical Informatics of the Rouen University Hospital (SIBM,
URL: www.cismef.org) has been working on developing tools to access health
knowledge (information retrieval and automatic indexing) in French [1{6]. More
recently, our team has worked on the evaluation of health information systems
and information retrieval and indexing in Electronic Health Records (EHRs) [
The rest of the paper is organized as follows. In Section 3 we introduce our extraction approach and tools used in QUAERO and CepiDC tasks and we describe our experimental setup. Section 4 reports on our results and on error analysis and re ections. Finally, Section 5 wraps up concluding remarks and outlines future work. 2 2.1
ECMT is developed to extract as accurately as possible from texts as input,
a list of candidate health concepts from the 55 KOS included in the Health
Terminology / Ontology Portal (HeTOP) [16]. The extraction is performed at
the phrase level of the text. A SOAP and REST Web services allow to provide a
response in XML for each concept and contains: the o set of the rst and the nal
word contained in the health concept, and which led to a medical concept in the
nal list, the identi er and its semantic type if the health concept is included
in the UMLS Metathesaurus, and the medical specialty of the concept. The
latter is based on manual semantic links between general medical specialties (e.g.
dermatology, oncology, etc.) [17] and the KOS included in HeTOP. ECMT relies
on bag-of-words and also pattern-matching designed for discharge summaries,
procedure reports or laboratory results which contain symbolic data (presence or
absence), numerical data and units of measurement. The method of bag-of-words
[
A new option named prioritization was added since 2015. It addresses the speci c issue related to the noise generated by multiple-terminology indexation. If this option is active, ECMT returns only the concept from the most reliable terminology, according to its semantic type (default value: false). When n identical terms from several terminologies are retrieved, semantic types related to these terms are computed and the most relevant is determined using set-theoretic operations. Then, the most pertinent term is retained based on a classi cation of the HeTOP resources devised manually for each semantic type available in UMLS. For example, indexing the term \asthme" (asthma) with ECMT results with 7 concepts retrieved within 7 di erent resources: SNOMED-int, NCIT, MeSH, Medline Plus, HPO, ICD-10 and ICDC. With the prioritization option activated, only one concept is retrieved according to the semantic type corresponding with \asthme" (T47-disease in this case) which is a MeSH concept. If no MeSH concept could be retrieved for a T47-disease concept, then an NCIT concept should be prioritized and retrieved, and so on. At this time, only 29 over 128 existing semantic types can be processed with this option.
Figure 2 gives an example of processing the phrase Cholestases intrahepatiques brogenes familiales et anomalies hereditaires du metabolisme hepatocytaire des acides biliaires with all ECMT default options but the activated prioritization option. ECMT extracts the MeSH terms acides et sels biliaires (CUI C0005391), cholestase intrahepatique (CUI C0008372), the ICD-10 term E70-E90 anomalies du metabolisme and the NCI term hereditaire (CUI C0439660). The user can also visualize the alternative terms and categories.
The CepiDC track aims at identifying only ICD-10 terms with several versions of this resource manually curated by CepiDC (see section 2.4). This dataset made of death certi cates revealed that many of the raw texts provided included spelling mistakes (french accents, inversions etc.). As ECMT is designed to perform only exact match using multiple terminologies, poor results have been obtained while analyzing the CepiDC corpus during the training phase. In this way, we choose to build CIMIND especially for the CepiDC track to focus on these particular issues.
CIMIND is designed to match ICD-10 concepts from the texts as input to ICD-10 terms in the relevant version of the ICD-10. The extraction is performed at the phrase level of the text using natural language processing techniques. The system is built using Python and Python/C extensions and provides a response in CSV format for each identi ed concept with: (i) the entry text, (ii) the o set of the rst and the nal word contained in the health concept, (iii) the ICD10 identi er and (iv) the ICD-10 term. CIMIND performs three main steps to identify ICD-10 concepts: Tokenization The input text is sliced into phrases, then words. Afterwards, stop words are ltered. Finally, spell checking is performed using the Enchant library. The Enchant library is a generic spell checking library with a C API providing dictionaries and corrections for a misspelled word.
Candidate selection To select ICD-10 term candidates eventually matching the input phrase, a method based on the phonetic encoding algorithm Double Metaphone (DM) [18] is used to operate a rst approximate term search. In this way, our system relies on a database storing pre-computed DM codes for each word available in each ICD-10 version dictionary. First, CIMIND computes DM codes for each word included in the analyzed phrase. Then, ICD-10 candidates with corresponding DM codes are retrieved from this database. This step provides quickly a list of relevant ICD-10 term candidates and allows us to perform time-consuming analyses on a reduced set of terms in the nal step. Candidate ranking Finally, a combination of the longest common substring and Levenshtein distance algorithms provides the candidate ranking. The most likely term having the highest score is retained as the matching ICD-10 term.
Figure 3 gives an example of processing the phrase HEMATOME INTRACEREBRAL AVEC OEDEME ET ENGAGEMENT SOUSFALCIFORME with CIMIND. CIMIND extracts the ICD-10 concepts engagement sous-falciforme (G935), hematome intracerebral (I619), and oedeme (R609).
82944;2013;1;85;2;1;HEMATOME INTRACEREBRAL AVEC OEDEME ET ENGAGEMENT SOUSFALCIFORME;NULL;NULL;;engagement sous-falciforme;G935 82944;2013;1;85;2;1;HEMATOME INTRACEREBRAL AVEC OEDEME ET ENGAGEMENT SOUSFALCIFORME;NULL;NULL;;hematome intracerebral;I619 82944;2013;1;85;2;1;HEMATOME INTRACEREBRAL AVEC OEDEME ET ENGAGEMENT SOUSFALCIFORME;NULL;NULL;;oedeme;R609
Regarding execution time, CIMIND is able to process a death certi cate as provided in the CepiDC corpus in about 80ms. 2.3
The information retrieval system of HeTOP, and thus of ECMT, operates on more than 55 terminologies in both French and English partially or totally translated into French, aligned with semantic relations. At the date of the challenge of the CLEF-eHealth 2016 task 2, thirteen KOS were migrated to In nispan, a distributed in-memory key/value data store with optional schema, and were available for ECMT: the Medical Subject Headings (MeSH), the Anatomical Therapeutic Chemical classi cation (ATC), the Classi cation Commune des Actes Medicaux (CCAM), the International Classi cation of Diseases version 10 (ICD-10), MedlinePlus, the Systematized Nomenclature of MEDicine International (SNOMED-Int), Pharmacology, the International Classi cation of Primary Care (ICPC), the Foundational Model of Anatomy Ontology (FMA), the Human Phenotype Ontology (HPO), the NCI Thesaurus (NCIT), the Online Mendelian Inheritance in Man compendium (OMIM) and the Human Rare Diseases Ontology (HRDO). Table 1 contains their metrics. Each concept of these KOS, when it is available in the UMLS, has a Concept Unique Identi er. It is the case for example for the ICD-10 and not for the CCAM. The QUAERO dataset The QUAERO French Medical Corpus dataset has been developed as a resource for named entity recognition and normalization in 2013 [19]. The data set has been created by Neveol et al. in the wake of the 2013 CLEF-ER challenge, with the purpose of creating a gold standard set of normalized entities for French biomedical text. A selection of the MEDLINE titles and EMEA documents used in the 2013 CLEF-ER challenge were selected for human annotation and are used in this challenge. Annotations are provided in the BRAT3 stando format and the annotation process was guided by concepts in the UMLS. Ten types of clinical entities which are UMLS Semantic Groups were annotated: Anatomy, Chemical and Drugs, Devices, Disorders, Geographic Areas, Living Beings, Objects, Phenomena, Physiology, Procedures. The annotations were made in a comprehensive fashion, so that nested entities were marked, and entities could be mapped to more than one UMLS concept. In particular: (i) If a mention can refer to more than one Semantic Group, all the relevant Semantic Groups should be annotated. For instance, the mention \recidive" (recurrence) in the phrase \prevention des recidives" (recurrence prevention) should be annotated with the category \DISORDER" (CUI C2825055) and the category \PHENOMENON" (CUI C0034897); (ii) If a mention can refer to more than one UMLS concept within the same Semantic Group, all the relevant concepts should be annotated. For instance, the mention \maniaques" (obsessive) in the phrase \patients maniaques" (obsessive patients) should be annotated with CUIs C0564408 and C0338831 (category \DISORDER"); (iii) Entities which span overlaps with that of another entity should still be annotated. For instance, in the phrase \infarctus du myocarde" (myocardial infarction), the mention \myocarde" (myocardium) should be annotated with category \ANATOMY" (CUI C0027061) and the mention \infarctus du myocarde" should be annotated with category \DISORDER" (CUI C0027051).
The CepiDC dataset Since 1968, the CepiDC, a French National Institute for Health and Medical Research (Inserm) laboratory, is dedicated to elaborate annually the national medical causes of death statistics in association with the French National Institute for Statistics and Economic Studies (Insee), the dissemination of the data and the studies and researches on the medical causes of death. These statistics are built from information from the certi cate of death. The CepiDC team handles a database containing more than 18,000,000 death records [20]. The CepiDC task consists of extracting ICD-10 codes from the raw lines of death certi cate text. The task is an information extraction task that relies on the text supplied to extract ICD-10 codes from the certi cates, line by line. The dataset includes 65,843 death certi cates processed by CepiDC over the period 2006-2012. The corpus is supplied in CSV format and each row contains twelve information elds associated with a raw line of text from an original death certi cate. The output comprises the 9 input elds plus two text elds used to report evidence text supporting the ICD-10 code supplied in the twelfth, nal eld. The tenth eld should contain the excerpt of the original text that supports the ICD code prediction. The dataset also includes four versions of a manually curated ICD-10 dictionary developed at CepiDC. 3 http://brat.nlplab.org/stando .html
For each track, the MEDLINE abstract titles and EMEA documents, the webbased service of ECMT is used. Before submitting our runs, we have tested ECMT with the following options actives: refined, categorizing, semantic network, prioritization and with the 7 (run2) or 13 (run1) available KOS for extracting entities and normalized entities. Run 1 uses the following resources: ATC, CCAM, ICDC, FMA, HPO, IDC-10, Medline Plus, MeSH, NCIT, OMIM, HPO, Pharma, SNOMED-Int. Run 2 uses the following resources: ATC, CCAM, ICD-10, Medline Plus, MeSH, Pharma, SNOMED-Int. For the concerns of the task and the evaluation, the ECMT output is converted into the BRAT format. Figure 4 is an example of the annotation le obtained with the following sentence: L' hyperplasie medullosurrenalienne: une etiologie rare de l' hypertension arterielle { rapport d' un cas.
T1 DISO 3 35 hyperplasie medullosurrenalienne #1 AnnotatorNotes T1 C0020507 T2 DISO 63 86 hypertension arterielle #2 AnnotatorNotes T2 C0020538 T3 ANAT 76 86 arterielle #3 AnnotatorNotes T3 C0003842
The results obtained for the challenge are presented in tables 2, 3, 4, 5 (phase 1 entities and normalized entities) and tables 6 and 7 (phase 2 normalization).
To support our discussion, the results obtained in CLEF eHealth 2015 are
presented in table 8 [
Phase 1: entities and normalized entities The results obtained for the phase 1 challenge are rather satisfactory, especially with the entity recognition with the following results: in exact match processing, we obtain a precision of 0.5381 and a recall of 0.3784 (run1) with the EMEA corpus and a precision of 0.6407 and a recall of 0.4375 (run2) with the MEDLINE corpus. In inexact match processing, we obtain a precision of 0.649 and a recall of 0.4869 (run1) with the EMEA corpus and a precision of 0.7668 and a recall of 0.5865 (run2) with the MEDLINE corpus.
For normalized entities, in exact match processing, we obtain a precision of 0.38 and a recall of 0.2687 (run1) with the EMEA corpus and a precision of 0.4776 and a recall of 0.3271 (run2) with the MEDLINE corpus. In inexact match processing, we obtain a precision of 0.4005 and a recall of 0.2842 (run1) with the EMEA corpus and a precision of 0.4974 and a recall of 0.3412 (run2) with the MEDLINE corpus.
As of last year, our results have been improved, especially in exact match entity recognition. For the MEDLINE track, we improved the precision in exact match entity recognition by 280% and the recall is improved by more than three times. Since we corrected the processing of special characters in documents and the computed o sets, we have been able to actually process the EMEA documents in exact match and improve our results in inexact match as 2015 F1 is 0.35390 and 2016 F1s are 0.5564 (run1) and 0.5233 (run2).
Indexing with multiple terminologies leads to having duplicate terms in the results that decrease the precision. This fact explains the di erences that can be observed between run1 (13 terminologies) and run2 (7 terminologies). Compared to last year, this issue has been considered and a new option has been added in ECMT. This option prioritization allows to retain only the most pertinent terms when several terminologies add up a same term in the output, and therefore reduce the noise. This ranking is operated according to the term semantic types. For each semantic type, a list of the most pertinent terminologies to be uppermost retained has been devised manually. However, as of today, only 29 semantic types over 128 are processed. The noise introduced by using multiple terminologies could then be even more reduced in the future.
Also, some errors in exact match results (compared to inexact match results) could be explained by slight di erences in terms used. The gold standard uses UMLS labels while ECMT outputs preferred labels in the original KOS. This leads to minor di erences between CLEF and ECMT outputs, such as \douleur" in CLEF output vs. \douleurs" in ECMT output. Finally, as no speci c processing was done to extract overlapping entities as described in the task, several nested entities are missed. Other entities are extracted with ECMT but are not in the gold standard. As they are more precise, these concepts should not be considered as noise.
Phase 2: normalization The results obtained for the phase 2 challenge which we participated for the rst time are also rather satisfactory. We obtain the following results: in exact match processing, we obtain a precision of 0.6044 and a recall of 0.4626 (run2) with the EMEA corpus and a precision of 0.5936 and a recall of 0.515 (run1) with the MEDLINE corpus. In inexact match processing, we obtain a precision of 0.605 and a recall of 0.463 (run2) with the EMEA corpus and a precision of 0.5938 and a recall of 0.5153 (run2) with the MEDLINE corpus.
In this phase as in the normalized entities track in phase 1, most errors in CUIs retrieved are due to di erences between our data and the gold standard's. As we used up to 13 terminologies from various sources, and HeTOP does not track versions of these resources yet, most of these errors are related to the data sources and can also be related to alignments between these sources (and their di erent versions) and the UMLS.
Track Precision Recall F1 entities, exact match 0.22840 0.13350 0.16850 entities, inexact match 0.70910 0.63660 0.67090 normalized entities, exact match 0.29530 0.18610 0.22830 normalized entities, inexact match 0.50030 0.36380 0.42130 entities, exact match 0.00400 0.00220 0.00280 entities, inexact match 0.43450 0.29860 0.35390 normalized entities, exact match 0.00440 0.00240 0.00310 normalized entities, inexact match 0.23050 0.14400 0.17730 CIMIND is used to analyze the CepiDC dataset and outputs the results in CSV format. The results obtained from this CepiDC track is presented in table 9. In this track, we obtained a precision of 0.6964 and a recall of 0.6634. The number of terms retrieved are rather decent, but comparing to results of other teams participating in this track, our error rate is not satisfactory. As the CIMIND system has been built expressly for the CLEF eHealth 2016 challenge, we lacked time to improve the nal step performed by our system by testing more edit distances and combinations of these methods and then upgrade performances. In this way, it would be quite interesting to participate again in such a task in the future.
SIBM-run1 Average Median
For the second year, the multilingual information extraction task 2 of the CLEF eHealth 2016 evaluation initiative allowed us to evaluate ECMT in a very speci c context (indexing MEDLINE titles and EMEA documents in French). ECMT is developed to index EHRs via a web-based service and also via a user-friendly interface. The actual version of ECMT (v3) is optimized to process around 70,000 EHR per day. Then, ECMT is not quite designed for the kind of datasets, abstract titles and EMEA documents, proposed in this challenge. Nevertheless, since our rst participation in 2015, we have been able to improve ECMT performances thanks to the rst evaluation which then revealed several issues related mainly to special characters and o sets computed by ECMT.
The main conclusion of this work and the obtained results is that improvements are still to be performed to reduce the noise related to multiple terminologyindexing as our di erent runs have revealed. Also, the recognition itself could still be enhanced. Moreover, this year's edition has revealed that version tracking of the resources available in HeTOP could be a major improvement for ECMT in the future. Regarding the CepiDC track, progress could have been achieved with more time and prior knowledge of the documents provided in the challenge. We plan on deepen these two approaches and to participate to other challenges in the future to keep track of our developments. 15. Neveol, A., Goeuriot, L., Kelly, L., Cohen, K., Grouin, C., Hamon, T., Lavergne, T., Rey, G., Robert, A., Tannier, X., Zweigenbaum, P.: Clinical information extraction at the CLEF eHealth Evaluation lab 2016. In: CLEF 2016 Evaluation Labs and Workshop Online Working Notes, CEUR-WS, September, 2016. 16. Grosjean, J., Merabti, T., Dahamna, B., Kergourlay, I., Thirion, B., Soualmia, L.F., Darmoni, S.J.: Health multi-terminology portal: a semantic added-value for patient safety. Stud Health Technol Inform 166(66) (2011) 129{138 17. Darmoni, S.J., Neveol, A., Renard, J.M., Gehanno, J.F., Soualmia, L.F., Dahamna, B., Thirion, B.: A medline categorization algorithm. BMC medical informatics and decision making 6(1) (2006) 7 18. Philips, L.: The double metaphone search algorithm. C/C++ users journal 18(6) (2000) 38{43 19. Neveol, A., Grouin, C., Leixa, J., Rosset, S., Zweigenbaum, P.: The Quaero french medical corpus: A resource for medical entity recognition and normalization. In: Proc BioTextM, Reykjavik, Citeseer (2014) 20. Pavillon, G., Laurent, F.: Certi cation et codi cation des causes medicales de deces. Bulletin epidemiologique hebdomadaire 30(31) (2003) 134{138