This paper reports on Task 1b of the 2015 CLEF eHealth evaluation lab which extended the previous information extraction tasks of ShARe/CLEF eHealth evaluation labs by considering ten types of entities including disorders, that were to be extracted from biomedical text in French. The task consisted of two phases: entity recognition (phase 1), in which participants could supply plain or normalized entities, and entity normalization (phase 2). The entities to be extracted were de ned according to Semantic Groups in the Uni ed Medical Language System R (UMLS R ), which was also used for normalizing the entities. Participant systems were evaluated against a blind reference standard of 832 titles of scienti c articles indexed in MEDLINE and 3 full text drug monographs published by the European Medicines Agency (EMEA) using Precision, Recall and F-measure. In total, seven teams participated in phase 1, and three teams in phase 2. The highest performance was obtained on the EMEA corpus, with an overall F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition and 0.872 for entity normalization.
Following healthcare laws that grant patients access to their own medical
information in the United States [
Challenges and shared tasks have had a signi cant role in advancing Natural
Language Processing (NLP) research in the clinical and biomedical domains [7,
8], especially for the extraction of named entities of clinical interest, and entity
normalization as evidenced by the previous CLEF eHealth labs [
We describe the dataset, the tasks and the evaluation metrics used for the CLEF eHealth 2015 Evaluation Lab Task 1b. 2.1
Description of the annotated data The data set is called QUAERO French Medical Corpus. It was developed as a resource for named entity recognition and normalization in 2013 [10].
The data set was created in the wake of the 2013 CLEF-ER challenge [6], with the purpose of creating a gold standard set of normalized entities for French biomedical text. A selection of the MEDLINE titles and EMEA documents used in the 2013 CLEF-ER challenge were submitted for human annotation. The annotation process was guided by concepts in the Uni ed Medical Language System (UMLS): { 10 types of clinical entities, as de ned by the following UMLS Semantic Groups [11], were annotated: Anatomy, Chemicals & Drugs, Devices, Disorders, Geographic Areas, Living Beings, Objects, Phenomena, Physiology, Procedures { The annotations were made in a comprehensive fashion, so that nested entities were marked, and entities could be mapped to more than one UMLS concept. In particular: (a) If a mention can refer to more than one Semantic Group, all the relevant Semantic Groups should be annotated. For instance, the mention recidive (recurrence) in the phrase prevention des recidives (recurrence prevention) should be annotated with the category \DISORDER" (CUI C2825055) and the category \PHENOMENON" (CUI C0034897); (b) If a mention can refer to more than one UMLS concept within the same Semantic Group, all the relevant concepts should be annotated. For instance, the mention maniaques (obsessive) in the phrase patients maniaques (obsessive patients) should be annotated with CUIs C0564408 and C0338831 (category \DISORDER"); (c) An entity whose span overlaps with that of another entity should still be annotated. For instance, in the phrase infarctus du myocarde (myocardial infarction), the mention myocarde (myocardium) should be annotated with category \ANATOMY" (CUI C0027061) and the mention infarctus du myocarde should be annotated with category \DISORDER" (CUI C0027051).
Signi cant work was done on the initial QUAERO French Medical Corpus in order to convert the annotation format from an in-line XML format to a stand-o format relying on text character o sets. In the process, annotation errors were corrected, which included systematic checking of annotation format and the introduction of discontinuous entity annotations. While discontinuous entities were part of the original QUAERO French Medical Corpus annotation guidelines, they had been poorly marked due to technical di culties linked to the in-line XML format.
Annotations on the training set are provided in the BRAT stando format [12] and can be visualized using the BRAT Rapid Annotation Tool [13]. Participants were also expected to supply annotations in this format.
The training set released in the CLEF eHealth 2015 Task 1b challenge comprised 833 MEDLINE titles and 3 full EMEA documents (divided between 11 les for readability through the BRAT interface). The test set comprised 832 MEDLINE titles and 3 full EMEA documents (divided into 12 les). Table 1 presents additional statistics describing the corpus contents. Dataset Excerpts Figure 1 shows two sample MEDLINE documents with the corresponding complete annotations for normalized entities. Figure 2 shows an excerpt of one EMEA document with the corresponding complete annotations for normalized entities.
MEDLINE title 1 MEDLINE title 2 La contraception par les dispositifs Meningites bacteriennes de l' adulte en intra uterins .1 reanimation medicale .2 MEDLINE title 1 annotations MEDLINE title 2 annotations T1 PROC 3 16 contraception T1 DISO 0 23 Meningites bacteriennes #1 AnnotatorNotes T1 C0700589 #1 AnnotatorNotes T1 C0085437 T2 DEVI 25 50 dispositifs intra uterins T2 LIVB 29 36 adulte #2 AnnotatorNotes T2 C0021900 #2 AnnotatorNotes T2 C0001765 T3 ANAT 43 50 uterins T3 PROC 40 60 reanimation medicale #3 AnnotatorNotes T3 C0042149 #3 AnnotatorNotes T3 C0085559
Document view using BRAT Named entity recognition. The task of named entity recognition consisted of analyzing plain text documents in order to mark the ten types of entities of clinical interest de ned in the lab (see Section 2.1). Participants could mark either plain entities (i.e. mark the text mentions referring to an entity of interest) or normalized entities (i.e. supply UMLS Concept Unique Identi ers corresponding to the entities in addition to marking mentions).
Entity normalization. The task of entity normalization consisted of mapping entities of clinical interest marked in biomedical text to a relevant UMLS CUI. 2.3
System performance was assessed by the usual metrics of information extraction: precision (Formula 1), recall (Formula 2) and F-measure (Formula 3; speci cally, we used =1.) for named entity recognition and entity normalization. 1 Contraception by intrauterine devices 2 Bacterial meningitis in adults in the intensive care unit. 3 What is Tysabri used for? Tysabri is used to treat adults with highly active multiple sclerosis (MS).
For normalized entity recognition, an exact match (true positive) was counted when the system's entity type, span and CUIs matched the reference.
For entity normalization, matches (true positives) were counted for each CUI supplied with an entity. As a result, if either the system or the reference supplied a list of CUIs associated with an entity, partial credit was awarded if the reference and system lists were not identical but a subset of the lists matched. 3
Participating teams included between one and six team members and resided in Belarus (team IHS-RD), China (team HIT-WI), France (teams CISMeF and LIMSI), India (team Watchdogs), the Netherlands (team Erasmus) and Spain (Team UPF).
For the plain entity recognition task, seven teams submitted a total of 10 runs for each of the corpora, EMEA and MEDLINE (20 runs in total). For the normalized entity recognition task, four teams submitted a total of 5 runs for each of the corpora (10 runs in total). For the normalization task, three teams submitted a total of 4 runs for each of the corpora (8 runs in total). 3.1
Participants used a variety of methods, some of which used machine-learning. Non-machine learning methods relied on lexical sources (medical terminologies and ontologies), translation software (statistical machine translation) or a combination of both and did not use the training corpus at all. Machine-learning methods relied on Conditional Random Fields (CRFs) for entity recognition, and used lexical resources as features. Many participants used the standard distribution of the UMLS as an onto-terminological resource. Teams from France used additional UMLS-related resources for French. Other teams relied on other sources such as Wikipedia, MANTRA and term translations provided by statistical machine translation tools. It can be noted that for the entity recognition task, none of the participants sought to address the case of discontinuous entities.
The CISMeF team participated in the plain and normalized entity recognition subtasks[15]. They trained CRF models for each entity type and each of the two corpora. They used lexical, part-of-speech and orthographical features, including 4-character pre xes and su xes for the current word and neighboring words. A lexicon of geographical names was used independently of the CRF to tag geographical entities. UMLS information was used only to identify CUIs, not to produce features.
The Erasmus team participated in all three subtasks [16]. They trained CRF models for each entity type and each of the two corpora. They used lexical, part-of-speech and orthographical features, including 4-character pre xes and su xes for the current word and neighboring words. A lexicon of geographical names was used independently of the CRF to tag geographical entities. UMLS information was used only to identify CUIs, not to produce features.
The HIT-WI team participated in all three subtasks [17]. They trained CRF models for each entity type and each of the two corpora. They used lexical, part-of-speech and orthographical features, including 4-character pre xes and su xes for the current word and neighboring words. A lexicon of geographical names was used independently of the CRF to tag geographical entities. UMLS information was used only to identify CUIs, not to produce features.
The IHS-RD team participated in all three subtasks [18]; however, they focused their e orts on the plain entity recognition subtask. They authors built 10 binary classi ers with the same sets of features: uni-grams and bi-grams and associated information such as case of the strings, presence of non-alphabetic characters, part-of-speech, syntactic function, UMLS semantic categories and occurrence in the general language. Their analysis of the contribution of the type of features shows that UMLS semantic categories have a strong impact on the results, while the contribution of syntactic features depends on the corpus.
The LIMSI team participated in the plain entity recognition subtask [19]. LIMSI's identi cation system is based on the combination of three classi ers, in order to deal with embedded entities (16% of entities in the training set): a rst CRF detects non-embedded entities, a second context-free CRF detects embedded entities, and a SVM identi es their semantic class. These classi ers rely on a set of features used in state-of-the-art classi cation systems, including token/POS ngrams, morphologic features, and dictionary consultation in language-dependent external sources.
The UPF team participated in the plain entity recognition subtask [20]. The team used an existing system, designed to annotate medical entities in English, based on a distant learning approach. Their goal was to evaluate the robustness of their method on a corps in language other than English, and the system was used \out of box", without using the training data. The method relies on several SVM classi ers (one per category) and a voting procedure (the best score) to select the result from all classi ers. Classi ers are not trained on the training corpus but on resources produced from external resources (French Wikipedia).
The Watchdog team participated in the plain entity recognition subtask [21]. Their system (Run 1) used a CRF on stemmed tokens with standard lexical features and the word position in the sentence (discretized into three bins). The originality of the approach is that UMLS features were obtained by translating words from French to English with the Bing translator before applying MetaMap on the resulting English words to obtain their semantic groups. A variant of the system (Run 2) directly used the UMLS features to detect entities for words where the CRF detected no entity, but performs less well than the initial method (Run 1). 3.2
Tables 2 and 3 present system performance on the plain entity recognition task. Tables 4 and 5 present system performance on the normalized entity recognition task. Team Erasmus had the best performance in terms of F-measure for both the EMEA and MEDLINE corpora. An analysis of the results showed that entity o sets were a technical di culty for many teams and resulted in zero or close-to-zero performance for runs exhibited formatting issues. To gain a better insight of method performance, we invited participants to submitted revised versions of their runs, where o set and formatting issues had been corrected. These submissions occurred after the lab deadline, and are shown in italic font in the tables.
Overall, in o cial runs, systems performed higher on the MEDLINE corpus (average F-measure of 0.396 for plain entities, and 0.336 for normalized entities) compared to EMEA (average F-measure of 0.279 for plain entities, and 0.311 for normalized entities). However, once format xes are taken into account, system performance is in fact higher on EMEA documents. This is explained by the fact that MEDLINE titles were short documents, comprising only one or two sentences at most. EMEA documents were much longer (several hundred sentences) and o set errors in some o cial runs often occurred beyond the rst sentence. Once the formatting issues are corrected, it appears that systems perform better on EMEA documents, which are much more redundant than MEDLINE titles. We released an improved version of the QUAERO French Medical corpus through Task 1b of the CLEFeHealth 2015 Evaluation Lab. This corpus contains entity annotations for ten entities of clinical interest, with normalization to UMLS CUIs. In the evaluation lab, we evaluated systems on the task of plain or normalized entity recognition as well as on the task of assigning CUIs to pre-identi ed entities (normalization). This is a unique biomedical NLP challenge|no previous challenge has provided such a large gold-standard annotated corpus in a language other than English. Results show that high performance can be achieved by NLP systems on the task of entity recognition and normalization for French biomedical text. However performance levels varied greatly between participating teams, indicating that the tasks are highly challenging. This corpus and the participating team system results are an important contribution to the research community and the focus on a language other than English (French) is unprecedented.
We want to thank all participating teams for their e ort in addressing a new and challenging task. We also want to thank Afzal Zubair from team Erasmus for his extensive testing of the evaluation script. The organization work for CLEF eHealth 2015 task 1B was supported by the Agence Nationale pour la Recherche (French National Research Agency) under grant number ANR-13-JCJC-SIMI2CABeRneT.
The CLEF eHealth 2015 evaluation lab has been supported in part by (in alphabetical order) PhysioNetWorks Workspaces; the CLEF Initiative; David Martinez, Guido Zuccon. Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In Pamela Forner, Henning Muller, Roberto Paredes, Paolo Rosso, and Benno Stein, editors, Information Access Evaluation. Multilinguality, Multimodality, and Visualization, volume 8138 of Lecture Notes in Computer Science, pages 212{ 231. Springer Berlin Heidelberg, 2013. 4. Liadh Kelly, Lorraine Goeuriot, Hanna Suominen, Tobias Schreck, Gondy Leroy, Danielle L. Mowery, Sumithra Velupillai, Wendy W. Chapman, David Martinez, Guido Zuccon, Joa~o Palotti Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury and Elaine Toms, editors, Information Access Evaluation. Multilinguality, Multimodality, and Interaction, volume 8685 of Lecture Notes in Computer Science, pages 172-191. Springer International Publishing, 2014. 5. Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Leif Hanlen, Aurelie Neveol, Cyril Grouin, Joao Palotti, Guido Zuccon Overview of the CLEF eHealth Evaluation Lab 2015. In Information Access Evaluation. Multilinguality, Multimodality, and Interaction. Springer International Publishing, 2015. 6. Dietrich Rebholz-Schuhmann, Simon Clematide, Fabio Rinaldi, Senay Kafkas, Erik M. van Mulligen, Chinh Bui, Johannes Hellrich, Ian Lewin, David Milward, Michael Poprat, Antonio Jimeno-Yepes, Udo Hahn, and Jan A. Kors. Entity recognition in parallel multilingual biomedical corpora : The CLEF-ER laboratory overview. In Pamela Forner, Henning Muller, Roberto Paredes, Paolo Rosso, and Benno Stein, editors, Information Access Evaluation. Multilinguality, Multimodality, and Visualization, volume 8138 of Lecture Notes in Computer Science, pages 353{ 367.
Springer Berlin Heidelberg, 2013. 7. Chapman WW, Nadkarni PM, Hirschman L, D'Avolio LW, Savova GK, Uzuner O.
Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions. J Am Med Inform Assoc. 2011 Sep-Oct;18(5):5403. 8. Huang CC, Lu Z. Community challenges in biomedical text mining over 10 years: success, failure and the future. Brief Bioinform. 2015 May 1. pii: bbv024. 9. Neveol A, Grosjean J, Darmoni SJ, Zweigenbaum P. Language Resources for French in the Biomedical Domain. Language and Resource Evaluation Conference, LREC 2014. 2014:2146-2151. 10. Neveol A, Grouin C, Leixa J, Rosset S, Zweigenbaum P. The QUAERO French Medical Corpus: A Ressource for Medical Entity Recognition and Normalization. Fourth Workshop on Building and Evaluating Ressources for Health and Biomedical Text Processing - BioTxtM2014. 2014:24-30 11. Bodenreider, O. and McCray, A. (2003). Exploring semantic groups through visual approaches. Journal of Biomedical Informatics, 36:414{432 12. BRAT Stando Annotation format. http://brat.nlplab.org/standoff.html Accessed: 2015-05-13. 13. Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou and Jun'ichi Tsujii (2012). brat: a Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of the Demonstrations Session at EACL 2012. 14. Karin Verspoor, Antonio Jimeno Yepes, Lawrence Cavedon, Tara McIntosh, Asha Herten-Crabb, Zoe Thomas, John-Paul Plazzer (2013) Annotating the Biomedical Literature for the Human Variome. Database: The Journal of Biological Databases and Curation, virtual issue for BioCuration 2013 meeting. 2013:bat019. doi:10.1093/database/bat019 15. Lina F. Soualmia, Chloe Cabot, Badisse Dahamna and Stefan J. Darmoni (2015) SIBM at CLEF e-Health Evaluation Lab 2015. CLEF 2015 Online Working Notes.
CEUR-WS. 16. Zubair Afzal, Saber A. Akhondi, Herman van Haagen, Erik Van Mulligen and Jan A. Kors (2015) Biomedical Concept Recognition in French Text Using Automatic Translation of English Terms. CLEF 2015 Online Working Notes. CEUR-WS. 17. Jingchi Jiang, Yi Guan and Chao Zhao (2015) WI-ENRE in CLEF eHealth Evaluation Lab 2015: Clinical Named Entity Recognition Based on CRF. CLEF 2015 Online Working Notes. CEUR-WS. 18. Maryna Chernyshevich and Vadim Stankevitch (2015) IHS-RD-BELARUS: Clinical Named Entities Identi cation in French Medical Texts. CLEF 2015 Online Working Notes. CEUR-WS. 19. Eva D'Hondt, Francois Morlane-Hondere, Leonardo Campillos, Dhouha Bouamor, Swen Ribeiro and Thomas Lavergne (2015) LIMSI @ CLEF eHealth 2015 - task 1b. CLEF 2015 Online Working Notes. CEUR-WS. 20. Jorge Vivaldi, Horacio Rodriguez and Viviana Cotik (2015) Semantic tagging of French medical entities using distant learning. CLEF 2015 Online Working Notes.
CEUR-WS. 21. Devanshu Jain (2015) Supervised Named Entity Recognition for Clinical Data.
CLEF 2015 Online Working Notes. CEUR-WS.