We propose a new multilingual, parallel corpus for Named Entity Linking benchmarking which comprises English, Portuguese and Spanish clinical case reports1. The medical diagnostic entities in the reports were annotated with the respective code of the International Classification of Diseases 10 - Clinical Modification (ICD10-CM) terminology and its Portuguese and Spanish versions. The result is a preliminary annotation set, which will be further validated and expanded by humans. Additionally, the ICD10-CM codes in the annotations will be mapped to the respective Medical Subject Headings (MeSH) identifiers when possible.
2.1 SciELO2 is a digital library for scientific articles, its majority written in Spanish, Portuguese and English. One of the main advantages is that many articles have versions in different languages. We extracted the abstract of clinical case reports (search filters: *AND subject area:(”Health Sciences”) AND type:(”case-report”) AND la:(”es” OR ”pt” OR ”en”)), and only considered those with the three versions simultaneously available (English, Portuguese and Spanish). We obtained 1917 abstracts in the three languages, corresponding to 639 clinical case reports, which we considered enough to test the annotation approach described below. 2.2
Annotation of medical diagnostic entities using NER and NEL
We used the python interface of MER [
The overall statistics pertaining the annotation process are available in Table 1. MER was able to recognise entity mentions in the text expressed in the three languages, but its NER performance was slightly higher in English text than in other languages as expected. Surprisingly, the NEL performance was higher in Portuguese text. As example, a sentence from a retrieved abstract is expressed in the three languages: (1 - English) “Among the identified nursing diagnosis was 2 https://scielo.org/ 3 www.cdc.gov/nchs/icd/icd10cm.htm 4 www.mscbs.gob.es/estadEstudios/estadisticas/normalizacion/home.htm 5 www.ctc.min-saude.pt/category/catalogos/
Multilingual corpus for clinical Named Entity Linking evaluation included: acute confusion, constipation and knowledge deficit.”; (2 - Portuguese) “Entre os diagn´osticos de enfermagem identificados inclu´ıram-se confusa˜o aguda, constipa¸c˜ao e conhecimento deficiente.”; (3 - Spanish) “Los resultados del estudio permitieron identificar los seguientes diagn´osticos de enfermer´ıa: confusi´on aguda, constipaci´on e conocimiento deficiente.”. MER was able to identify the italicised entity “constipation” in the English sentence (1) because there is a ICD10-CM term with the same designation: “Constipation” (code K59.0). However, the Portuguese and Spanish equivalents “constipa¸c˜ao” (sentence 2) and “constipaci´on” (sentence 3) were not recognised nor linked because the respective terms in the ICD10-CM have a different designation: “Obstipa¸c˜ao” and “Estren˜imiento” (code K59.0).
English Portuguese Spanish
Abstracts retrieved 639 639 639 Abstracts with annotations 217 197 199 Ratio of annotated abstracts 0.340 0.308 0.314
Entity mentions 533 432 465 Entity mentions per annotated abstract 2.456 2.193 2.340
Linked entity mentions 463 432 389 Linked entity mentions per annotated abstract 2.134 2.193 1.955
Ratio of linked entity mentions 0.867 1.000 0.837
The resulting corpus is available at https://github.com/lasigeBioTM/
MultiNEL-corpus. The future work consists in the human validation of the
annotation set, as well as its expansion with new annotations. This validation will
be performed either by expert analysis or by crowd-sourcing, a less expensive
approach that has shown comparable results to the expert analysis [