This report outlines the Task 1 of the ShARe/CLEF eHealth evaluation lab pilot. This task focused on identi cation (1a) and normalization (1b) of diseases and disorders in clinical reports. It used annotations from the ShARe corpus. A total of 22 teams competed in Task 1a and 17 of them also participated Task 1b. The best systems had an F1 score of 0.75 (0.80 Precision, 0.71 Recall) in Task 1a and an accuracy of 0.59 in Task 1b. The organizers have made the text corpora, annotations, and evaluation tools available for future research and development.
A large amount of very useful information { both for the medical researchers and
the patients { is present in the form of unstructured text within the clinical notes
and discharge summaries that form a patient's medical history. Adapting and
extending NLP techniques to mine this information can open doors to better,
novel, clinical studies on one hand, and help patients understand the contents of
their clinical records on the other. Organization of this shared task helps establish
state-of-the-art baselines and paves way to further explorations. The shared task
was one of three shared tasks organized at the CLEF eHealth Evaluation Labs [
The ShARe corpus7 comprises of annotations over de-identi ed clinical reports from from US intensive care (version 2.5 of the MIMIC II database8.) The corpus consisted of discharge summaries and electrocardiogram, echocardiogram, and radiology reports. Although the clinical reports were de-identi ed, they still needed to be treated with appropriate care and respect. Hence, all participants were required to register to the lab, obtain a US human subjects training certi cate9, create an account to a password-protected site on the Internet, specify the purpose of data usage, accept the data use agreement, and get their account approved. 3
The annotation of disorder mentions in clinical reports was carried out as part of
the ongoing ShARe project10. For this task in the evaluation lab, the focus was
on the annotation of disorder mentions only. As such, there were two parts to
the annotation: identifying a span of text as a disorder mention and mapping the
span to a UMLS [
The following evaluation criteria were used:
7 https://www.clinicalnlpannotation.org
8 Multiparameter Intelligent Monitoring in Intensive Care http://mimic.physionet.org
9 The course was available free of charge on the Internet, for
example, via the CITI Collaborative Institutional Training Initiative at
https://www.citiprogram.org/Default.asp or the US National Institutes of Health
(NIH) at http://phrp.nihtraining.com/users/login.php.
10 https://www.clinicalnlpannotation.org
11 https://uts.nlm.nih.gov/home.html
12 Note that this de nition of Disorder semantic group did not include the Findings
semantic type, and as such di ered from the one of UMLS Semantic Groups, available
at http://semanticnetwork.nlm.nih.gov/SemGroups
1a. correctness in identi cation of the character spans of disorders,
1b. correctness in mapping disorders to SNOMED-CT codes,
In Tasks 1a and 1b each participating team was permitted to upload the outputs
of up to two systems. Task 1b was optional for Task 1 participants. Teams
were allowed to use additional annotations in their systems, but this counted
towards the permitted systems; systems that used annotations outside of those
provided were evaluated separately. The evaluation for all tasks was conducted
using the blind, withheld test data. The participants were provided a training
set containing clinical text as well as pre-annotated spans and named entities
for disorders (Tasks 1a and 1b). For Task 1a, participants were instructed to
develop a system that predicts the spans for disorder named entities. For Tasks
1b, participants were instructed to develop a system that predicts the
SNOMEDCT code. The outputs needed to follow the annotation format. The corpus of
reports was split into 200 training and 100 testing. The system performance was
evaluated agaist the criteria by using the F1 score in Task 1a and Accuracy
in Tasks 1b. We relied on non-parametric statistical signi cance tests called
random shu ing [
A total of 22 teams competed in Task 1a and 16 of them also participated Task 1b. The performance of these systems is detailed in Tables 1 and 2. The best systems had an F1 score of 0.75 (0.80 Precision, 0.71 Recall) in Task 1a and an accuracy of 0.59 in Task 1b. 6
We have created a reference standard with high inter-annotator agreement and evaluated systems on the task of identi cation and normalization of diseases and
System ID (fteamg.fsystemg) disorders apprearing in clinical reports. The results have demonstrated that an NLP system can complete this task with reasonably high accuracy. We plan to annotate more data and perform another evaluation in the near future.
We greatly appreciate the hard work and feedback of our program committee members and annotators, including, but not limited to David Harris, Glenn Zaramba, Erika Siirala, Qing Zeng, Tyler Forbush, Jianwei Leng, Maricel Angel, Erikka Siirala, Helja Lundgren-Laine, Jenni Lahdenmaa, Laura Maria Murtola, Marita Ritmala-Castren, Riitta Danielsson-Ojala, Saija Heikkinen, and Sini Koivula. This shared task was partially supported by Shared Annotated Resources (ShARe) project NIH 5R01GM090187, VAHSR&D HIR 08-374, NICTA (National Information and Communications Technology Australia), funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program), and NLM 5T15LM007059.