We present our participation in Task 1a of the 2013 CLEFeHEALTH Challenge, whose goal was the identi cation of disorder named entities from electronic medical records. We developed a supervised CRF model that based on a rich set of features learns to predict disorder named entities. The CRF system uses external knowledge from specialized biomedical terminologies and Wikipedia. Our system performance was evaluated at 0.598 F-measure in the context of strict evaluation and 0.711 F-measure in the context of relaxed evaluation.
Electronic medical records (EMRs) represent rich data repositories loaded with valuable patient information. Automated tools are required to process this patient information and make it available to medical professionals and specialized medical systems. These automated tools take as input the plain text of EMRs and output data of interest. For example, named-entity extraction tools process the plain text of EMRs and extract instances of named entities (i.e., noun phrases) that can be classi ed into a certain semantic category.
The 2013 CLEF-eHEALTH challenge aims to develop methods and resources that make EMRs more understandable by both patients and health professionals. The challenge spans over three tasks. We participate in the rst task, which focuses on the identi cation of disorder named entities in electronic medical records, and develop a system that can perform NER on medical text. We propose a solution that combines a rich feature set with external knowledge gathered from both specialized and general domain knowledge repositories.
We present a named-entity recognition system specialized in the medical
domain. Our system learns a CRF model from the training data, based on a rich
feature set that combines external knowledge sources with information gathered
from the EMR text. We discuss the system design, present the system results on
the 2013 CLEF-eHEALTH [
The natural language processing (NLP) community has organized specialized
competitions to evaluate and help improve the state of the art in various NLP
domains. Competitions in the general domain include the Text Retrieval
Evaluation Conferences [
Data
The corpus used for the 2013 CLEF-eHEALTH challenge consists of de-identi ed
plain text EMRs from the MIMIC II database, version 2.5 [
Annotation of disorder noun phrases (NPs) was carried out as part of the
Shared Annotated Resources project [
System design We developed a supervised linear-chain Conditional Random Fields (CRF) model using 10-fold cross validation on the training set data. We rst present the preprocessing steps we performed on the datasets. We then describe the model feature set together with the CRF feature patterns. The feature production architecture is schematized in Figure 1.
Data pre-processing Before using the challenge corpora for training and testing, we performed several pre-processing steps: { the training and test corpora provided by the challenge organizers were deidenti ed and thus contained special de-identi cation marks; to turn deidenti cation code into more normal phrases, we performed re-identi cation with pseudonyms on the input text. { EMR documents present in general a well-structured form, with a header, document body, and a footer. The header and footer contain information relevant to clinical administration, but the disorder NPs are only encountered inside the document body. We thus removed the header and footer from the EMR documents and performed analysis on the document body only. Full text
Tokenize, Re-identify
Re-identified, tokenized text
Detect document type and split
Header Body Footer
Tokenized sentences Fig. 1. Diagram of feature production.
Sections cTAKES MetaMap Exact match Wikipedia
Section labels POS-tagged, parsed,
UMLS information (1)
UMLS information (2)
UMLS information (3)
Wikipedia semantic classes
System features Given a sentence s = : : : w 2w 1w0w1w2 : : : and a token of
interest wk, we de ne features over wk and n-grams centered at wk.
1. Lexical and morphological features: we include information on the token
and on the token lemma in the form of unigrams over wk 1, wk, wk+1 and
bigrams over wk 1wk. We also include as unigram features the following
token characteristics: token contains only upper case characters, token is a
digit, is capitalized, or is a punctuation. Additionally we include as unigram
features over wk token su xes ranging from 1 to 4 characters. Finally we
add a 5-gram feature which detects patterns containing two slashes, such
as \m/r/g", which may reveal up to three disorders (such constructs are
split into 5 tokens by our tokenizer), and apply it over wk 4, wk 2, wk (the
non-slash positions of the pattern).
2. Syntactic features: we tokenize and parse the EMR plain text using the
cTakes [
All features pertaining to multi-token expressions instead of only single tokens (for instance, being a UMLS term with a given semantic group, or being an abbreviation) are encoded with the begin inside outside (B-I-O) scheme: given a label L, the rst token is labeled B-L, the next tokens are labeled I-L, and tokens not having this feature are labeled O. All features can use both unigrams and bigrams of classes : this leverages the speci c capabilities of linear-chain CRFs to label sequences instead of isolated tokens.
Problem formulation We model the problem as a supervised classi cation task with three labels: B-Disorder, I-Disorder, and O (outside). We include in the gold standard of the training set the contiguous entities and only partially took into account the non-contiguous entities. Observing that non-contiguous entities generally follow the general SNOMED v3.5 model, with a morphology / dysfunction part, which is akin to a disorder, and another (generally, anatomy) part, we only include the morphology / dysfunction part of non-contiguous entities, based on their UMLS semantic types, labelling them with B-Disorder and I-Disorder classes. This allows us to handle them as though they were contiguous entities without polluting the gold standard labels with disorder-labeled anatomy segments that would perturb training and classi cation.
We used the Wapiti [
Evaluation metrics We evaluate the system's ability to correctly identify the spans of disorder NPs. The evaluation measures are precision, recall, and F-measure, de ned as Precision =
T P
T P + F P Recall =
T P
T P + F N F-measure = 2
P recision Recall P recision + Recall (1) (2) (3) where T P = count of system NPs presenting same span as gold standard NPs; F P = count of system NPs presenting divergent span from gold standard NPs; F N = count of gold standard NPs not present in the system NPs. 1 http://wapiti.limsi.fr/
We compute the system NP span overlap to the gold standard NP under two settings: the strict evaluation setting, where the system NP span is identical to the gold standard NP span, and relaxed evaluation setting, where the system NP span overlaps the gold standard NP span. 4.2
Results Our best run evaluated at 0.730 F-measure on the training data and 0.598 Fmeasure on the test data under the strict evaluation setting; under the relaxed evaluation setting, it obtained an 0.887 F-measure on the training data and 0.711 F-measure on the test data. In general, the system precision was higher than the system recall (0.814 precision on the test data vs. 0.473 recall under strict evaluation, and 0.964 precision vs. 0.563 recall on the test data under relaxed evaluation). The system recall is lower as we did not handle the non-contiguous NPs that accounted for approx. 10% of the training and test data. The 2013 CLEF-eHEALTH Task 1a is the second challenge after the 2010 i2b2/VA Shared-Task aiming to identify disorder named entities in clinical text. Even though the nal goals of the two challenges were similar, they di ered in several structural points. First, the 2010 i2b2/VA corpus had token-based annotations and was already segmented into tokens, thus requiring no further pre-processing. In contrast, the 2013 CLEF-eHEALTH corpus marked annotations at the character level and pre-processing was desirable, which motivated the rst steps of our pipeline. Secondly, entity boundaries were de ned di erently in the two challenges. A rst example is by the determiners which were included as part of the annotation in the 2010 i2b2/VA Shared-Task but were excluded in the 2013 CLEF-eHEALTH challenge (e.g., a brain tumor vs brain tumor ). A second example is the non-contiguous entities included only in the 2013 CLEF-eHEALTH challenge (e.g., given the EMR sentence \The pain reported by the patient is occurring in lower back", the annotated entity is \pain. . . in lower back"). A nal di erence between the two challenges is the corpus size: approx. 18,550 problem entities in the i2b2 training corpus, compared to 5,874 disorder entities in CLEF-eHEALTH training corpus.
Error analysis on the training data revealed some systematic mistakes performed by our NER model. The majority of the incorrectly predicted entities were NPs with morphological structure resembling the one of the disorder NPs (e.g., anastomosis contains the Greek su x `-osis' meaning abnormal condition and resembles the name of several disorders like necrosis, osteoporosis, but can be both a disorder and a procedure), disorder names used as ndings (e.g., the noun phrase esophageal varices is a nding based on the context which showed non-bleeding grade III esophageal varices ), and disorder entities used in a negated context (e.g., NP fasciculations inside the context tongue midline without fasciculations ). The system also predicted parts of NPs it was trained on as being standalone entities; for example, the phrase left ventricular was predicted as a disorder entity after the system encountered the phrase left ventricular aneurysm during training.
In general, our NER system failed to identify the non-contiguous named entities (it could at best identify some of their parts), abbreviations or NPs rarely encountered in the training set (e.g., aaa, 3vd, inability to walk ), misspelled disorder entities (e.g., hematochezeia) and the full span of several NPs longer than 2 tokens (e.g., chronic subdural hematoma). Out-of-vocabulary tokens, i.e., terms not found in the UMLS because of lack of coverage, variants, or misspellings, were an important source of lack of recall. 5
We present a clinical NER system designed for participation in Task 1a of the 2013 CLEF-eHEALTH challenge. We design our system as a CRF with a rich feature set and external knowledge gathered from specialized terminologies and general domain knowledge repositories. Our system evaluates at 0.598 F-measure in the strict evaluation context and 0.711 F-measure in the relaxed evaluation context, obtaining a mid-range position.
Our entity detection system presents good precision but performs worse in terms of recall. In order to improve system recall, additional textual data can be integrated into the CRF model. We expect that including the Brown word clustering information as part of the feature vector would provide a fallback for some out-of-vocabulary tokens and help increase accuracy based on its unsupervised word classes. The non-contiguous entities are only partially handled by our system, thus better handling of all parts of non-contiguous entities, for instance through syntactic dependencies, should result in improved recall. 6
This work was partly funded through project Accordys2 funded by ANR under grant number ANR-12-CORD-0007-03. The rst author was funded by the 2 Accordys: Agregation de Contenus et de COnnaissances pour Raisonner a partir de cas de DYSmorphologie f tale, Content and Knowledge Aggregation for Case-based Reasoning in the eld of Fetal Dysmorphology (ANR 2012-2015).
Ch^ateaubriand Science Fellowship 2012{2013. We acknowledge the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health with grant number R01GM090187.