We present our participation in Task 2 of the 2013 CLEFeHEALTH Challenge, whose goal was to determine the UMLS concept unique identi er (CUI), if available, of an abbreviation or acronym. We hypothesize that considering only the abbreviations of the training corpus could be su cient to provide a strong baseline for this task. We therefore test how a fully supervised approach, which predicts the CUI of an abbreviation based only on the abbreviations and CUIs seen in the training corpus, can fare on this task. We adapt to this task the processing pipeline we developed for CLEF-eHEALTH Task 1, entity detection: a supervised MaxEnt model based on a set of features including UMLS Concept Unique Identi ers, complemented here with a rule-based component for document headers. This system con rmed our hypothesis, and was evaluated at 0.664 accuracy (strict) and 0.672 accuracy (relaxed), ranking second out of ve teams.
The 2013 CLEF-eHEALTH challenge [
Clinical notes often use abbreviations and acronyms (henceforth collectively named `abbreviations' in this paper). While the number and variety of abbreviations is unlimited thanks to the plasticity of language and the inventiveness of the human mind, the distribution of abbreviations can be expected to follow a Zip an law, with a small number of distinct abbreviations accounting for a large part of the total number of occurrences of abbreviations. We therefore hypothesize that the sample of abbreviations present in the training corpus provided by CLEF-eHEALTH for the task should contain most frequently occurring abbreviations and should hence account for a large part of the occurrences of abbreviations in the test corpus.
We present a supervised abbreviation resolution system speci cally adapted
to the abbreviations of the CLEF eHealth corpus. Our system learns a MaxEnt
model from the training data, based on a simple feature set that combines UMLS
knowledge with information gathered from the EMR text. We present the system
design, its results on the 2013 CLEF-eHEALTH [
Abbreviations and acronyms are pervasive in technical and scienti c text,
including in health and life sciences. They create obstacles to various information
extraction tasks, for instance for coreference resolution in electronic medical
records [
When abbreviation expansions are not explicit in source texts, abbreviation
processing can be divided into two steps. The rst is the detection of
abbreviations, which determines which expressions in a text are abbreviations (or
acronyms) [
Task 2 of the ShARe CLEF eHealth 2013 Challenge addresses the abbreviation disambiguation step. Gold standard annotations of abbreviation boundaries are given for the abbreviations present in clinical notes, which removes the need to address the abbreviation detection step: the task consists of disambiguating and normalizing each abbreviation into a UMLS Concept Unique Identi er 1 Example from the gold-standards.html.
MEDSTRACT corpus:
http://medstract.org/
(CUI), or the CUI-less label if no UMLS CUI is available. It is close to the task
de ned in [
The corpus used for the 2013 CLEF-eHEALTH challenge consists of de-identi ed
plain text EMRs from the MIMIC II database, version 2.5 [
Abbreviations and acronyms are annotated in terms of span and UMLS Concept Unique Identi er (CUI) when available. If no CUI is available for the abbreviation the CUI-less code is used. The training set contained 3; 805 annotations, while the test set contained 3; 774 annotations (see Table 1). CUI-less abbreviations accounted for about 5% in the training set and 6% in the test set. { Rule-based resolution of abbreviations in document headers (Section 3.4); { Supervised resolution of abbreviations in document body (Section 3.3).
We used a supervised linear-chain Conditional Random Fields (CRF) system which we restricted to Maximum Entropy mode (see below). We trained a model using 10-fold cross validation on the training set data, keeping 1/11th of the data for nal tuning of the model. We rst present how we formulate the problem. We then describe the pre-processing steps we performed on the datasets and the model feature set together with the CRF feature patterns.
Problem formulation Abbreviation resolution is de ned here as the determination of the CUI of an abbreviation, or else the attribution of the CUI-less label. An abbreviation may be ambiguous, i.e., be found with di erent CUIs in di erent contexts. For instance, in the training set, CATH is used for Catheterization [C0007430] and for Drainage Catheters [C0879005]. When ignoring case, 526 distinct abbreviations are found in the training set, 94 of which are ambiguous given the entries in the training set.
When all candidate CUIs for an abbreviation are known, abbreviation expansion can be addressed as a supervised classi cation task|this is what we do here. We label each abbreviation of the training set with its gold standard CUI, using a B-I-O scheme where each CUI introduces a B- and possibly some I- labels. For instance, the two tokens of abbreviation \O2 sat" are labelled as B-C0523807 and I-C0523807. This results in 738 distinct labels. An obvious limitation of this design is that only those CUIs seen in the training set can be assigned in the test set, but we found it worth trying.
We used the Wapiti [
Tokenize, Re-identify
Re-identified, tokenized text
Detect document type and split
Header Body Footer
Rules
Abbreviation
CUIs Tokenized sentences
Sections cTAKES Exact match
Gold standard
Section labels (POS-tagged, parsed),
UMLS information (1)
UMLS information (2) Abbreviation boundaries
System features Given a sentence s = : : : w 2w 1w0w1w2 : : : and a token of interest wk, we de ne features over wk and n-grams centered at wk. 1. Lexical and morphological features: we include information on the token in the form of unigrams over wk 1, wk, wk+1 and bigrams over wk 1wk. Additionally we include as unigram features over wk token pre xes ranging from 1 to 4 characters. We also add a 5-gram feature which detects patterns containing two slashes, such as \m/r/g", which may help disambiguate between disorders and abbreviations, and apply it over wk 4, wk 2, wk (the non-slash positions of the pattern). 2. Document structure features: the feature set contains as features the document type (e.g., radiology report, discharge summary) and the section type (e.g., Findings, Laboratory data, Social History). We extract the section type using a rule-based section extraction tool that identi es the occurrence of section names within the EMR. The section extraction tool uses a list of 58 manually de ned section names. Both document type and section type are unigram features over wk. 3. UMLS features: we include UMLS information from two sources. We rst run cTAKES over the texts and keep concept unique identi ers (CUIs, dened over unigrams wk). We use an additional UMLS mapping where we directly search for UMLS strings within the EMR text through exact match and include the concept unique identi er (CUI) of the identi ed phrase. The direct UMLS mapping features are unigram features over wk. 4. Abbreviation feature: gold standard boundaries for abbreviations are provided. These are used as unigram features over wk.
All features pertaining to multi-token expressions instead of only single tokens (for instance, being a UMLS term with a given CUI, or being an abbreviation) are encoded with the Begin Inside Outside (B-I-O) scheme: given a label L, the rst token is labeled B-L, the next tokens are labeled I-L, and tokens not having this feature are labeled O. An advantage of linear-chain CRFs is that they can include label bigrams in the model they learn. However, in our experiments on the training set, we were not able to use label bigrams because of the large number of labels (738 in the model we nally used): with label bigrams, the number of generated features for a CRF pattern which involves x observed values and y labels is x y2, and it seems that the feature space generated when we tested with these label bigrams was to vast for the CRF to nd a solution when training. Having no label bigrams means that our CRF is used basically as a Maximum Entropy classi er, with no dependency on the previous label. The total number of features generated for evaluation by the CRF for this model is about 50 million, among which it selects about 400,000. 3.4
We examined abbreviations in the training set headers and wrote rules to label them. These rules are shown in Table 2. They handle the 6 abbreviations found in the document headers of the training set. The resulting annotations are merged with those obtained by the CRF.
We evaluate the system's ability to correctly generate the codes (CUIs or CUIless) of abbreviations. The evaluation measure is the accuracy of codes, de ned as
Accuracy =
Correct Total (1) where Correct = Number of pre-annotated acronyms/abbreviations with correctly generated code; Total = Number of pre-annotated acronyms/abbreviations.
In some cases the human annotators generated a ranked list of codes for a given abbreviation. This makes it possible to de ne two variants of the accuracy score. In the strict variant, only the top code selected by the annotators (one best) is considered. In the relaxed variant, a code is considered correct if it is contained in the full list generated by the annotators (n-best).
Rule-based resolution of abbreviations in headers obtained the results displayed on Table 3. 109 abbreviations were located in the headers; the rules identi ed 100, out of which 97 matched the gold standard. The two substitutions (the gold standard CUI was di erent from that proposed by the system) are both cases where human annotators marked as CUI-less an M (Male) or F (Female) abbreviation in the header, which seems to be an error in the gold standard.3 Since we did not constrain rules to apply to gold standard abbreviation boundaries, the system also identi ed an M abbreviation which the human annotators forgot to mark in a document header (insertion).4 If this is correct, rule-based resolution of abbreviations had perfect accuracy.
However, the headers were processed only through these rules, and other abbreviations in the headers were thus not examined. 10 such abbreviations were missed (deletions), so that overall, our resolution of abbreviations in headers achieved an accuracy of 0.89 against the gold standard. All in all, the hundred of abbreviations present in headers are only a small part of the 3,774 abbreviations of the test corpus (about 3%), so this part of the method only contributed a small fraction of the performance of the system. 4.3
This section presents the global results that we obtained on the full documents, merging codes obtained through rule-based and supervised methods. We submitted one run which achieved an accuracy of 0.664 under the strict evaluation and 0.672 under the relaxed setting.
The training set contained 696 distinct CUIs (plus the CUI-less label), whereas the test set contained 706 distinct CUIs (plus CUI-less ) (see Table 1). When training our nal CRF model we kept 1/11th of the corpus for development, resulting in a model which has knowledge for 654 CUIs (plus CUI-less ). Only 346 of these were present in the test set, which means that our system could only identify about one half of the 706 distinct CUIs of the test set.
However, considering the number of occurrences of these CUIs draws a quite di erent picture. Table 4 shows that among the 3774 occurrences of abbreviations contained in the test set, 2906 had CUIs seen in the training corpus (including 221 occurrences of CUI-less, see Table 5). Correctly identifying these known CUIs would have led to an accuracy of 2906=3774 = 0:770, which would outperform the best system of the challenge (0.719). Our hypothesis according to which considering only the abbreviations of the training corpus could be su cient to provide a strong baseline is therefore con rmed.
Besides, our system correctly identi ed the CUIs of 2354 occurrences of abbreviations, i.e., 81% of the best it could have done given its inherent limitations. In the case of unseen abbreviations, our system can only be right if the abbreviation is CUI-less and the system does assign this code to it; this occurred for 151 occurrences. Whether or not an abbreviation had a CUI did not a ect much the performance of the system, as can be seen on Table 5.
We conclude this discussion with a word on insertions and deletions. These
were not expected given the task de nition: given gold standard abbreviation
boundaries, determine the CUI of the abbreviation. Using a CRF with a B-I-O
scheme allowed us to keep the very same framework as we prepared for Task 1
of the challenge, Entity Recognition [
For instance, the gold standard considered the string NCAT as two tokens NC [CUI-less] AT [CUI-less] while the system tagged it as one token: NCAT [CUI-less]. Conversely, the system tagged 02 sat [C0523807] as two tokens 02 [C0030054] sat [C0523807], and tagged LE's [C0023216] as three tokens LE [C0023216] ' [C1444662] s [C0023901]. A split into multiple tokens occurred more frequently than the reverse situation, which explains why insertions (285) are more numerous than deletions (163). Since the gold standard boundaries are given, a post-processing step could be added in the future to predict a code for the actual abbreviation span based on the CRF output. The cleanest way to cope with this issue though will be to rewrite the system to quit the linear-chain framework of our Task 1 system and switch to a standard Maximum Entropy (or other) classi er. 5
We present an abbreviation resolution system prepared for participation in Task 2 of the 2013 CLEF-eHEALTH challenge. We design our system as a derivation of our Task 1 framework into a supervised Maximum Entropy classi er with lexical, document structure, and UMLS features, complemented by rule-based processing of abbreviations in document headers. Our system obtains an accuracy of 0.664 (strict) and 0.672 (relaxed), which gives it a second position among ve systems. We have tested the bold hypothesis that relying only on the CUIs seen in the training set is enough to provide a strong baseline. The data in the test set and the obtained results show that this hypothesis holds. We have seen that in principle this framework might even fare better than the current best challenge participant. Further work can go into two directions. One is to keep the current framework and strengthen it, for instance by removing deletions and insertions, and by studying the current causes of substitutions. The other is to move to dynamic generation of candidate expansions. This may require to abandon the neat supervised classi cation framework which was possible here, but is needed to extend processing to abbreviations unseen in the training set or in existing databases such as the UMLS Specialist Lexicon LRABR. 6
We acknowledge the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health with grant number R01GM090187. The last author was funded by the Ch^ateaubriand Science Fellowship 2012{2013.
This work was partly funded through project Accordys5 funded by ANR under grant number ANR-12-CORD-0007-03.