Objective: Abbreviations and acronyms are widely used in the clinical documents. This paper describes using of a machine learner to automatically extract spans of abbreviations and acronyms from clinical notes and map them to the UMLS (Unified Medical Language System) CUI (Concept Unique Identifier). Tasks: A Conditional Random Field (CRF) machine learner was used to identify abbreviations and acronyms. Firstly, the training data was converted to the CRF format. The different feature sets were applied with 10-fold cross validation to find the best feature set to create the machine learning model. Secondly, the identified spans for abbreviation/acronyms were mapped to the UMLS (Unified Medical Language System) CUIs. Thirdly, a rule based engine was applied for disambiguation of terms with multiple abbreviations or acronyms. Approach: A novel supervised learning model was developed that incorporates a machine learning algorithm and a rule-based engine. Evaluation of each step included precision, recall and F-score metrics for span detection and accuracy for CUI mapping. Resources: Several tools which were created in our laboratory were used, including a Text to SNOMED CT (TTSCT) service, Lexical Management System (LMS) and Ring-fencing approach. Also a set of gazetteers which had been created from the training data was employed. Results: A 10-fold cross validation on the training data showed 0.911 of precision, 0.887 of recall and a F-score of 0.899 for detecting the boundary of abbreviation/acronyms and an accuracy of 0.760 for CUI mapping while the official results on the test data showed strict accuracy of 0.447 and relaxed accuracy of 0.488 which is the third team out of the five participating teams. A supervised machine learning method with mixed computational strategies and rule based method for disambiguation of expansions seems to provide a nearoptimal strategy for automated extraction of abbreviation/acronyms.
Clinical notes usually contain a large number of abbreviations and acronyms
without mention of their definition. Also, they often have multiple expansions related to
the context in which they have been used. For instance “BS”, may have two different
expansion of “Bowel Sound” or “Breath Sound”. The context which the “BS” is used
may help unlock the meaning of this abbreviation. In addition, the type of clinical
document may affect the encoding of an abbreviation or acronym. For instance, the
expansion of an abbreviation may be different if it is used in a discharge summary
compared to a radiology report. Although the context and the document type may
create some criteria for interpretation of the ambiguous abbreviations or acronyms in
the clinical notes, correctly expanding them is still a challenging task for NLP
systems. The main purpose of the current work is to map abbreviations and acronyms
from clinical documents to UMLS (Unified Medical Language System)[
This paper is organized as follows: Section 2 covers the related works. Section 3 presents the methods which have been used to identify spans of abbreviation/acronyms and mapping them to CUIs. Section 4 explains experimental results followed by a discussion and conclusion. 2
Successful NLP techniques have been invented for Named Entity recognition and
concept extraction in the general domain while the same tasks are more challenging in
the clinical domain. Extracting concepts like drug names, diagnosis and symptoms
has attracted several researchers. Patrick et.al [
Named entity recognition or concept extraction and classification tasks usually
involve detecting and interpreting abbreviations and acronyms which makes them more
complicated in both the general and medical domains. In [
The authors of [
The main objective of the Task2-ShARe/CLEF eHealth challenge is to identify and map acronyms and abbreviations found in clinical texts to a unique identifier from a controlled vocabulary. Based on the annotation guidelines provided for Task2 an acronym or abbreviation is defined as a substitute for a concept in the text. So, the task consists of two main steps: firstly, identifying and annotating acronyms and abbreviations in the clinical documents and secondly, assigning to acronyms and abbreviations a concept unique identifier from the defined medical terminology, UMLS. Some of the annotations may not match UMLS concepts and should be assigned the value “CUI-less”. Also, based on the guidelines some of the abbreviations and acronyms like measurement units, non-medical acronyms or abbreviations etc. have to be excluded from the final results. 3.2
The dataset for Tasks 2 consists of de-identified clinical free-text notes from the
MIMIC II database, version 2.5 (mimic.physionet.org). A set of 200 notes are
provided for the training task and 100 notes are provided for testing. Notes were authored in
the ICU setting and note types include discharge summaries, ECG reports, echo
reports, and radiology reports. For this task, the focus is normalization of pre-annotated
acronyms/abbreviations to UMLS concepts. Annotators were instructed to annotate
all acronyms/abbreviations that were contained in narratives and not contained in a
list. Annotators were multiple nursing students trained for this task, followed by an
open adjudication step [
A Conditional Random Field machine learner (CRF) was used to identify
“Abbreviation/ Acronyms” in this work. Firstly, the training data is converted to the CRF
format. The CRF format is like a spread sheet in which each column represents a
feature and the last column represents the output tag. BIO tagging notion [
Different feature sets have been applied with 10-fold cross validation to find the best model. Moreover, five categories of features include Context features, Lexical features, Grammatical features, Ring-fence feature and SNOMED CT (Systematized Nomenclature Of Medicine Clinical Terms) features which have been used to construct the machine learning model. The base line feature set were the tokens with a context window of 5 words, that in addition to a token itself 2 tokens before and 2 tokens after the token were also included in the computation. Then the other features were applied sequentially to find out the optimum feature set. In the feature selection process firstly the CRF feature generator was added to train the model and compute the results and record the performance. If the performance increased the F-score with adding a feature, this feature was thought to be useful and it was retained, otherwise, it was removed from the feature set.
To be able to use the tools which have already been developed in our laboratory there was a need to do some pre-processing tasks on the corpus and annotations which the challenge organizer has provided for the Task2. One of these tasks was converting the annotations from XML or pipeline format to our .ann format.
In addition, the whole corpus was loaded into our Lexicon Management System (LMS). The LMS takes care of all new lexical knowledge generated by experts and automatic agents (Knowledge Discovery) and feeds it into the verification process or any other process that needs this information as a Knowledge Reuse process. So, by using the LMS the lexical features of the whole tokens in the training corpus were prepared to feed to the CRF feature generator. LMS initially categorizes the types of all tokens in the corpus as “Known”, “Unknown” or “Unseen”. “Known” means the primary characteristics have been defined for tokens, “Unknown” means the tokens are not resolved yet and “Unseen” means the tokens are un-reviewed. LMS enables checking each “Unseen” and “Unknown” token and add any information about that token to make it known. Spelling corrections, expansions and semantic categories can be set to make a token as known. Moreover, the lexicon is not a simple list of words but an organization of the words into semantic groups and the form of different representations of words. The following semantic groups are defined in the LMS as the words class of the tokens in the corpus or the whole lexicon: Compound Words: In a great deal of clinical terminology, productive forms of words are regularly used. An example is the word vesicle which has the combining form vesico-. The convention will be that the combing form is shown with the hyphen in the LMS, and the canonical form of the compound will include the hyphen, e.g. vesico-ureturic. Compound words are usually defined by two words separated by a non-letter character, typically a hyphen or slash. The hyphen carries the usual morphological interpretation, but the slash is still to be resolved. Neologisms: These are the words constructed to represent new forms typically used in names of organizations or products, e.g. Bayview, HealthCare. This excludes drug names which although neologisms are not included in this category but listed separately. Abbreviations: Shortened forms of words that are not acronyms. e.g. using “backgrd” instead of “background”. Acronyms: Words which are formed from the first letters of a phrase. The letters are usually in uppercase and should be preserved in their orthographic form. Automatic: The words that have been processed and categorized by direct computational methods without manual intervention. Named Entity: A set of classifications of different entity types like drug names, equipment, person names, locations, etc.
Using the above facilities in the LMS valid properties such as spelling corrections and expansion of abbreviation/acronyms were assigned and also assigned semantic classes to tokens to resolve unknown and unseen tokens. Finally, all the properties of the known tokens were extracted from the LMS and applied as one or a set of features in the machine learning model. 3.4
To find out the best feature set to feed to the CRF machine learner five categories of features have been used in the experiments. They were Context features, Lexical features, Grammatical features, Ring-fence features and SNOMED features. Context Features. Context features provides the content information for a token.
The surrounding words usually convey useful information about a token which help in predicting the correct tag for each token. This feature has been used with a window of five, meaning that in addition to the token itself, the 2 tokens before and the 2 tokens after the target token are used as features for predicting the output tag. Orthographic Features. Includes the case tag with the values “Lower” for the tokens with all lowercase characters, “Upper” for the tokens with all uppercase characters and “Title”, for the tokens which start with an uppercase character but follow with the lowercase ones. Lexical Features. Include the expansions of abbreviations/acronyms and correction of misspelt words. As explained above, the LMS provides most of the required lexical features. In addition the lowercase form of tokens has been used as another feature. Grammatical Features. Include Lemma, part of speech (POS) and chunk features.
The GENIA Tagger has been used to produce these features from the training set.
By applying the lemma form of the words a more general description of the words
has been possible. Also, as a low level grammatical information the POS tags of
the words will help in determining the boundaries of instances. The chunk feature
in a similar way assists in determining expression boundaries.
Ring-fence Feature. The existence of complex and compound abbreviations and
acronyms like “R Tib/FIb XR”, “Abd U/S” or “PERC G/G-J TUBE PLMT”
necessitate a solution to welding these forms together. The ring fencing method which
was originally invented in this laboratory to identify complex patterns like scores
and measurements was used for this task. The basic idea is to put a fence around a
group of tokens and dose not let the tokenizer separate them into smaller chunks
but rather to make them an indivisible token. To accomplish this task a process of
running a Trainable Finite State Automata (TFSA) [
As mentioned above to find out the best feature set in the feature selection process each feature is sequentially added to the CRF feature. The feature was retained in the feature set if it increased the F-score otherwise it was removed from the feature set. So, from the above features which were tested in the experiment only orthography features did not help in increasing the performance of the machine learner and consequently it was removed from the feature set. 3.5
After correctly finding the spans of abbreviation/acronyms in the text, the next
step was to map them to the UMLS CUIs. To accomplish this task a simple java
program (CUI_Mapper) was developed which employed the APIs provided by the
UMLS web site [
Moreover, a list of all ambiguous or multi-expansion abbreviations/acronyms has been created. Figure 2, shows the main steps of the mapping algorithm: Detected Span
NO
Is ambiguous?
YES
Is in CUI-Gaz YES
NO Return CUI
Get 5 top matches using CUI-Mapper
Get context
Compute the similarity score ss using Context-Gaz
SS=0
NO
YES Get the more frequent
CUI Using Freq-Gaz
Get the CUI OF best match if exist, otherwise,return
CUI-less
According to the Figure2, there are different paths to follow to match a correct CUI or CUI-less for a span: 1. If the span is not in the ambiguous list but there is a match for it in the CUI-Gaz simply returns the matched CUI from the CUI-Gaz. 2. If the span is not in the ambiguous list and there is not a match for it in the CUIGaz : (a) Get the 5 top matches using the CUI-Mapper; (b) Get the context feature of the span; (c) Compute similarity scores (SS) among the span’s context features and 5 top matches based on the number of common words among them and return the CUI with the most similar match. 3. If the span is in ambiguous list, get context features for the span from the corpus and then compute the similarity score SS between this context and each of the candidate expansions for the span in the Context-Gaz. (a) If the value of SS is zero for all candidate expansions it means there was not any similarity among the contexts of the detected span and the contexts of the previously annotated spans in the training set. So, return the more frequent CUI using CUI-Gaz and Freq-Gaz. (b) If the value of SS is non-zero at least for one of the expansions it means the algorithm was able to find a similarity among the context of the detected span and the context of the previously annotated spans in the training set. So, match the span with the expansion with higher value of SS and return its CUI using Expansion-Gaz and CUI-Gaz. 4. If there is not any match in the step 2, or there is not any similarity or frequency score for the span in the step 3, return CUI-Less.
In addition, there were many abbreviations and acronyms in the training set which were not annotated. They were all extracted from the training set using the LMS and created the More-Expansion-Gaz as explained before, their expansion was applied in model M11 and improved the F-score from 0.833 to 0.840. Applying this on model M6 (model M12) confirmed the effects of applying more expansions in improving the results. So this feature was retained in the final model. Finally, applying the SNOMED CT top category in model M13 lead to another improvement as expected and M13 became the best model with a precision of 0.882, recall of 0.802 and the best F-Score of 0.840 for submission to the Task2 of the challenge. In models 14-16 the expansion forms and SNOMED top category with different window size was tried but they failed to improve the results.
After finding the optimum feature set to construct a model for CRF machine learner, model M10, other experiments were conducted. In a separate process, errors in the annotations provided for the ShARe/CLEF eHealth Task2 corpus were sought by performing validation on the ShARe/CLEF eHealth Task2 training data with a 100% train and test strategy. The results from 100% train and test using model M10 is illustrated in table 2:
The number 115 for the false negatives reflects that there are some annotation errors in the training corpus or the machine learner has seen the tokens before but insufficient context was provided to capture those tokens as abbreviation/ acronyms. Also, the number of 63 false positives illustrates some more annotation errors in the training set. The errors of both categories were corrected manually as a first step, so that the model would not learn from the incorrect examples. In addition, more investigation on the false positives and false negatives revealed that there are some more abbreviations and acronyms which needed to be ring-fenced together as the machine learner was able to tag only part of them. So, some new patterns were defined in the ring fence training set, and also merged more expansions from the LMS with the expansions of annotations in the next steps. Considering these 3 steps as a pre-processing task the experiment was repeated with 10-fold cross validation with approximately the same feature sets. The final results of 10-fold cross validation are shown in table 3 which illustrates approximately the same effect to the 2-fold experiment by adding new features but with higher scores. The best score was recorded for Model M12 with the precision of 0.911, recall of 0.887 and F-score of 0.899. The CUI mapping algorithm on the detected spans of 2765 out of 3660 showed accuracy of 0.760 on the training data. Finally, by applying the optimum model (M12) and the CUI mapping algorithm on the official test data the final results of 0.447 for strict accuracy and 0.488 for relaxed accuracy were published by the challenge organizer in which our team was third team out of five. 5
A machine learning model has been introduced that was designed to participate in the ShARe/CLEF eHealth Task2 challenge. The model was based on the CRF machine learner for detecting the spans of abbreviation and acronyms and a rule-based engine to map the detected spans to the UMLS CUIs. Evaluation results showed 0.911 of precision, 0.887 of recall and 0.899 of F-score for span detection experiment based on 10-fold cross validation of the training data and an accuracy of 0.760 for CUI mapping while the official results on the test data showed a strict accuracy of 0.447 and relaxed accuracy of 0.488 in the CUI mapping in which our team was third team out of five. The results demonstrated that the performance of this system for detecting spans of abbreviation/acronyms is promising but more accurate rules are required for CUI mapping. In addition, further improvements should be possible by adding new features to the model and also enhancing the performance of TTSCT and ring-fencing algorithms. In addition, not all the features which the LMS provides for lexical verification have been used. All these tasks will be our focus of interest in future work.
This work is supported by the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health: R01GM090187. We also would like to give a special thanks to Dr. Stephen Crawshaw and other members in the Health Information Technologies Research Laboratory for their valuable contributions. 7