Objective: There are abundant mentions of clinical conditions, anatomical sites, medications and procedures in clinical documents. This paper describes use of a cascade of machine learners to automatically extract mentions of named entities about disorders from clinical notes. Tasks: A Conditional Random Field (CRF) machine learner has been used for named entity recognition and to capture more complex (multiple word) named entities we have used Support Vector Machines (SVM). Firstly, the training data was converted to the CRF format. Different feature sets were applied using 10-fold cross validation to find the best feature set for the machine learning model. Secondly, the identified named entities were passed to the SVM to find any relation among the identified disorder mentions to decide whether they are a part of a complex disorder. Approach: Our approach was based on a novel supervised learning model which incorporates two machine learning algorithms (CRF and SVM). Evaluation of each step included precision, recall and F-score metrics. Resources: We have used several tools which are created in our lab including TTSCT (Text to SNOMED CT) service, Lexical Management System (LMS) and Ring-fencing approach. A set of gazetteers was created from the training data and employed in analysis as well. Results: Evaluation results produced a precision of 0.766, recall of 0.726 and F-score of 0.746 for named entity recognition based on 10-fold cross validation; and precision, recall and F-measure of 0.927 for relation extraction based on 5-fold cross validation on the training data. On the official test data on strict mode a precision of 0.686, recall of 0.539 and F-score of 0.604 was achieved. Based on the results our team was the 11th out of 25 participating teams. In the relaxed mode a precision of 0.912, recall of 0.701 and F-score of 0.793 was recorded and our team was the 12th. A multi stage supervised machine learning method with mixed computational strategies seems to provide a reasonable strategy for automated extraction of disorders.
Clinical notes usually contain a large number of references to clinical conditions,
anatomical sites, medications and procedures with various surface forms for the same
concept. Using the rich lexical and ontological resources in the clinical domain like
the Unified Medical Language System (UMLS, https://uts.nlm.nih.gov/home.html) or
SNOMED CT (Systematized Nomenclature Of Medicine Clinical Terms) facilitates
normalization of mentions for medical concepts in which the results can be used to
leverage the upper level applications of information extraction or knowledge
discovery. Such a fundamental task is the focus of Task 1 of the ShARe/CLEF eHealth 2013
challenge. Task 1 includes the recognition of references to concepts that belong to the
UMLS semantic group disorders and the mapping of each mention to a unique UMLS
CUI [
Moreover, the context of a clinical concept might have valuable information which helps normalizing and finding the correct CUI for that concept. The context is in turn affected by the type of clinical document. For instance, a mention of a clinical concept may have a different CUI if it is used in a discharge summary compared to a radiology report. Although, the context and the document type may create some trivial criteria for normalization of recognized named entities, it is still a challenging task for NLP systems.
The document types which have been provided for training in Task 1include
discharge summaries, ECG reports, echo reports, and radiology reports. A Conditional
Random Field machine learner (CRF) [
The paper is organized as follows: Section 2 contains a brief explanation of the related work. Section 3 presents the methods which have been used to identify spans of disorders. Section 4 explains the experimental results followed by a discussion and conclusion. 2
Successful NLP techniques have been developed for Named Entity recognition and
concept extraction in the general domain while the same tasks are more challenging in
the clinical domain. Extracting concepts like drug names, diagnosis, symptoms has
attracted several researchers. Patrick et.al [
The Mayo Clinic information extraction system [
The main objective of the Task1-ShARe/CLEF eHealth challenge is to identify a
span of text in the note that corresponds to the mention of a disorder on the one hand
and then mapping it to a CUI from the provided terminology (SNOMED CT) on the
other hand[
Based on the annotation guidelines provided for Task1 a disorder reference is
defined as any span of text that can be mapped to a concept in the SNOMED-CT
terminology, which belongs to the disorder semantic group [
The dataset for Tasks 1 consists of de-identified clinical free-text notes from the MIMIC II database, version 2.5 (mimic.physionet.org). A set of 200 notes is provided for the training task and 100 notes are provided for testing. Notes were authored in the ICU setting and note types include discharge summaries, ECG reports, echo reports, and radiology reports. The training data consist of 3864 annotations for disorder mentions. Some of them are a single annotation (e.g. “headache” or “hypothyroidism”) while the others are multiple adjacent tokens (e.g. “neck stiffness” or “MCA aneurysm”) or multiple tokens with a distance from each other (e.g. “abdomen … nontender”). About 30 per cent of the annotated disorders belonged to the last category, multiple tokens with some distance between each other. In conversion of xml annotations to .ann format (section 3.3) each single token of a phrase disorder was annotated as a single disorder which increased the number of annotations from 3864 to 5949. Then a relation was defined among any 2 sequential tokens of the phrase disorders. The details will be explained in the section 3.5. 3.3
Figure 1 shows the main work-follow for identifying references to disorders from the
clinical documents. A Conditional Random Field machine learner (CRF) was used to
identify mention of disorders in this work. Firstly, the training data was converted to
the CRF format. The CRF format is like a spread sheet in which each column
represents a feature and the last column represents the output tag. The BIO tagging
convention [
Different feature sets have been applied with ten-fold cross validation to find the best model. In the feature selection process firstly a feature was added to the CRF feature generator to train the model. Then the result was predicted and the performance was recorded. If the performance increased the F-score with adding a feature, this feature was thought to be useful and retained in the feature set; otherwise, it was removed from the feature set.
To be able to use the tools which have already been developed in the Health Language Laboratories, School of Information Technology, The University of Sydney, there was a need to do some pre-processing tasks on the corpus and annotations which the challenge organizer has provided for the Task1. One of these tasks was converting the annotations from XML or pipeline format to our own .ann format.
In addition, the whole corpus was loaded into the Lexicon Management System (LMS). The LMS takes care of all new lexical knowledge generated by experts and automatic agents (Knowledge Discovery) and feeds it into the verification process or any other process that needs this information (Knowledge Reuse). So, by using LMS it was possible to prepare the lexical features of the tokens in the training corpus to feed to the CRF feature generator. The LMS categorizes the types of the all tokens in the corpus as “Known”, “Unknown” or “Unseen”. “Known” means the token has been learned previously and so the primary characteristics have been defined for token, “Unknown” means the tokens are not resolved yet and “Unseen” means the tokens are un-reviewed yet. LMS enables checking each “Unseen” and “Unknown” token and also adding any known information about that token to make it known. Spelling corrections, expansions and semantic categories can be set to make a token as known. Moreover, the lexicon is not a simple list of words but an organization of the words into semantic groups and the form of different representations of words. The following semantic groups are defined in the LMS as the words class of the tokens in the corpus or the whole lexicon:
SHAre/CLEF eHealth
Task1 XML
Annotations
Convert To ann format SHAre/CLEF eHealth
ann Annotations SVM Machine Learner Relation between Spans
Post Processing Spans for Disorders
CRF Machine Learner Compound Words: In a great deal of clinical terminology, productive forms of words are regularly used. An example is the word vesicle which has the combining form vesico-. The convention will be that the combing form is shown with the hyphen in the LMS, and the canonical form of the compound will include the hyphen, e.g. vesico-ureturic. Compound words are usually defined by two words separated by a non-letter character, typically a hyphen or slash. The hyphen carries the usual morphological interpretation, but the slash is still to be resolved. Neologisms: These are the words constructed to represent new forms typically used in names of organizations or products, e.g. HealthCare. This excludes drug names which although neologisms are not to be included in this category. Abbreviations: Shortened forms of words that are not acronyms. e.g. using “backgrd” instead of “background”. Acronyms: Words which are formed from the first letters of a phrase. The letters are usually in uppercase and should be preserved in their orthographic form. Automatic: The words that have been processed and categorized by direct computational methods without manual intervention. Named Entity: A large set of classes of different entity types like drug names, equipment, person names, locations, etc.
Using the above facilities in the LMS valid properties like spelling corrections and expansion of abbreviation/acronyms were assigned and also semantic groups were set to tokens to resolve unknown and unseen tokens. Finally, all properties of the known tokens were extracted from the LMS and applied as one or a set of features in the machine learning model.
Similar to the feature generation process for CRF machine learning, feature sets for SVM machine learner were created to extract relationships between pairs of entities. The details of CRF and SVM experiments will be explained in the following sections. 3.4
To find out the best feature set to feed to the CRF machine learner for identifying
spans of disorder references five categories of features have been used in our
experiments including:
Context Features. Includes the Bag-of-words which provides the context
information for a token. The surrounding words usually convey useful information
about a token which help in predicting the correct tag for each token. This feature
has been used with a window of five tokens. This means that in addition to the
token itself, the 2 tokens before and the 2 tokens after the target token are considered
for predicting the output tag.
Orthography Feature. Includes the case tag with the values “Lower” for the
tokens with all lowercase characters, “Upper” for the tokens with all uppercase
characters and “Title”, for the tokens which start with an uppercase character but
following with the lowercase ones.
Lexical Features. Includes the expansions of abbreviations/acronyms and
correction of misspelling words. As explained before, the LMS provides most of the
required lexical features. In addition the lowercase of words has been used as another
feature.
Grammatical Features. Includes Lemma, part of speech (POS) and chunk
features. The GENIA Tagger has been used to produce these features from the
training set. By applying the lemma form of the words a more general description of the
words has been possible. Also, as a low level grammatical information the POS
tags of the words will help in determining the boundaries of instances. Chunk
features in a similar way assists in determining expression boundaries.
Ring-fence Feature. The existence of complex and compound phrases mainly for
scores and measurements and also for other named entities in the clinical domain
necessitate a solution to welding these complex phrases together. The ring fencing
method which was originally invented in this Laboratory to identify complex
patterns like scores and measurements is used here. The basic idea is to put a fence
around a group of tokens and not allow the tokenizer to break them into smaller
chunks but rather keep them together as an indivisible token. To accomplish this
task a phase of running a Trainable Finite State Automata (TFSA) [
The focus of the experiments was on identifying spans of any “single token” or “multiple adjacent tokens” which are a reference to a disorder while for identifying the reference to a disorder with “multiple separate tokens” SVMs experiment (section 3.5) were utilized in similar way. 3.5
Once the named entity recognition (NER) task was completed, an SVM was used to classify the relationships between parts of multi-word disorder mentions. Each token of a complex mention of “Disease_Disorder” has been identified and a relationship defined of “part_of_disorder” between each two consecutive tokens in the mention. Also, six categories of features were used to train the SVM to compute valid relationships between pairs which are: Context features. Includes three words before and three words after each entity in a relation, words between the two entities, words (inside) of each entity and distance between two of the entities in a relation. Orthography features. Includes title case of first entity and second entity. Lexical features. Identifies that if the two entities of a relation are in an acronym form or not. Semantic features. Includes the types of the two entities determined by the CRF classifier and the entity types between the two entities. Grammatical features. Includes lemma, POS tag and chunk feature of both entities. Similar to NER experiment with CRF, theses features were extracted using the GENIA Tagger. SNOMED features. Includes SNOMED CT id (cid), term (term-tag) and top category (cat-tag) for each of the two entities in a relation. Similar to NER experiment with CRF, theses features extracted using TTSCT. 4
To improve the above result more features were added and another experiment was run with ten-fold cross validation. New patterns were defined in the ring fence algorithm to capture more complex patterns and also the CUIs from the CUI-Gaz were applied as another feature. An orthography feature (case feature) was used in the new experiment as well. The final results in ten-fold cross validation are shown in Table 2. According to Table 2, applying the case feature slightly increased the F-Score while applying the CUI feature improved the F-Score by about 0.07. The best score was recorded for Model M11 with the precision of 0.766, recall of 0.726 and F-score of 0.746.
Table 3 illustrates the SVM results for classifying the relationships between the adjacent spans of disorders which were identified using the CRF machine learner in the previous step. Class 1 represents a valid relationship of “part_of_disorder” between pairs of adjacent entities where they are both annotated as “Disease_Disorder” and class 0 represents the relationship of any other types of entities. As tokens of a complex mention of a Disease_Disorder all appears in one sentence, to improve the results relationship was only created among entities in a single sentence in the training process.
According to the results in the Table 3, among the features which have been used for finding the best model for training of the SVM, the majority of context features (used in models M1, M2, M9) and the only semantic feature (used in model M3) were useful and improved the results for both classes while using of the other features in models M4 to M8 and M10 to M14 decreased the scores. So, the best model for training the SVM was model M9 with F-score of 0.622 for class 1, 0.960 for class 0 and 0.927 for both classes.
A cascade of machine learning models that was designed to participate in the ShARe/CLEF eHealth Task1 challenge has been introduced. The models were based on a CRF machine learner for detecting the spans of disorder references and a SVM machine learner to identify relationships between spans which are a part of complex references for disorders. Evaluation results showed precision of 0.766, recall of 0.726 and F-score of 0.746 for NER and 0.927 for all three scores for relation extraction on the training data while the official results on the test data showed precision of 0.686, recall of 0.539 and F-score of 0.604 in the strict mode and precision, recall and Fscore of 0.912, 0.701 and 0.793 in the relaxed mode. The results demonstrated that the performance of this system still needs improvement for the purpose of the task 1 of the challenge; however a multi-stage supervised machine learning method with mixed computational strategies seems to provide a near-optimal strategy for automated extraction of disorders. Further improvements are possible by adding new features to the model and also enhancing the performance of TTSCT and ring fencing algorithms. Thus far not all the features which the LMS provides for lexical verification have been used. These tasks will be the focus of interest in future work. 6
This work is supported by the Shared Annotated Resources (ShARe) project funded by the United States National Institutes of Health: R01GM090187. We also would like to give a special thanks to Dr. Stephen Crawshaw and other members in the Health Information Technologies Research Laboratory for their valuable contributions. 7 [7] J. Patrick, Y. Wang, & P. Budd, "An automated system for conversion of clinical notes into SNOMED clinical terminology", in Proceedings of the fifth Australasian symposium on ACSW frontiers - Volume 68, Australian Computer Society, Inc.: Ballarat, Australia. pp. 219-226, 2007.