-The Institute of Medicine reports a growing demand in recent years for quality improvement within the healthcare industry. In response, numerous organizations have been involved in the development and reporting of quality measurement metrics. However, disparate data models from such organizations shift the burden of accurate and reliable metrics extraction and reporting to healthcare providers. Furthermore, manual abstraction of quality metrics and diverse implementation of Electronic Health Record (EHR) systems deepens the complexity of consistent, valid, explicit, and comparable quality measurement reporting within healthcare provider organizations. The main objective of this research is to evaluate an ontology-based information extraction framework to utilize unstructured clinical text for extraction and reporting quality of care metrics that are interpretable and comparable across healthcare institutions.
The Institute of Medicine reports a growing demand in
recent years for quality improvement within the healthcare
industry[
The current “standard” information extraction systems
perform at the lexical or statistical layers of the clinical
narratives; however, the embedded semantic layers should
also be addressed properly in order to enhance the efficiency
of such systems. It has been shown in non-healthcare related
fields that semantic modeling and ontological approaches can
be used effectively for interoperability operations among
diverse environments [
Development and application of ontologies in the domain
of quality measurements have recently become the focus of
some researchers. Lee et al.[
The real meaning of a concept is relative to the context in which the concept is expressed and, therefore, can be represented in different ways in a given ontology. Identification of such contexts and their representational variations in expression and providing equivalencies among such representations are crucial tasks in any knowledge modeling and information extraction activity, especially in clinical expressions where contexts are defined mostly by section headers (like Family Medical History or Assessment).
While transcription departments in relatively large
hospitals tend to follow standards for documenting section
headers, healthcare providers are often allowed to create their
own versions of section headers in clinical notes. Denny et al.
[
The main objective of this research is to evaluate ontological components in a natural language processing (NLP) system for the purpose of unambiguous extraction of quality of care metrics. Such complementary addition to the existing information extraction system helps enterprise data integration more efficiently (time & cost) in terms of unambiguous data exchange and more objective analytics as part of the enterprise reporting system.
The dataset that we received from MD Anderson (MDA) Quality Engineering Department included the National Surgical Quality Improvement Program (NSQIP) data elements abstracted from 2,085 patients who had undergone surgery in 2011. It includes a spreadsheet of quality of care metrics, such as patient’s Diabetes or Hypertension, as Boolean values (Yes/No) for each patient. We considered this reported operational dataset as the gold standard for our study.
All transcribed documents of the 2,085 patients were extracted from the MDA Electronic Medical Record (EMR) repository (46,835 notes). Python scripting was used to eliminate unwanted characters and extract section headers. A typical clinical note is composed of regions of texts. Each region consists of a section header (like Chief Complaint, History of Present Illness, Physical Exam, etc.) and the relevant content in free text format.
Abstractors at MDA abstract and report quality of care metrics in the preoperative risk assessment section of the form and send them to NSQIP. We have selected the top 5 of these variables in terms of frequency of positive cases (Boolean value=”Yes”) among our gold standard and for the purpose of our research. These metrics include Diabetes Mellitus, Hypertension, Transient Ischemic Attack (TIA), Cardiac Surgery, and Nervous System Tumor.
Quality of care metrics are generally documented by
physicians in clinical notes. Abstractors have to read such
notes and manually extract and report them to NSQIP. It
should be mentioned that abstractors are nursing staff who
have extensive training in NSQIP abstraction protocols &
guidelines. They are also actively participating in NSQIP
certification, auditing, and training programs. Shiloach et al.
[
We implemented the National Institute of Health natural
language processing engine (MetaMap v2012) [
In order to decrease the size of the XML data obtained
from the previous step we pruned unwanted XML elements
from MetaMap’s output. Subsequently, we converted the
XML files into a RDF format and loaded them into a local
instance of AllegroGraph® repository. We also used SPARQL
Protocol and RDF Query Language [
FIGURE I. NLP PIPELINE & ONTOLOGY COMPONENTS
In order to evaluate our section header extraction algorithm we randomly selected 500 test notes (100 notes from each identified quality of care metric category) and evaluated for Precision and Recall. Notes were examined by subject matter experts, annotated for section headers, and compared to the automated section header extraction algorithm. Precision, Recall, and F-measure were calculated as 99%, 97%, and 98% respectively.
In order to build our section header ontology from all
extracted section headers we used SKOS narrower and
broader properties for classifying section headers into
hierarchies and closeMatch, and exactMatch properties [
We identified the root concept for each of the selected
quality of care metrics in SNOMED terminology (Jan 2013
version) and extracted all of their children (or subtypes). The
SNOMED root concepts include: Cardiac Surgery Procedure,
Tumor of Nervous System, Diabetes Mellitus, Hypertension,
and Transient Ischemic Attack. According to the quality of
care metric definition for Diabetes Mellitus, a patient should
also take a diabetes related medication in order to be reported
as a diabetic patient. For this purpose, we included diabetes
mellitus medications in the ontology, with mappings to
RxNorm, from the same reference [
For this ontology we created seven main classes, together with their relationships, in Web Ontology Language: Patient, Note, Region, Utterance, Phrase, Mapping, and Negation. All 46,835 RDF instances described in the method section were imported into the clinical note ontology within AllegroGraph® repository. The number of instances and associated data type properties for each class are shown in Table 1. Including relationships in instance count, the repository contained 70,907,728 triples. We used SPARQL for filtering unwanted concepts (within quality of care metric ontology), negated concepts, and irrelevant sections (within section ontology) from our query results.
We calculated Precision (P), Recall (R), and Micro Fmeasure (F) to evaluate the percentage agreement between our approach and the gold standard. When there are multiple classes of contingency tables, averaging the evaluation scores provides a more general picture of all classes combined. Micro-averaging is the most common averaging method in which each extracted instance is given the same weight. For each quality of care metric under study we sequentially calculated Precision, Recall, and F-measure in 4 conditions to measure the cumulative effect of the two ontologies and the negation context on the base NLP output. For a given quality of care metric, we first performed a query and looked for the root quality metric concept like Diabetes Mellitus. We captured the result of comparing the outcome of this query with the gold standard as the base NLP output layer and in the form of Precision, Recall, and F-measure values. Then we included the quality of care metric ontology in our query and once again calculated agreement measures. We executed our query two more times after adding negation context and section ontology to the previous queries and calculated agreement measures twice more (Table II). False Positives and Negatives (FP, FN) were calculated when there was a disagreement between each query result and the gold standard.
In order to compare isolated effect of each ontology and the negation context on the base NLP output we computed agreement tests in a non-cumulative mode as well. The microaverage results of agreement tests for each layer is compared separately to the gold standard and the difference in Fmeasure with the base NLP output is calculated (Table III).
Recent trends in health care information systems show an increase in requirements for reporting of quality of care metrics by health care organizations, specifically for the government mandated programs with huge financial incentives. Healthcare providers consider EMR the best source for extracting patient information because it most accurately reflects the process of patient care. Nevertheless, such a valuable source of data is usually in narrative format, hence, inaccessible for easy structured reports, and highly costly and time consuming for manual extraction by clinical abstractors.
Our study introduced a framework that may contribute to the advances in “complementary” components for the existing information extraction systems. The application of ontology components for the NLP system in our study has provided mechanisms for increasing the performance of such tools. The pivot point for extracting more meaningful quality of care metrics from clinical narratives is the abstraction of contextual semantics hidden in the notes. We have defined some of these semantics and quantified them in multiple layers in order to demonstrate the importance and applicability of an ontologybased approach in a quality of care metric extraction system. The application of ontology components introduces powerful new ways of querying context dependent entities from clinical narratives.
It is apparent that the effect of ontology components on information retrieval metrics (Precision, Recall, F-measure) are largely dependent on the type of the quality of care metric. Our study shows ontology layers added to the base NLP output, in general, had an increased effect of up to 63% to the performance. The cumulative increase in F-measure was highest for Nervous System Tumors, Cardiac Surgery, and TIA (63%, 57 %, and 32% respectively) and lowest for Hypertension and Diabetes (9% & 1 % respectively) which could be due to the format of representation of these concepts within the clinical narratives. Also, we were able to show and compare the effects of each ontology and negation context in isolation to the base NLP output. It seems section header ontology has a greater effect on the overall F-measure increase compared to the negation context and quality of care metric ontology on all quality metrics except for Nervous System Tumors and Cardiac Surgery. On a micro-average level, for all the 5 concepts combined, section header ontology shows 11% and 5% higher values when compared to the quality of care metric ontology and negation context respectively.
Our ontology-based framework achieved an overall 0.82 F-measure (Micro) which may be high enough to be considered, at minimum, as a decision support tool. Based on the tolerable false positives or false negatives rates, for a given information extraction task, this framework can be considered as an introductory or complementary abstraction method and significantly reduces abstractor’s time for extracting quality of care metrics hidden in the clinical narratives.
We have developed a framework that helps identify contextual semantics within clinical text and extract more meaningful and unambiguous quality of care metrics for the patient care process. Furthermore, by providing bindings to standard terminologies (like SNOMED) the current approach would help quality of care metric extraction process become more objective in nature and deliver structured data for populating clinical warehouses, explicit benchmarking, cohort studies, and other clinical analytics where coded data is vital.
We believe that an ontological approach toward knowledge modeling and information extraction of quality of care metrics from clinical narratives can provide a unique way of improving the clarity of meaning by providing necessary layers of disambiguation, for both human and computational systems. The use of ontology in information extraction system increases the expressivity control of extraction and helps disambiguate the retrieved concepts. This study illustrates the importance of the “complementary” role of ontologies in the existing natural language processing tools and how they can increase the general performance of the quality metrics extraction task.
Rigorous evaluations are still necessary to ensure the quality of these “complementary” NLP systems. Moreover, research is needed for creating and updating evaluation guideline and criteria for assessment of the performance and efficacy of ontology-based information extraction in healthcare and to provide a consistent baseline for the purpose of comparing alternative approaches.