Medication reconciliation is one of the most important priorities of national and international patient safety e orts, due to the numerous deaths and adverse drug reactions caused by inappropriate medication use. One of the main challenges of general practitioners (GPs) is to get an overview of changes in the patients' medications after transitions between healthcare institutions. This paper presents how Natural Language Processing of free text notes such as discharge summaries is used to automatically extract information about medications and how this can be compared to the patient's existing medication list in an electronic patient record (EPR) system. The functionality has been developed in a user initiated project, as a cooperation between four di erent vendors and with a strong involvement of the end users. The functionality is available for most Norwegian GPs and is seen as a very useful tool in the medication reconciliation process.
1.1
Medication reconciliation is the proposed formal, systematic strategy to
overcome medication information communication challenges and reduce unintended
medication discrepancies that occur at transitions in care [
Copyright c 2014 by the paper's authors. Copying permitted for private and academic purposes.
The lack of medication reconciliation is seen as a signi cant challenge to patient safety. Several studies have shown that di erent healthcare professionals, the patient and the relatives do not have the full overview of the patients' prescribed medications, particularly after transitions between di erent healthcare insitutions. Not having the full overview of medication use is one of several causes of adverse drug reactions [3{5].
Several ongoing initiatives focus on processes where di erent health care providers such as physicians, nurses, and pharmacists cooperate with patients and their relatives to ensure accurate and consistent medication lists across transitions in care. In Norway, three of eleven focus areas in the Norwegian Patient Safety Programme: In Safe Hands 1 are related to medications: Medical reconciliation, drug review in nursing homes and drug review in home care services. Other programmes have been introduces in other countries. For example, the Institute for Safe Medication Practices in Canada2 support medication reconciliation at a provincial, national and international level, and the Agency for Healthcare Research and Quality3 in the US has developed a toolkit for organizations to develop medication reconciliation based on knowledge of best practice.
However, the medication reconciliation process is tedious and time-consuming, and there has been a lack of electronic systems that facilitate the process of comparing and adjusting medication lists from di erent sources such as discharge summaries from hospitals or nursing homes and the "medications in use" list in the general practitioners (GPs) electronic patient record (EPR) system. In Norway, this has been done by the GPs, who had to print lists of medications on paper from their own EPR system as well as lists received in e.g. discharge letters, and then comparing each medication manually. The result then had to be entered into the EPR system. If the patient uses many medications the comparison process becomes complex and sometimes neglected.
This paper presents how natural language processing of free text notes such as discharge summaries has been used to automatically extract information about medications and how this can be compared to the patient's existing medication list in an EPR system. The functionality has been developed in a user initiated project, as a cooperation between four di erent vendors and with a strong involvement of representative end users. 1.2
Several studies have shown how information technology and natural language
processing (NLP) is used to facilitate the medication reconciliation process [
Basic lexical approaches, such as keyword matching, may sometimes be
appropriate for detecting simple concepts from medical free-text [
Traditional deep-linguistic grammars can be di cult to implement and are
prone to producing too many and ambiguous results. A simpler approach is to
use partial or shallow parsing [
The project was initiated in 2011 by the Norwegian College of General Practice (Norsk forening for allmennmedisin - NFA) Reference Group for EPR. The members of the reference group are general practitioner specialists who have a special interest in ICT and EPR systems, and a particular interest in how to obtain improved systems that can function as useful tools for the GPs in their daily work with patients. Developing a tool for medication reconciliation was the top priority of the reference group, and a requirements speci cation for the solution was developed in cooperation with the Norwegian Centre for Informatics in Health and Social Care. The suggested solution was to develop a module for extracting and comparing medication information from di erent sources. During the period from September 2011 to September 2012, a project plan was developed and agreed between NFA, Vivit4, and the three major Norwegian general practice EPR system vendors. Vivit's role in the project was to develop functionality for recognizing and extracting medication information that could be integrated with 4 Vivit is a small Norwegian company with high competence and experience in the eld of health informatics. Vivit focus on user-centered design and development, including empirical methods for user-centered requirements elicitation and analysis, usability testing and evaluation of clinical information systems, and methods for search, de-identi cation, and secondary use of electronic patient information. The company was founded in 2009 by former health informatics researchers from The Norwegian University of Science and Technology (NTNU). all the major Norwegian general practice EPR systems. The partners signed the project agreement in September 2012, and the rst version of the system was implemented and tested by pilot users in February 2013. In May, 2013, version 1.0 was launched, and from January, 2014, the solution has been available for the majority of Norwegian GPs. 2
The main objective of the project was to develop a method for extracting medication information from unstructured text.
Several issues had to be resolved to ensure a successful project outcome: The discharge notes that were to be matched against the medication information in the primary care EPR system were provided as free-text with no semantic or structural markup. Input from several di erent hospital and elderly care EPRs were to be expected, with no standardized ways of structuring the information. Accordingly, a method for extracting medication information from natural language had to be devised.
A prerequisite for the project was that the new medication reconciliation functionality had to work with three di erent EPR systems developed by separate, competitive, vendors. Each system had its own approach towards storing medication information, hence a joint interchange format had to be developed. 3
A set of 100 extracts from discharge notes were collected from various general
practice patient records. The source material was discharge notes that were sent
via electronic messaging from the hospital where the patient in question had
undergone treatment. Since we were only interested in medication information,
only the parts of the discharge note that contained such information were used.
This also helped ensure that there was no identifying information in the
extracted text. To evaluate the system, a gold standard was needed. An annotator
was given the task of marking up all relevant medication information
(medication name, dosage and frequency) in the 100 training notes. The annotation was
done independently from the software development. The annotation was
performed by a health informatics researcher. Based on the information available to
us, we created a set of EBNF-like [
Upon project delivery, the results were evaluated on a sample of 25 notes from the original 100 note test set. Table 1 summarizes the results. To understand the results, note that dosage and/or frequency will never be extracted if an associated medication is not found. If a medication has no dosage, it will still count as a true positive for MD and MDF if no dosage is extracted. Also note that no false positives were found for this evaluation. From user feedback we later learned of (and xed) false positives, but in practice these are fairly rare.
Match type
Medication (M) Medication and dosage
(MD) Medication, dosage and frequency (MDF)
The module has been implemented by the three EPR vendors that participated in the project. Figure 1 shows a part of a screenshot from the Infodoc Plenario EPR system 5.
The lower left part of the gure shows the incoming text that has been pasted from a discharge summary. The lower right part of the gure shows the information that has been recognized by the Vivit module in bold text. The upper left part of the gure shows the initial medications in use list, and the upper right part of the gure shows the recognized medications from the discharge summary. The table is sorted in order to match similar medications on corresponding lines.
The physician has to assess every entry in the list in order to accept or reject the suggested changes of the list. New medications can easily be prescribed as the new values are automatically transferred to the prescription user interface. 4.2
All the vendors involved in the project have implemented the solution in their EPR systems, hence the solution is available for practically all Norwegian GPs. However, the vendors have di erent processes for making the new functionality known to their customers and not all GPs are aware of the functionality. In addition, the vendors have implemented the module in di erent ways, and the user interface solution may a ect how the users perceive the usability of the medication reconciliation solution. A systematic evaluation of the solution is currently being carried out, but the results are not yet ready. However, the responses from many GPs are positive, although there is obviously room for improvement. Some GPs are enthusiastic and state that 5 http://www.infodoc.no/ The new tool for synchronizing medication lists is really simplifying the task of comparing and adjusting medication lists between our patient records and the hospitals. (Specialist General Practitioner, using the module with InfoDoc EPR system) and
The new tool is very useful and makes my daily work easier! (GP, using the module with WinMed 3.0 EPR system)
Other users nd the functionality useful, but miss more information about dosage and frequency, which is important when comparing medications. Further, the module is able to recognize and correct some spelling mistakes, but not all. If this could be improved the usability would have been better. At the moment, medication from unstructured text is better recognized than text in a semistructured format, which is often used in the Care Sector in the Municipalities. Finally, some GPs would like the functionality to be more automatic, as it still may take considerable time to review the patients medications even with the module. After the di erent lists are compared, the physician must determine which medications the patient should use or not, and this has to be done for example by clicking on each individual medication. 5
The new module does not make the medication reconciliation process automatic, but it o ers a tool that can be seen as a decision support system that enables physicians to easily import medication information from various sources into the EPR system. The EPR systems were developed in the same programming language (C#), but they used di erent versions of C# and .NET. The developed functionality had to work with all systems, so a lowest common denominator approach had to be used when writing the software. This also put some constraints on the use of third-part libraries.
There is no standardised way of denoting medication information. As an example, when denoting medication frequency, the terms x 1 and [1+0+0+0] both mean the same thing (once a day). To some extent the project had to cope with such syntactic di erences.
As with almost all clinical text, spelling errors are common in discharge notes. It was a strong requirement that minor spelling mistakes should be handled. We had access to 100 discharge notes as training/example data. This is a fairly small amount of data, which made the use of machine learning methods di cult. Moreover, no gold standard was available, meaning that this had to be developed as part of the project. The annotation was performed by a health informatics researcher with background in computer science. Using more annotators with di erent backgrounds (e.g. pharmacists, healthcare professionals) could probably have increased the validity of the annotation.
The recall rate for medications is mostly explained by the use of medication names that were not found in the FEST database (typically colloquial terms) and major spelling errors that the module was not able to correct. The lower recall for frequency descriptions can be ascribed to a larger variety of expressions when describing frequencies than what is the case for dosages. The evaluation of the module shows that most medications are recognized, while the recall for dosage and frequency is lower. However, as it is easy to change dosage and frequency values, the users nd the functionality very helpful as long as most of the medications are recognized. As more end-users start using the new functionality, feedback and error reports are basis for continuously improvement of the grammar. With partial parsing, the grammar strictly de nes the elements that we are able to extract. This means that every false positive requires adding additional rules to the grammar. To make this work, the parser developer must take care so that grammar additions do not break previous functionality. In our experience this calls for a structured, iterative approach to grammar development. Having a full set of unit test cases makes grammar development a lot easier and safer. A positive side-e ect is that with this approach the precision is usually very high, at the expense of lower recall. Another problem with shallow parsing approach is that it is not ideal for extracting complex narrative. For simple medication, dosage and frequency extraction we have seen that this is not a big problem due to the usually structured notation. However, building a grammar that e.g. extracts the reasoning behind a prescribed medication will be a lot more di cult, this because reasons usually are given in natural language. A shortcoming with the evaluation was that it was performed on the same material as was used for building the grammars, this due to the relatively small amount of data available to us. 5.1
Experiences from several ICT projects in the health care sector show that a lot of projects do not involve end users in the development. The results are often systems that can be found annoying and time-consuming, failing to meet the needs of the end users. The project presented in this paper was initiated by highly active and engaged users with a real need for improved functionality of their EPR systems. The involvement of the users and their participation in the pilot testing and the approvement of the solution was a clear advantage in order to ensure the success of the project.
The project presented in this paper has shown how joint e orts and cooperation of end users and vendors have led to the development of new, common functionality that supports the medication reconciliation process. 6.1
As part of the project follow-up, the medication reconciliation module will be continuously improved and updated bi-monthly with the latest version of FEST as well as general quality improvement. This will ensure that the module stays up to date.
Repeating the evaluation on a new set of test data will give a clearer view of real-world precision and recall. Another important aspect is to measure enduser satisfaction with the medication reconciliation tool and how this a ects their daily work. For this purpose, a survey is being performed on end users and results are likely to be ready later this year.
To improve recall, making use of machine learning technologies is a viable option. This will, however, require more test data. A bene t of having a handcrafted extraction module with high precision is that it can probably be used for automated annotation (i.e. applying the module to new test data) so as to make the gold standard creation job much easier.
ATC EBNF EPR/EHR FEST