A commercially used expert system using multiple-classification rippledown rules applied to the domain of pharmacist-conducted home medicines review was examined. The system was capable of detecting a wide range of potential drug-related problems. The system identified the same problems as pharmacists in many of the cases. Problems identified by pharmacists but not by the system may be related to missing information or information outside the domain model. Problems identified by the system but not by pharmacists may be associated with system consistency and perhaps human oversight or human selective prioritization. Problems identified by the system were considered relevant even though the system identified a larger number of problems than human counterparts.
A drug-related problem (DRP) can be broadly defined as “…an event or
circumstance involving drug therapy that actually or potentially interferes with desired health
outcomes”[
Home medicines review (HMR) is a Commonwealth Government funded service
conducted by accredited pharmacists to identify and address DRPs among eligible
patients [
An HMR is initiated for eligible consenting patients by a GP. Eligible patients are
identified if they regularly take 5 or more medications among other criteria [
A core component of an HMR is an interview between the pharmacist and the
patient, with interview typically conducted in the patient’s home. The interview, elicits
additional information such as: actual medication use, additional non-prescribed
medications, an understanding of the patient’s motivation behind actual rather than
directed medication use, and the patient’s health and medication knowledge [
The amassed information is reviewed by the pharmacist to identify actual and
potential DRPs. The pharmacist writes a report of findings for the patient’s GP, which
includes recommendations to resolve any actual or potential problems. Consultation
between the GP and the patient culminates in an actionable medication management
plan designed to trial changes to existing therapy, and ideally, lead to improved
medication use and improved patient health outcomes [
An important component is the professional skill of the pharmacist to be able to identify clinically relevant DRPs from the available information. This requires a wide scope of knowledge, not only of medications, but of evidence-based guidelines and contemporary management of a variety of medical conditions.
Evidence-based guidelines can be difficult to implement due to their apparent
complexity. An example is provided from Basger et al.’s Prescribing Indicators in
Elderly Australians: “Patient at high risk of a cardiovascular event (b) is taking an
HMG-CoA reductase inhibitor (statin)”[
A commercial product developed by Medscope, Medication Review Mentor
(MRM)[
No prior research had been undertaken to determine the clinical decision support
capacity of this commercial software, apart from contemporary research by the
authors. This contemporary research by the authors assessed opinions from
pharmacology experts and had determined that MRM is capable of identifying clinically relevant
DRPs [
This evaluation attempts to provide light on the scope of DRPs that can be identified by this software by presenting summary counts and examples of the types of problems that were identified by MRM and by pharmacists. This paper evaluates the similarities and differences between pharmacist findings and MRM findings more in terms of a qualitative comparison by highlighting common findings, extremes of difference and discussing the possible advantages and limitations of the software, as well as discussing areas for potential improvements. 2
The decision support component of MRM is a knowledge-based system which uses
MCRDR as its inference engine. MCRDR provides the knowledge engineer a way to
incrementally improve the quality of the knowledge base through the addition of
either new rules – which are added when the system fails to identify a DRP, or
refinements to existing rules – which are added when the system incorrectly identifies an
inappropriate DRP. The system’s knowledge base is managed by medication review
experts, who regularly review cases, examining the findings of the system for that
case, and then adding/refining rules until the system produces a wholly correct set of
findings for that case [
Australia-wide data collected during 2008 for a previous project, examining the
economic value of HMRs, was used for this study [
The HMR data were entered into MRM and DRPs identified by MRM were
recorded. MRM utilized a wide range of information including basic patient
demographics such as age and gender, medication type including strength, directions
and daily dose. MRM could calculate daily dose from strength and directions in many
cases. Duration of use of medication could be entered, which included options of less
than 3 months and more than 12 months. Medications were assigned Anatomic
Therapeutic Chemical classifications (ATC) [
Diagnoses could be entered and were based on the ICPC2 classifications [
At the time of the data entry and collections of results, August 2011, MRM
contained approximately 1800 rules [
Direct comparison of the DRPs identified by MRM and those identified by the original pharmacists was not possible due to the individual textual nature of each DRP. Each DRP identified by either the pharmacist or MRM was mapped to a concept (defined here as a theme) that described the DRP in sufficient detail to allow comparisons of similarity and difference between pharmacists and MRM. The themes often described the type of drug or disease and other relevant factors involved. The development of a list of themes and the mapping of DRPs to themes was performed manually by the author, a qualified pharmacist.
Examples of the text of two DRPs identified by a pharmacist and by MRM in the same patient are shown in Table 1. These DRPs were assigned the theme Hyperlipidemia under/untreated, which captured the basic problem identified within the text of each DRP.
These themes provided a common language for comparison of the DRPs found by
the original pharmacist reviewer and MRM. The initial themes were created where at
least two of three published prescribing guidelines for the elderly [
A list of 129 themes was developed. Many themes described disease states and/or drug classes describing identified DRPs in general terms. A descriptive analysis of the themes was performed.
The number of unique themes found in each patient was considered more important than the raw number of themes found in each patient. That is where two DRPs matched the same theme in the same patient, that theme was counted once. The reason behind this decision was to compare the number of different types of conceptual problems that could be identified across patients rather than raw numbers across patients.
Each theme identified in each patient was allocated into one of three categories: 1. Identified by pharmacists only, 2. Identified by MRM only or 3. Identified by both. The patient cohort was predominantly female, with an average age of 80 and an average of 12 medications and 9 diagnoses, as described in Table 2. Pharmacists identified a total of 2020 DRPs, an average of 3.5±1.8 per patient, with a range of 0 to 13 DRPs. MRM identified 3209 DRPs, of which 256 were excluded due to duplicated findings, leaving 2953 MRM DRPs, and an average of 5.2±2.8 per patient, ranging from 0 to 16 DRPs.
The 2953 MRM DRPs were able to be assigned to 100 different themes that described in general terms the central issue of each of the DRPs. Similarly, the 2020 pharmacist DRPs were able to be assigned to 119 different themes. Ninety of these themes which were identified by pharmacists were also able to be identified by MRM. Within these 90 themes, the software was able to identify the same issues as the pharmacists in one or more of the same patients for 68 particular themes.
The number of different themes identified by MRM or by pharmacists per patient was considered more important than the raw totals. The 2953 MRM DRPs were aggregated into 2854 themes. Pharmacist DRPs which were clearly identifiable as compliance or non-classifiable cost-related problems and outside the scope of MRM’s ability to identify were excluded, leaving 1726 pharmacist DRPs which were aggregated into 1680 themes.
MRM was able to identify the same themes as identified by pharmacists in the same patients 389 times, a 23% (389/1680) overlap of pharmacist findings by theme and patient. This then left 1291 themes identified by pharmacists only and 2465 themes identified by MRM only. For each patient a Jaccard coefficient was calculated as the number of themes in common divided by the number of different themes found by either MRM or pharmacists. For the 570 patients Jaccard coefficients ranged from a minimum of 0 to a maximum of 1, with a mean of 0.092 ± 0.117.
The top five themes by number of patients in common are shown in Table 3. Not surprisingly several of the most common themes found align with common health conditions in this cohort, namely hyperlipidemia and osteoporosis.
Some of the problems that can be identified by the software are shown in Tables 3 and 4. Table 3 shows there is some overlap of the ability of MRM to find the same kind of problems as pharmacists in the same patients. However, both pharmacists and MRM find many instances of the same problem in different patients. Table 4 shows examples of some of the themes at the extremes of overlap. The two example themes calcium-channel blocker and reflux and anti-lipidemic drug, no indication were identified in many patients by MRM but only once each by pharmacists. Similarly, the two example themes vitamin, no indication and combine medications into combination product illustrate that pharmacists identified many patients with particular problems that MRM could not identify. Osteoporosis (or risk) may require calcium and or vitamin D Renal impairment and using (or check dose for) renally excreted drugs Hyperlipidemia under/untreated Sedatives long-acting or sedative long term NSAID not recommended (heart disease/risk of bleed/other)
Patients MRM found 137 122 83 55 59 117 48 31 31 28
Patients: pharmacists + MRM 49 24 20 18 17 205 146 94 68 70
The majority of the unique pharmacist themes involved non-classifiable, mostly drug cost and compliance, problems. These pharmacist-only themes were not captured in the knowledge domain model. Although the majority of unique MRM themes could have been identified by pharmacists they were not. This was not due to lack of information on the part of pharmacists but more likely to be due to pharmacists having additional knowledge that rendered these issues moot. It is also possible that pharmacists were not aware of or simply missed these particular issues. Alternatively, the software may have produced erroneous findings.
The wide variety of variables including temporal context encapsulated in the model were manifested in the broad scope of problems that could be identified by the software. For 68 themes (out of 100 themes identified by MRM) the software showed the ability to identify the same issues that pharmacists could find in the same patients. In some circumstances half to all instances of a theme identified by pharmacists was also identified by MRM; most of the themes shown in Table 3 are examples of this.
The broad scope of themes and similarity of identification of themes in the same patients as pharmacists is encouraging; however, there were many patients who had particular problems identified by either MRM or pharmacists but not by both. Further, twenty-two themes were identified by MRM and by pharmacists without any patients in common. Several explanations are posited to account for these differences.
The first and main point is knowledge not captured and subsequently not able to be utilized by the software. Extending this point, knowledge may have been available but not entered into the software because it was not recorded anywhere by either the patient’s GP or the reviewing pharmacist. Several themes stated some drugs had no indication for use because no suitable diagnosis was assigned to those patients. An example in Table 4, anti-lipidemic drug, no indication, shows MRM found many instances of this potential problem but pharmacists did not identify this as an issue. Does this mean pharmacists were aware of the indication for the drug? Or does it suggest pharmacists missed the opportunity to identify unnecessary medication?
Overall MRM found more problems than pharmacists. It is not unreasonable to
suggest pharmacists may lack consistency in identifying DRPs. Correspondingly, it is
not unreasonable to suggest MRM exemplifies consistency, as it is after all computer
software. Several studies examining clinical decision support, including two
prototypes on which MRM was based, have identified that humans lack consistency or lack
the capacity to identify all relevant problems in contrast with the software [
MRM did find substantially more problems than pharmacists, which raises some concerns about potential alert fatigue, a known limitation of many clinical decision support systems, wherein the system identifies so many irrelevant problems that the user simply ignores it entirely. It should be noted a portion of MRMs findings were duplications, 256 of 3209 DRPs. The central requirement and unfortunately concomitant problem of clinical decision support is the need to have sufficient information to present findings in context of the patient’s current clinical situation. The application of MCRDR attempts to address the problem of context through incorporation of an extensive array of variables integrated with a knowledge base of many patient cases and inference rules.
However, it appears that MRM may not suffer from alert fatigue, as separate
research that we have conducted, concerning the clinical relevance of the DRP findings
of MRM and of pharmacists, was recently completed [
A specific advantage of this implementation of MCRDR was the use of case-based reasoning, allowing the knowledge domain expert to readily add new rules and refine existing rules. This method incrementally increases the precision of rules in context of the uniquely varied situations encountered through amassing knowledge of individual patients. This is an important point, as the development of new medications, or new applications of existing medications, and ever expanding medical knowledge needs to be to be incorporated into such software on an ongoing basis to maintain the relevance of the knowledge base.
Due to the ability to easily add and refine the rules and knowledge-base a followup study may produce different, likely improved results. A subsequent investigation applying the same patient cases to the software and comparing the differences may be performed to determine whether DRP identification can be further enhanced over time.
MRM appears to work well in the HMR domain, but improvements may include a greater extent of variables such as compliance or cost-related concepts to widen problem detection scope as well as increasing accuracy of problem identification. Rule refinement to reduce the occurrence of duplicated DRPs is warranted. Another potential issue involves medication classification which was based on the ATC classification system. The ATC classification system included codes for combination products. There may be limitations when attempting to create rules based on individual ingredients within combination products as each individual ingredient is not uniquely identified. Additionally, with the impending implementation of national electronic health record standards, data entry limitations such as transcription errors or missed data entry may be minimized by implementing these standards. 6
The use of ripple-down rules in this software did perform well in the complex and detailed HMR knowledge domain. It showed a reasonable degree of similarity with the human experts in the both the range of problem types that could be identified within its scope of knowledge, and in the frequency of problems found. MRM cannot find some of the problems that pharmacists could find, some things will always be missed because of incomplete data.
The truly interesting aspect is the software’s capacity to identify more problems than pharmacists. This capacity to identify more problems did not appear to involve lack of relevance, but it is likely to be a strong indication of the consistent methodical ability of the machine to identify problems. This finding alone justifies the use of such a tool. MRM cannot replace pharmacists but may help pharmacists make good decisions and avoid missing important problems. 7
The author Gregory Peterson is an investor in Medscope Pty Ltd which developed MRM. The MRM software was based on the work of author Ivan Bindoff. Gregory Peterson was involved with the work of Ivan Bindoff as researcher and supervisor. Peter Tenni, a researcher previously involved with Ivan Bindoff’s work, is currently the manager of the clinical division of Medscope Pty Ltd. 8