A POC clinical information system was designed and deployed on a Motion tablet to allow data collection at the bedside by specialist auditors. Collected data is then stored in a Microsoft SQL Server Environment in real time, allowing immediate feedback through corporate level reporting. The reporting framework consists of self-service non-interactive reports and scorecards, and self-service interactive reports (dashboards) where users can seek detailed information from an organizational level down to a patient level. Information is presented both numerically and graphically, including in a traffic light paradigm, to highlight clinical risks and their relative urgency. Increased information provision to end users via the intranet scorecard and dashboard systems was able to efficiently and effectively close the audit and feedback loop within the organization. Previous variability in information provision was systematically improved to push information to the appropriate decision makers in a timely and efficient manner. Since this health service is one of the first across the country to undergo accreditation under new national standards, the system as described is a novel approach to meeting this compliance challenge. This paper will outline the design features of the system, implementation and training challenges, and proposed future directions for the application and the collected data.
The aim of this work is to improve the provision of quality of care information to healthcare executives, managers and clinicians, in order to target organizational strategies to achieve clinical improvements and increase patient safety. This initiative has been undertaken in the context of new nationally driven quality and safety standards for healthcare in Australia [1]. This aim has been achieved through the use of a new mobile data collection platform linked to tailored business intelligence tools. Business intelligence tools are known to have a place in assisting in the management of facilities and systems in range of industries [2,3] and healthcare is no different [4,5].
Recently the Australian Government, through its healthcare quality agency, the Australian Commission for Quality and Safety in Healthcare, released a series of clinically focused standards ("the standards") that health services need to meet in order to be accredited. Failure to meet the standards can have adverse effects on the financial state and reputation of health services.
Medication safety Patient identity checking Clinical handover Blood product usage Pressure injury prevention Managing deterioration in patients condition and
Falls prevention.
Typically, in order to supplement the often limited range of data available to measure compliance with standards like these, additional ad-hoc audit work has been performed to attempt to measure the necessary dimensions of care. One of the practical and information management challenges imposed by this approach is the need to randomly audit numerous patients for each different kind of audit - pressure care, falls management, drug interactions and so forth. As a result, such audits often need to be conducted in a rolling fashion through multiple clinical areas so as not to create an implementation burden for frontline staff. However this in turn delays feedback to these staff and reduces the frequency of measurement - effectively creating a series of "photographs" in relation to clinical quality and safety, rather than a constantly running "movie".
As mentioned above, a variety of clinical audits have been a long
standing part of clinical practice improvement in healthcare
[6,7,8,9]. Historically, many clinical audit [10,11]
processesdesigned to collect information about an organizations or
individuals compliance with clinical quality standards (eg - care
measures to prevent pressure ulcers)- have been centered around
primary data collection on paper based structured or unstructured
assessment tools. An example is the Surgical Tool for Auditing
records (STAR), used for auditing clinical records in the UK [12].
Historically, data has then typically been secondarily entered into
electronic systems for subsequent analysis, reporting and
benchmarking. Obviously such processes are error prone and far
from ideal from an information management perspective. They
also introduce a potential delay between primary data collection
and subsequent feedback to key staff that can influence practice
and improve outcomes for patients.
There has certainly been much work done around the world on
how to better support the measurement of clinical care - be that
the measurement of the outcomes of care or the processes of care.
Examples include the work of Wong and Giallonardo [13],
Roodpeyma et al [14] and that of Nseir et al [15]. In addition
there has been work done on describing systems for, or
approaches to, storing healthcare data for subsequent analysis and
reuse [16]. This could be at a hospital level [
In addition, as previously mentioned, there had been work done in
the business intelligence space in healthcare [
Finally, the other relevant context here is the use of mobile
devices in healthcare. An extensive literature review published in
2013 showed that there are a wide number of contexts in which
mobile devices and mobile apps are being used in healthcare [
In this work, the worlds of healthcare standards and care
measurement, mobile device usage and business intelligence in
healthcare are considered together through the lens of a specific
case study In this case study, the nursing division of a large
tertiary - quaternary Australian health service conceived of a
modular, electronic audit tool to allow data collection at the
bedside. Data collection at the bedside using a range of devices is
a topic of interest amongst nursing staff given the nature of their
work [
A POC clinical information system was designed and deployed on a Motion tablet (Figure 1) to allow data collection at the bedside by specialist auditors. Collected data was then warehoused in a Microsoft SQL Server Environment in real time, allowing immediate feedback through corporate level reporting. The reporting framework consists of self-service non-interactive reports and scorecards (Microsoft Reporting Services), and selfservice interactive reports (Qlikview) where users can seek detailed information from an organizational level down to a patient level. Information is presented both numerically and graphically, including in a traffic light paradigm, to highlight clinical risks and areas for action. The application was developed using an iterative prototyping approach with a phased implementation, commencing with a pilot amongst the core nursing staff involved in the development process. The modules contained within the application are as follows:
As can be seen from this list, there is an alignment, but not an absolute correlation, between the modules and the list of government standards. This is in part because the desired scope of the POC application was broader than just those topics mandated by the standards. 2 45 The context in which the POC application was developed was characterized by several constraints. Foremost of these was a limited time frame- approximately 6 months from the need being raised to an outcome being expected. In addition, there was little if any, additional funding available to support the work. However, the importance of establishing an electronic tool to support clinical audit data collection and reporting for the purposes of demonstrating compliance with new national healthcare standards cannot be overstated
The POC system (Figure 2) is a Microsoft Excel-VBA application and hence can be run on any platform capable of running Excel with macros enabled. One of the drivers for the use of this platform was the fact that the available programming resource had already demonstrated proof of concept in this space with another, smaller audit application, and the responsible managers were impressed with the results. In addition, it could be seen that development on this platform would enable completion of the project under the immense time pressures surrounding the project, and within existing resource allocation. At project initiation it was always accepted that the application may need to be ported to a more stable long term platform when time allowed. One of the reasons for this is the limitation of the approach to security implicit in this product. Security is not achieved at an application level, but rather at a folder access level. So the Motion devices are only accessible to the small range of trained and approved users (the auditors) and the application file is in a folder that can only be accessed by those trained users. Whilst far from ideal, this was felt to be adequate for the intended use of the application in the short to medium term, particularly given that the application does not allow viewing of, or writing to, the official medical record of the patient.
From a graphical user interface (GUI) perspective, the POC application was specifically designed to be deployed on a Motion Staff undertaking clinical audits may be time poor, and at times overawed by the amount of data to be collected in the audit process. With these issues in mind, the GUI has several specific design features to enhance the usability of the application. These design features include a modular design - at this stage users can select a "full audit" - all 10 modules; or a single audit module from a pick list. Another relevant GUI design feature in this regard is the progressive exposure of questions to users as they proceed down the relevant logic chain. So, for example, within a given module, if the larger number of questions a user may have to answer is 20, they will not be presented with these all at once. Each question will only be displayed on the screen as the last one is answered. This reduces the cognitive load on the user and allows them to focus on accurately answering the question at hand, rather than mentally skipping ahead to answer the next question or the one after that.
As is the case with most good surveys, a consistent feature across all POC modules is a highly structured question format with codification of answers, and with minimal need for, or opportunity to enter, free text answers. Data entry is supported by stylus with a rapid system response time.
Patient identity validation is undertaken automatically by the application, by checking the entered patient identification number against the organizational data warehouse. The patient‟s gender and date of birth are then retrieved so the user can ensure they are auditing the correct patient. As well as ensuring that the collected data will be linkable with other data about the patient, this reduces the risk of incorrect patient details being collected as can be the case with traditional primary data collection on paper. Very significantly, the POC application was designed from the outset to be connected to what is known as the organizational "information grid". This grid- analogous to the electricity grid, where data is equivalent to electricity- is designed to allow the reliable and predictable flow of data and information to users irrespective of how or where it is generated. The aforementioned data warehouse is one of the key components that has been constructed inside “the grid”.
The services available to the business from “the grid” include data extraction and distribution, non-interactive (static) reporting through Microsoft Reporting Services, and interactive (dynamic) reporting through Qlikview. It is these latter 2 services that the POC application leverages off to provide feedback to stakeholders regarding collected data. 3.4
There is a growing awareness of the utility of mobile devices and
applications deployed on them across all of healthcare [
It is also important to note however, that there are a number of
potential issues in relation to deploying and using healthcare
applications on mobile devices, be they smart phones, handheld
tablets or larger and heavier tablets like the Motion tablet. These
include the need for a medical grade tablet device choice at the
bedside versus more popular choices such as the iPad. In a
bedside setting it is critical that end user devices are physically
robust but that they can also withstand trauma and are able to be
wiped down to prevent the spread of infection. Some pieces of
computing equipment are certainly known to harbor infectious
organisms [
The POC system has had a strong history of use since its release in April 2013.
Since the commencement of the POC audit program, in excess of 635 unique patients have been randomly audited in a 4 month period across 3 campuses of an academic health service. This number will grow substantially over time. This has been across 21, 12 and 4 wards respectively at the 3 main campuses of the health service-these being the main acute campus, the aged and sub-acute care campus, and the community based hospital campus.
The interactive report (Figure 3) to which the data contributes has had 675 views in 4 months – equating to about 42 views per week.
The longest standing non–interactive report (Figure 4) to which the data contributes has had 1382 views in 4 months – equating to about 80 views per week. Below is an example of the noninteractive report showing the summary traffic lights across the entire organization for all measures captured in the POC application.
Both kinds of reports can be viewed on a variety of devices, as they are intranet based, including on the Motion tablets themselves. In the case of the non- interactive reports, these are updated as soon as the data is submitted from the POC application.
Stakeholder acceptance of the solution - the POC application, the Motion tablets and the reporting options - has on the whole been excellent. One of the reasons for this, on the management front at least, is that the solution replaces a mixture of manually executed processes, with a large administrative burden, that resulted in incomplete and insufficient detailed data stored in disparate Excel spreadsheets.
Instead, the management staff responsible for running the clinical audit program, and those charged with driving towards improved outcomes off the back of it, now have more complete and robust data that can be fed back to relevant staff in reports almost instantaneously. In addition that data is stored in a centrally supported, robust technical environment and can be kept for as long as needed and re-used, along with other relevant data from "the grid" for a range of research and evaluation activities in addition to its core and immediate purpose.
Some of the direct feedback obtained from stakeholders has been very encouraging. For example one of the nurse managers stated “It is great that we can see the data instantaneously and can use it to inform our practice”. The auditors, who were nurse educators, made positive statements such as “(the) ward appreciate receiving the feedback as we complete the audit and can address issues at the time at a local level”. Another auditor saw the solution and its use as “A great educational opportunity. We are getting to know what the wards learning needs are by having the ability to look at the data as it is entered”.
In relation to the Motion tablets themselves, some comments
included that they were "user friendly" and "easy to use". Others
felt that it was "great not to have to double handle data!" when
compared to the old method of primary data recording on paper
then transcribing into an electronic source. Certainly the broader
literature has many examples of tablet devices being well received
amongst users. For example, work from the US has examined the
positive benefits of distracting children with iPads as an
alternative to sedation or anesthesia for medical procedures [
Increased information provision to end users via the intranet scorecard and dashboard systems was able to efficiently and effectively close the audit and feedback loop within the organization. Previous variability in information provision was systematically improved to push information to the appropriate decision makers in a timely and efficient manner. Since this health service is one of the first across the country to undergo accreditation under the new national standards, the system as described is a novel approach to meeting this compliance challenge.
There were a number of specific lessons learned from the development and implementation activities in this project.
One of the key lessons in relation to the project and its relative success was the nature of the requirements elicitation process. Even though the software development approach could be best described as iterative prototyping, the many dozens of questions across all audit modules, and their complex conditional logic, were captured and kept up to date in a semi- formal requirements document. This document became critical in maintaining good communication and clarity between the development team and nursing subject matter experts (SME's).
The other critical aspect of this part of the project was the very direct and regular access that the developer had to the SMEs both on terms of explanations about requirements and feedback on released development versions of the software.
These 2 things combined to ensure a stable, well tested application at release that needed little post release work.
Despite the measures outlined above, released software can still have bugs, even when developed with an abundance of resources applied to the process. The key software issue discovered in the case of the POC application was the identity checking function. In short the function appeared to work well when checking against test databases and in small scale production use. However when greater production use began there were cases where known inpatients could not have their details checked and hence could not have their care audited as the application was designed with this check as a "gate keeping" step.
An investigation identified the problem which was remedied by having the check look at more than one underlying database for auditable patients. Once this was done the bug was resolved. Ironically this was not due to a fault with the application itself but with an inherited architectural decision in the underlying databases.
Another unforeseen issue was that of wireless coverage in ward areas. This was a problem especially in the main campus of the health service. In short, multiple staff using the application in multiple locations highlighted the fact that wireless coverage was somewhat patchy. Users soon found ways around this, but the problem expedited an investigation by the IT department of wireless coverage across the main building.
Some of the issues for consideration in relation to the future of the POC application and its use include porting the application to a more robust software platform, and potentially expanding the range of devices on which the application is deployed (eg Windows based phones or smaller, handheld tablets), and increasing the range of usage of the collected data. It is also highly likely that the business will drive changes in the core application - for instance an 11th module relating to audit of timely patient access to care is already under development. The health service has an IT architecture team and that team has begun investigating a more robust software platform on which the same data collection functionality could be delivered. It is highly likely, that in line with the standard approach to patient centric data collection used in the institution, the Cerner Millennium Clinical system, will serve as that platform. A key challenge will be the extent to which the Cerner platform can support some of the GUI features of the current POC application, For example, the progressive “unhiding” of questions as users navigate through the application. There is no question however, that deployment on this platform would allow great levels of security around the use of the application and, critically, direct integration with the remainder of an individual‟s clinical record.
There are also opportunities off the back of this work to further examine the use of mobile devices across our institution, and how that usage compares to best practice from around the world. This is of interest from both a heath and IT research perspective, and an operational perspective. Such opportunities are also important in shaping future “bring your own device” (BYOD) strategies for the organization, particularly given the part time (senior staff) and or transient (junior staff) nature of much of its medical workforce.
What will be critical, in terms of assessing the long term impact of
the POC application, is to see the effect of the tool and the
feedback loop to clinicians and managers, in terms of reducing
bad practice and maximizing good practice. So for example in
performing falls risk assessments, or in reducing harder end points
like hospital acquired pressure ulcers. An example of such an
evaluation is the previous quoted work by Tuffaha et al [12].
Let us consider a specific example of how the collected data has
been, and will continue to be used to improve patient care. In
Figure 5 below, the percentage of audited patients having had a
falls risk assessment upon admission is tracked from pre-POC
implementation (February) thorough to the time of writing (July).
In that time that percentage has risen from 42% to 76%.
Now lets us consider Figure 6 below. In this graph we can see
that the percentage rate of audited patients having had a falls plan
documented within 24 hours of admission has risen from 42% to
63% over the same time period. These trends may partially be
explained by an increasing sample base over time, but also will to
some extent reflect efforts, based on the early results of the audit,
to raise the rate of both of these key markers of good nursing care.
It is critical to note that such trend data, which can also be
examined by sub-group (ward, campus) was previously not
readily available, and certainly not in a timely fashion with
automated reporting. Hence clinicians, managers and executives
have been “flying blind” to some extent in relation to how well
their clinical workforce has been performing in regards to these 2
key pieces of documentation. Given the known problems relating
to falls in hospitals all over the world [
Furthermore, use of grid data will enable interesting correlations between, for example, shift by shift staffing, or hospital occupancy; and the impact on the processes and outcomes of care as they relate to individuals on various hospital wards. Work like this would be of a particularly novel nature.
This paper outlines how a novel POC software application deployed on a mobile device has been able to successfully supplant much paper based auditing practice resulting in strong uptake and positive feedback from users and other key stakeholders. The coupling of this application – albeit that it is likely to be a temporary deployment vehicle - with a robust corporate approach to information management, has paved the way for greater amounts of data collection, and greater data quality in this space, and in turn improved outcomes for patients. Another essential benefit has been an increase in the ability of the health service to demonstrate its compliance with new national healthcare standards.
The 3 main authors would like to acknowledge the efforts of our other key contributors – Suzanne Metcalf and Pam Ingram from Nursing Management, and Lucy Nie from the Health Informatics department, for their key roles in bringing the system to life. In addition, all the authors would like to acknowledge the work of Dr. Roger Hawkins – Manager, Analytics in the Clinical Performance Unit of the health service for his original work on the POC software application. Finally, we would like to acknowledge Mr Zaf Alam and Mr Hien Le from the Health Informatics and IT departments respectively for their work in establishing the reporting around the POC data.
[1] “NSQHS Website”. http://www.safetyandquality.gov.au/ourwork/accreditation/ Accessed 18/6/2013. [2] F. Mohammadi and N. Hajiheydari. “How Business Intelligence Capabilties Contributed Managerial Decision Making Styles”. International Journal of e-Education, e-Business, eManagement and e-Learning. 2012 Vol 2, No 1; February pp 2833. [3] J.K. Ho. “Strategic Innovation through Business Intelligence: Linking Competitive Forces to Profitabilty” International Journal of e-Education, e-Business, e-Management and e-Learning. 2012 Vol 2, No 4; August pp 280-285. [4] I. Cattinelli, E. Bolzoni, M. Chermisi, F. Bellocchio, C. Barbieri, F. Mari, C. Amato, M. Menzer and E. Gatti. “Computational Intelligence for the Balanced Scorecard; Studying performance trends of Haemodialysis Clinics” Artif Intell Med. 2013 Jun 11. pii: S0933-3657(13) [5] J. Van De Graaff and A.Cameron.“Quest for Business Intelligence in Healthcare”. Healthc Financ Manage. 2013 Feb;67(2):44-6, 48. [6] R.A. Ralph and M. Delaney. “Dual purpose cataract audit: improvement of care, improvement of documentation.” QRB Qual Rev Bull. 1979 May;5(5):29-31. [7] J.D. Vail. “Retrospective anesthesia nursing audit-process or outcome criteria?” Mil Med. 1979 Aug;144(8):537-9. [8] D.K. Dhariwal and A.J. Gibbons. “The CRABEL Score – Setting standards in Maxillofacial Medical Note Keeping”. Br J Oral Maxillofac Surg. 2004 Jun;42(3):200-2. [9] M. Chopra, S. Stirling, D. Wilkinson, C. Connolly and D.McCoy. “Paediatric admissions to a rural South African hospital: Value of hospital data in helping to define intervention priorities and allocate district resources”. S Afr Med J. 1998;88(6 Suppl):785-8. [10] S. Gilam and A.N. Siriwardena. “Frameworks for Improvement: Clinical Audit, the Plan-Do-Study-Act Cycle and Significant Event Audit”. Qual Prim Care. 2013 May;21(2):12330. [11] J. Barlow and G. Krassas. “Improving management of Type 2 Diabetes-Findings of the Type2 Care Clinical Audit”. Aust Fam Physician. 2013 Jan-Feb;42(1-2):57-60. [12] H. Tuffaha, T. Amer, P. Jayia, C. Bicknell, N. Rajaretnam and P. Ziprin.“The STAR score: a method for auditing clinical records”. Ann R Coll Surg Engl. 2012 May;94(4):235-9. [13] C. Wong and L. Giallonardo. “Authentic leadership and nurse-assessed adverse patient outcomes”.J Nurs Manag. 2013 Jul;21(5):740-52. doi: 10.1111/jonm.12075. [14] S. Roodpeyma, M. Hekmat, M. Dordkhar, S. Rafieyian and A. Hashemi. “A prospective observational study of paediatric cardiac surgery outcomes in a postoperative intensive care unit in Iran”. J Pak Med Assoc. 2013 Jan;63(1):55-9. [15] W. Nseir, S. Haj, B. Beshara, J. Mograbi and O. Cohen. “Seeking out high risk population: the prevalence characteristics and outcome of diabetic patients of arab ethnicity hospitalized in internal medical and acute coronary units in Israel”. Int J Endocrinol. 2013;2013:371608. doi: 10.1155/2013/371608. Epub 2013 Jun 18. [16] P. Embi, S. Kaufman and P. Payne. “Enabling KnowledgeDriven Health Care”. Circulation. 2009; 120: 2393-2399 7 50