Clinical Proof-of-Concept – A Evaluation Method for Pervasive Healthcare Systems Jakob E. Bardram IT University of Copenhagen Rued Langgaards Vej 7, DK-2300 Copenhagen, Denmark bardram@itu.dk ABSTRACT care research is specifically targeted towards technology – Pervasive Healthcare – i.e. designing pervasive computing i.e. aiming at understanding, designing, building, and testing technologies for healthcare usage – is an especially promis- new types of pervasive computing technologies for health- ing area within pervasive and ubiquitous computing re- care purposes. search. However, it is extremely difficult to evaluate such systems because establishing clinical evidence for medical A common methodological approach to ubiquitous comput- benefits would require longitudinal, randomized, double- ing research is to design and implement a technical ‘Proof- blind, placebo-controlled trials involving a homogeneous of-Concect’ for a proposed new ubiquitous computing tech- patient population and medical condition. This would not nology or application, and subsequently evaluate this imple- only require huge resources in terms of clinical staff and pa- mentation in a limited setup. Marc Weiser defined the con- tient participation, but would also require the technology to cept of a technical Proof-of-Concept as: be fully developed and ready for large scale use. The lat- ter is simply not feasible when doing technological research The construction of working prototypes of the neces- into new types of pervasive healthcare technologies. In this sary infrastructure in sufficient quality to debug the vi- paper, I suggest the method of ‘Clinical Proof-of-Concept’ ability of the system in daily use; ourselves and a few as a method for evaluating pervasive healthcare technologies colleagues serving as guinea pigs. [8]. in order to establish the clinical feasibility of the technology before entering large-scale clinical trials. The method has Looking at the research questions posed by pervasive health- been applied in a couple of cases and I report on lessons care, this research approach seems to be lacking some rigor learned from this. in order to investigate whether the technology solve health related challenges. We would, for example, never be able to understand or evaluate to which degree a technical prototype INTRODUCTION for elderly people would be successful, if it is only tried out Applying Ubiquitous and Pervasive Computing technologies by our colleagues in a research laboratory. for healthcare purposes is gaining increasing interest and is growing into a research field of its own called ‘Pervasive From a medical perspective a technical proof-of-concept is Healthcare’ [3, 1]. The research questions and the technolo- not acceptable for introducing new medical technology or gies being investigated within Pervasive Healthcare are quite treatment. In most healthcare systems, clear clinical evi- diversified ranging from using biomedical sensor technology dence needs to exist before a new medical technology is put for patient monitoring and prophylactic treatment, to mo- into use for patient treatment. Evidence-based medicine [7] bile and context-aware systems inside hospitals. Pervasive is the clinical methodological approach for establishing this Healthcare has a lot in common with established medico- evidence. Evidence-based medicine categorizes different technical research areas like biomedical engineering, medi- types of clinical evidence and ranks them according to the cal informatics, and telemedicine, but is distinct in its funda- strength of their freedom from the various biases that beset mental approach and goals; pervasive healthcare systems are medical research. The strongest evidence for therapeutic in- often designed for patients and not for clinicians, and they terventions is provided by systematic review of randomized, embody technologies growing out of the ubiquitous com- double-blind, placebo-controlled trials involving a homoge- puting research, including sensor technology, context-aware neous patient population and medical condition. In contrast, and mobile computing, large interactive displays, etc. Sim- patient testimonials, case reports, and even expert opinion ilar to Ubiquitous Computing research, Pervasive Health- have little value as proof because of the placebo effect, the biases inherent in observation and reporting of cases, and difficulties in ascertaining who is an expert. Copyright is held by the authors. Permission to make digital or hard copies Such strong evidence is, however, impossible to obtain while of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial we are still in the research and development phase of new advantage and that copies bear this notice and the full citation on the first technology. So an important question is how we can strike a page. balance between these two extremes; design and implement UbiComp ’08 Workshop W2 – Ubiquitous Systems Evaluation (USE ’08), limited technical proof-of-concepts which at the same time September 21st, 2008, Seoul, Korea. This position paper is not an official publication of UbiComp ’08. are suited to provide sufficient clinical evidence for further sufficient for establishing the viability of the technical setup research and development and its use in a real-world deployment. And the trial may run for a couple of weeks – rather than the months normally In this paper, I suggest a methodological approach called required for a clinical trial. ‘Clinical Proof-of-Concept’ which is aimed at creating ini- tial clinical evidence for the medical benefits of a pervasive The methods used during this CPoC should be targeted at healthcare technology. This approach has been used in a collecting evidence, which demonstrate that the technology couple of cases and I will report on these cases, how the clin- seems promising in addressing its specific goal. It may be ical proof-of-concept was carried out, and what we learned relevant to gather initial clinical evidence for the medical from these cases. benefit of the technology. For this purpose, trying to measure some clinical effects is essential during the CPoC. For exam- The contributions of this paper is the presentation of the ple, in order to establish any clinical effect in the monitoring methodological approach of a Clinical Proof-of-Concept to- of hypertension, blood pressure data may be compared over gether with specific examples of its use in two cases. By the time span of the CPoC and questionnaires regarding the suggesting this approach, it is my aspiration that more per- patients’ awareness and handling of their blood pressure may vasive healthcare technologies can be subject to initial eval- be issued and analyzed. uation and scrutiny before entering large scale clinical trials, while at the same time actually being put into test in a lim- Even though the clinical evidence may be biased by differ- ited real-world deployment. Using this approach, a more in- ent factors and hence not as strong as would be required cremental and experimental approach to the construction of in Evidence-Based Medicine, providing initial clinical evi- pervasive healthcare technologies can be achieved, which in dence for the working of the technology is still essential in the end will lead to developing more appropriate and usable order to justify further development and evaluation. Further- pervasive healthcare technologies. At the same time, the ap- more, the Clinical Proof-of-Concept may simultaneously proach would enable us to reject and dismiss those technolo- work as a ‘dry-run’ for testing the data collection methods, gies, which show little clinical promises before large amount which later are to be used during the clinical trial. For exam- of resources are spent on developing the technology and run- ple, if a questionnaire is handed out to the participants, this ning clinical trials. questionnaire and the timing of it may be subject to change based on experiences obtained during the clinical proof-of- concept. CLINICAL PROOF-OF-CONCEPT To rephrase the definition from Marc Weiser, I am defining Apart from establishing initial clinical evidence, a core pur- a Clinical Proof-of-Concept (CPoC) as: pose of a CPoC is to investigate the usefulness and usability of the proposed solution. To a certain degree, I would ar- The construction of working prototypes of the neces- gue that this is the main purpose of a CPoC for two reasons. sary functionality and infrastructure in sufficient qual- Firstly, the clinical benefit of a pervasive healthcare technol- ity to investigate evidence for improving health in daily ogy may be significantly diluted if the technology is hard use for a suitable period of time; a limited but relevant to use for the patient. For example, it is obvious that if the set of people serving as subjects. blood monitoring technology is hard to use, then limited ef- fect on hypertension management may be found during the More specifically, the technology should be a working proto- clinical trial. Secondly, it is essential to catch and remedy type that is usable (but not necessarily user-friendly), works such usability problems in an early phase before resources on its own, and is focused on addressing specific research are invested in developing the technology, producing it in questions. This technology should be deployed in a real large numbers, and deploying it for a clinical trial. clinical setup, should be used by real users (researchers are hands-off), for a short, but sufficient period of time, which – These arguments may seem trivial. However, some sort of depending on the research question – may range from 1 day usability problems are often hard to discover and are often to 3 months. unexpected. By running a CPoC which actually puts the technology to a test in a real-world setting with real users For example, you may want to test a system for monitoring for a certain period of time, many of the more complex us- hypertension and evaluate if users are able to control their ability problem may surface. And often, ideas for changing own blood pressure over time, thereby reducing hyperten- and improving on the technology arise when seeing it in ac- sion, which again – according to medical literature – have a tual use and by working closely together with the users to positive effect on a wide range of heart diseases. In this case, find a solution to the problem. a CPoC would involve a technical prototype which runs on its own and is able to monitor blood pressure, but it may Methods for usability inspection would typically be quali- not be particular secure, robust, or integrated in a country- tative in nature, involving observations, questionnaires, and specific healthcare system. It should, however, be able to run studies of perceived usefulness and usability. with limited interference from the researchers, while some ‘Wizard-of-Oz’ techniques may be applied. The deployment Figure 4 show the temporal progression of research meth- would include a limited amount of people – e.g. 10 – which ods as the technology is developed and mature. Time-wise, is not statistically significant for hard medical evidence, but Figure 1. The timing of a Clinical Proof-of-Concept is between a laboratory proof-of-concept and a full clinical trial. Figure 2. A patient using the home-based monitoring system in a briefcase for monitoring her blood pressure. a clinical proof-of-concept lies in between the technical lab- oratory proof-of-concept and a full-scale clinical trial. was focusing on studying issues of medical treatment, di- CASES vision of work, communication, patient self-understanding, and the technology in actual use [2]. Our study – in ac- In order to illustrate how a CPoC can be used in pervasive cordance with most medical studies of home-based mon- healthcare research, I will use two cases as examples. The itoring of hypertension – gave evidence that this kind of first case is concerned with home-base blood pressure mon- blood pressure monitoring provides more accurate measure- itoring, and the second case is concerned with developing ments. Our findings, however, also revealed that the relation- context-aware technologies for improving patient safety in- ship between the GP and the patient changed when this new side the operating room. These two case are very distinct in computer-mediated home-based treatment for hypertension many respects – technology, users, deployment settings, and was introduced. More specifically, we found four specific goals – but as such, they illustrate the breath of the CPoC aspects of this transformation caused by pervasive monitor- approach. ing and treatment technology: Blood Pressure Monitoring • A new division of work emerged, which transferred the The first project was concerned with home-based monitoring act of monitoring and interpreting the blood pressure data of hypertensive patients. Hypertension is a direct cause of a from the GP to the patient. number of heart diseases, including congestive heart failure and stroke, and substantial clinical evidence indicates, that • The medical treatment of hypertension and the life quality frequent blood pressure monitoring helps prevent hyperten- of the patient was improved. However, new demands for sion [6]. For this reason, many pervasive healthcare projects monitoring the incoming data and the patient’s progres- have addressed hypertension. This project was done in 2002 sion in treatment were inflicted upon the GP. when state-of-the-art blood pressure monitoring was based • The communication pattern between the patient and GP on a cuff. Our goal was to deploy the technology in a lim- was fundamentally changed from a contextual rich con- ited pilot study and perform a CPoC (even though we did versation to an asynchronous message exchange. not call it that at that time). The technology for home-based monitoring consisted of a suitcase with a traditional blood • Because the patient was more involved in the monitor- pressure monitor, a PDA, and a GSM modem as shown in ing and treatment of hypertension, he or she became more Figure 2. self-aware on the nature of high blood pressure and what affects it. In this project, the suitcase were given to the patients by their general practitioner (GP). The system had three main fea- This clinical CPoC was insufficient to establish clinical ev- tures: (i) it allowed the patient to measure the blood pressure idence for improved hypertension treatment of the patients. several times a day and this data was sent to a central server For this purpose, the time frame of the study was too short, for the GP to observe; (ii) the GP could prescribe medicines the sort of methods applied insufficient for determining clin- and the patient could indicate that (s)he was complying to ical evidence, the number of patients were too small, and no the prescription; and (iii) it enabled communication between control group was involved. The study, however, were suf- the patient and the GP, using both text and voice messaging. ficiently large to study, understand, and argue that this kind of home-based monitoring would transform the patient-GP During the first months of a longer deployment period, we relationship and make the patient capable of managing their carried out a series of interviews and field studies of this own blood pressure in a more efficient way. And since pre- home-based monitoring and treatment system. This study vious clinical studies have shown that regular self-conscious attention to your blood pressure reduces the risk of hyper- tension, this was clearly a strong indicator that this kind of technology would be useful. At the same time, the CPoC revealed a series of usability and deployment issues which needed to be looked into be- fore running larger scale trials involving a larger amount of patients and GPs. The technology were subsequently im- proved and deployed in a large clinical trial with 10 GPs, 120 patients, and a control group. Context-aware Patient Safety The Context-aware Patient Safety and Information System (CAPSIS) monitors what is going on inside the operating room and use this information to show timely medical data to the clinicians, and to issue warnings if any safety issues are detected [4]. CAPSIS monitors events like the status of Figure 3. The deployment of the system inside the OR; the operation; the status and location of the patient; loca- the surgeon and the sterile nurse read medical data on tion of the clinicians who are part of the operating team; the screen to the right while the scrub nurse interacts and equipment, medication, and blood bags being used in- with the patient safety system on the screen to the left. side the operating room. This information is acquired and handled by a context awareness infrastructure, and a special sions. By actually deploying the technology inside the OR, safety service is used for overall reasoning which actions to and asking the operating team to use the technology during take or warnings to issue. The goal is to supplement human close to real-world surgeries, a wide range of issues surfaced safety vigilance with a machine reasoning counterpart. which would have not been found otherwise. Especially is- sues regarding the physical working environment of an OR CAPSIS was deployed and tested in a CPoC where it was and the tight teamwork taking place during surgery surfaced. used for one day by a full surgical team performing simu- Some examples of issues that were discovered during this lated operations inside an operating room. In total, 8 oper- CPoC include; ations were executed during the day, involving both opera- tions with no warnings as well as different types of warn- • The user interface had to be improved in several place, ings, including wrong patient, wrong operating table, wrong including issues like coloring, highlighting, and font size blood, and incomplete team. In addition, medical records, on the screen due to the distance from the operating table radiology images, and the operation checklist were pre- to the screens. sented on displays using the context-aware triggers. A pic- ture from the CPoC is shown in Figure 3. Everything were • The procedures regarding attaching a RFID enabled arm- done exactly as real surgeries, except that no real patients band to the patient needed to be scrutinized because pa- were involved and the acting patients were not sedated or ac- tient safety now was depending on that this was done cor- tually cut. The acting patients, were, however treated as any rectly. If the wrong armband was attached to a patient, real patient, including being admitted to ambulatory surgery unpredictable and potential severe safety hazards may oc- and scheduled in the scheduling system. cur. The goal was to provide objective measurements on the use- • Better support for handling and registering scanning of fulness and usability of our design while, at the same time, blood bags were needed. When moving from a limited test investigate the detailed user reaction to the system and the in a lab to a CPoC in the OR where a substantial volume user interface in a more qualitative fashion. For this purpose, of blood may be needed, the existing method for checking we used a multi-method evaluation setup where we (i) asked correct blood did not scale to e.g. 10-20 blood bags. The the users to perform the operations while thinking aloud, (ii) reason for this was due to a number of highly interlinked investigated perceived usefulness and usability based on a aspects, ranging from the organizational procedure for or- questionnaire [5], and (iii) made a semi-structured follow- dering and getting blood, to the physical layout of the OR, up interview. and the way the RFID technology were working. • Lack of triangulation, which is the medical safety term Based on this evaluation, the clinicians concluded that the for ensuring that a safety check is done by combining the system would be very useful for ensuring patient safety and patient, the procedure, and the clinical staff. Even though was very easy to use. Most of the patient safety issues mon- this was part of the overall systems design, triangulation itored by CAPSIS were found to improve patient safety, and did not work inside the OR on the individual level. several of the findings resonate with the recommendations from the state-of-the-art regarding patient safety. Moreover, • The operating team had to change their safety procedures the CPoC revealed a series of usability issues which we just before surgery in order to leverage the capabilities of had not captured previously, despite several prototyping ses- the system. It is important to note, however, that this CPoC setup is during surgery. Hence, a new methodological setup is re- not providing ‘hard clinical evidence’ for improved patient quired in any subsequent clinical trial. safety inside the OR. We do not know if this system – if build and deployed – would improve on patient safety. This Third, because the technology is deployed in a real setting would require a randomized clinical trial over a longer pe- for a non-trivial period of time, a CPoC is well-suited for riod of time involving a control group, which again would investigating the usability of a pervasive healthcare system. require a full working system ready for large-scale and long- Especially non-trivial usability problems which arise from time deployment. Providing such Evidence-Based Medicine complex interaction between different types of technologies, is, however, not the purpose of a Clinical Proof-of-Concept; users, real deployment settings, and long-term use may be it is rather to investigate the feasibility of the proposed solu- discovered during a CPoC. For example, the blood-pressure tion for further development. By asking the involved clini- monitoring CPoC revealed that it was hard for some patients cians how they perceive the system’s potentials for improv- to type a message to the GP and this functionality was hence ing patient safety, we are given sound indications regarding changed to use voice messages instead of text messages. the feasibility and directions for further development. And in the patient safety project, as wide range of usabil- ity issues regarding the user interface were found. Beside this indication of potential clinical evidence for im- proving patient safety, the core benefit from running this Fourth, due to the real-world deployment a CPoC helps re- CPoC is the different problematic issues regarding the cur- veal and evaluate the physical usage of the technology. The rent prototype which must be addressed before making a physical aspects of the technology is especially important for larger clinical trial. As illustrated above, these issues are a pervasive healthcare systems since medical devices and sys- mixture of technical, usability, physical, and team-oriented tems are notoriously tied to monitoring or influencing phys- aspects which need to be addressed in concert. But most im- ical properties of a human body or a physical environment portantly, these complex and interrelated issues would prob- in homes or hospitals. For example, a wide range of is- ably never have been found without running a CPoC. The sues regarding the physical handling of the blood-pressure next step would be to incorporate the suggestions for im- cuff and the handling of the PDA were revealed during the provement and then apply more rigorous clinical methods CPoC. This subsequently lead to the design of a cartoon-like for evaluating the degree to which the system improves on step-by-step instruction card, which were place on the front patient safety. Note, however, that the only reason for such of the suitcase, as shown in figure 2. In the patient safety an investment is based on the fact, that the CPoC indicated project, the physical layout of the OR, the physical handling that the system potentially would improve patient safety. of patients, blood bags, and instruments turned out to have significant impact on the use of the system. DISCUSSION Finally, often pervasive healthcare systems needs to exist What have we learned from our use of clinical proof-of- and work in a larger social and organizational context. A concepts so far? CPoC is equally suited for initial investigation of the impact arising from this larger deployment context. Especially in First of all, a CPoC reveals a wide range of technological the blood-presure project we found a significant change in problems and issues. For example, in the blood pressure the division of work and communication between the GP and monitoring project, the CPoC revealed all sorts of problems the patient, and the CPoC revealed some of the important de- with wires, handling software updates, and sustaining power tails of how the technology would potentially influence the on the devices while not in use. In the patient safety project, way the treatment of hypertension were achieved. In the pa- the CPoC revealed all sort of issues ranging from the work- tient safety project, the CPoC helped judge the fit between ing of the RFID technology to the use of the software on the system and the complex and dynamic teamwork taking large touch-screens. Hence, a CPoC is useful in determin- place inside an OR. ing the sort of technological issues which are related to real- world use by real users for a longer period of time and on a larger deployment scale. CONCLUSION In this paper, I have proposed to apply a Clinical Proof-of- Second, even though a CPoC seldom is done in a way which Concept as a methodological approach for evaluating perva- justify any ‘hard’ clinical evidence, it is still useful in or- sive healthcare systems. A CPOC involves a focused study der to both provide initial clinical evidence for a potential in a real-world deployment setting, involving real patients medical effect, as well as providing important information and users, while being limited in time, scope, and clinical on how this clinical effect should be collected. For exam- rigorousness in the methods applied. By being a stepping ple, the clinicians in the patient safety project unanimously stone in the middle of a laboratory-based evaluation and a agreed that the system had the potential for improving pa- full-scale clinical trial, the CPoC is able to provide valuable tient safety inside the OR. This do not count as clinical ev- information regarding the clinical applicability of the sys- idence, but nevertheless it encourage further development. tem, its usability, and issues regarding the physical and orga- At the same time the CPoC revealed that the methods used nizational deployment of the system. In this way a CPoC is a for evaluating the system were appropriate for judging per- more dedicated and cost-effective approach for establishing ceived usefulness, but they were not appropriate for provid- initial clinical evidence as well as being a invaluable source ing clinical evidence for the improvement of patient safety for improving the technology at a stage before resources are invested in final development and clinical trials. Acknowledgments The hypertension project was done together with Anders Thomsen and Claus Bossen. The context-aware safety sys- tem for operating rooms were done together with Niels Nørgaard and the surgical staff at Horsens Sygehus in Den- mark. REFERENCES 1. J. E. Bardram. Pervasive healthcare as a scientific dicipline. Methods of Information in Medicine, 3(47):129–142, 2008. 2. J. E. Bardram, C. Bossen, and A. Thomsen. Designing for transformations in collaboration: a study of the deployment of homecare technology. In GROUP ’05: Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work, pages 294–303, New York, NY, USA, 2005. ACM Press. 3. J. E. Bardram, A. Mihailidis, and D. Wan, editors. Pervasive Healthcare: Research and Applications of Pervasive Computing in Healthcare. CRC Press, 2006. 4. J. E. Bardram and N. Nørskov. Designing Context-aware Safety Systems for the Operating Room. In Proceedings of Ubicomp 2008: Ubiquitous Computing, Seoul, Korea, Sept. 2008. 5. F. D. Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3):319–339, September 1989. 6. M. A. M. Rogers, D. A. Buchan, D. Small, C. M. Stewart, and B. E.Krenzer. Telemedicine improves diagnosis of essential hypertension compared with usual care. Journal of Telemedicine and Telecare, 8:344–349, 2002. 7. D. L. Sackett, W. M. Rosenberg, J. A. Gray, R. B. Haynes, and W. S. Richardson. Evidence based medicine: what it is and what it isn’t. BMJ, 312(7023):71–2, 1996. 8. M. Weiser. Some computer science issues in ubiquitous computing. Communications of the ACM, 36(7):75–84, 1993.