The aim of this paper is to introduce a smart mHealth application based on the augmented reality (AR)-paradigm that can support patients with common problems, related to management of their medication. This smart mHealth application is designed and implemented as a medication coach intelligent agent, called Medication Coach Intelligent Agent (MCIA). The MCIA will have to manage different types of information such as the medication plan (medication regime) of the patients, medication restrictions, as well as the patient's preferences and sensor input data from an AR-headset. Considering all this information, the MCIA leads with holistic decisions in order to offer personalized and unobtrusive interventions, in an autonomous way, to the patients. From a long term perspective, the MCIA should also evaluate its performance over time and adapt in order to improve its interventions with the patients. To show the feasibility of our approach, a proof-of-concept prototype was implemented and evaluated. The results show a high potential for using the MCIA in real settings.
Medication adherence is a global problem [
Several attempts have been made in order to address this problem such as different mHealth-applications3 and different types of robots. Even though some interesting robots are on its way such as Pillo 4, robots have by nature, the limitation of being more or less fixed in its location. ? Contact author. 3 Practice of medicine and public health using mobile devices, apps, smart devices and smartphones 4 https://www.pillohealth.com
Some of the critiques towards current mHealth- applications is that they lack several
basic adherence attributes [
Non-adherence can be either intentional or unintentional [
Against this background, this paper introduces a novel solution to lead with the medication adherence problem based on the AR-paradigm and intelligent coaching systems. In particular, we introduce the so-called Medication Coach Intelligent Agent (MCIA). The MCIA has proactive and reactive behavior in order to support the medical management of patients. Moreover, the MCIA has autonomous reasoning capabilities that allow the MCIA to lead with long-term goals in the settings of medication plans. As part of the results of this paper, an architecture of the MCIA is introduced. This architecture aims for a technologically scalable solution based on an AR-headset and multi-agent systems. We also present a usability evaluation of a proof-of-concept prototype of the MCIA.
The rest of the paper is organized as follows. In Section 2, different issues regarding medication management are discussed. In Section 3, a theoretical framework regarding the MCIA is presented. In Section 4, an implementation of the MCIA in the settings of the Microsoft HoloLens is presented. In Section 5, an evaluation of our proof-of-concept prototype is described. In Section 6, a short review of the related work is presented. In the last section, conclusions and future work are outlined. 2
The research in this project was developed as nurses from home healthcare brought
attention to several patients having problems maintaining medication adherence through
self management. Self management includes strategies and activities a person performs
to live well with illness and it can be performed by the individual or in
collaboration with a significant other [
The target group for the research in this project are patients from groups 2 and 3. The purpose is to investigate if AR-technology (using an AR-headset) may be used as a tool to increase their ability to improve and maintain medicine-related self management, thereby contributing to them staying independent for a longer time, delaying need for home healthcare and facilitate medication adherence. 2.2
Many patients have several different medicines, as a strategy to simplify handling pills they use pill dispensers, where the medicine is distributed on a weekly basis. A common problem for the target group, and a reason why many patients need help managing their medication, is the continuous variation regarding their medicines, names of medicines and the visual appearance of pills and packages. This leads to patients and their nonprofessional helpers being confused when handling medication and preparing pill dispensers, which in turn leads to needing a nurse to come to their homes on a regular basis, preparing the dispenser for them.
What actually causes the variation is that the pharmacies can deliver different brands for the same type of medicine. The names and boxes varies with the brand and this makes the patients insecure and afraid of preparing their dispensers. The underlying reason for the frequent exchange of medicine brands are the rules for interchangeable medicines which are applied in most European countries. According to the rules, if a patient gets prescription for a particular medicine, the pharmacy always has to offer the brand with the lowest price if the medicines are interchangeable6.
One of the main aims of the MCIA is to make patients and their helpers confident enough, using an AR-headset, to prepare their dispensers and therefore remove or delay the need of home healthcare (nurse). Another priority of the MCIA, in order to consider a long term experience, will be to help the patients to follow their medication plans through self management. 2.3
For this project a characterization of medicines has been made since there are differences in how different types of medicines should be managed by the MCIA. All medicines comes with prescription from a doctor. Prescriptions include adherence information about how the medicine should be taken regarding dose and time schedule. Adherence information is personalized for each patient, printed on adhesive labels at the pharmacy and attached to each package of medicine. 6 https://www.apoteket.se/kundservice/receptlakemedel-sa-fungerar-det/
Medicines to be taken on a Regular Basis: The prescription label on these medicines state dose and the specific times each dose of the medicine should be taken. The goal of the MCIA is to make sure that the patient takes these medicines at the times they are specified.
Medicines to be taken On Demand: This type of medicines are medicines that the patient can take when he or she feels the need. Examples of common on demand medicines are pills to decrease pain or anxiety. The information on the prescription label states strength per dose, minimum time interval between doses and maximum amount of doses allowed in 24 hours. The goal of the MCIA will be to make sure that the user does not exceed the maximum dosage per day and occasion. 3
The aim of this section is to formally introduce both data sources and a multi-criteria decision making approach for supporting the decision making processes of the MCIA. 3.1
Let us start introducing the basic definition of a time point. A time point is a time stamp hDate; time_clocki. T denotes all the possible time points. Now let us introduce the basic definition of a medicine. A particularly interesting attribute of a medicine is its Anatomical Therapeutic Chemical (ATC)-code. The ATC-classification is an internationally accepted classification system, based on active ingredients and their therapeutic, pharmacological and chemical properties7. By AT C_codes, we denote a finite set of ATC-codes. Hence, a medicine is defined as follows:
A medicine m is a tuple of the form h ; &; p; ; i, such that m 2 AT C_codes R [0; 1] I Active_ingridients; where & 2 R denotes a substance concentration in milligrams, p 2 [0; 1] denotes a priority degree, 2 I denotes a time interval such that I = T T . M denotes the set of all possible medicines.
Medicines will be managed in terms of events. An event is something that happens and which should be acknowledged by the MCIA. For example, if a reminder is presented to a patient whereupon the patient takes the medicines, the MCIA should notice that event and not present any more reminders regarding the same medicines for the same occasion. Events could also be things like eating food, drinking milk or other things that might have an impact, or have an effect, on the users’ medication. This project however, only considers events regarding taking of an oral medicine, and is defined as follows:
An event e is a pair hm; ti, such that e 2 M events. 7 http://www.hpra.ie/homepage/medicines/medicines-information/ atc-codes T . E denotes the set of all possible
Constraints regarding medications can appear in three different ways. Firstly, it can
be a medicine incompatibility, i.e. some medicines should not be taken at the same time
as others, since it may have an impact on the effect of one or both of the medicines
[
A medication plan is the general plan which the MCIA wants the patient to follow. It is the foundation of the goals of the MCIA, and it is adherence to the medication plan that will be the primary source of feedback on how well the MCIA performs. The medication plan consists of all medicines that are prescribed, and also a set of constraints which should be considered while taking these medicines.
A medication plan M P is of the form M P = h(m1; m2; : : : ; mn); i such that mi 2 M(1 i n), = fC1; C2; : : : ; Cng, Ci 2 Constraints(1 i n).
Medication adherence is the measure of how well patients follow their medication plan. This is important information for the MCIA, since it can be seen as the result of its actions and decisions. Medication adherence can be divided into two parts, overall adherence (how well the patient is following the medication plan), and the individual adherence for a specific medicine. It is not easy to measure adherence since there are many factors which it depends on, e.g. skipping one medicine one time might be fine, while skipping another is not. Elementary factors of estimating the adherence are the priority of each medicine, which indicates how important it is to take the medicine, and the history of the intakes (compliance to the plan). The two definitions of adherence are presented below. An adherence function will be used to calculate adherence.
Let m be the medication adherence of a medicine m such that m = f1( m; p) where f1 is an adherence function for individual medicines, m is the history for medicine m and p be its priority.
Let be the over all medication adherence = f2(P1n mi ) where, f2 is an overall adherence function, and mi is the individual adherence for medicine mi(1 i n).
Let us point out that the adherence functions f1 (w.r.t. Definition 4) and f2 (w.r.t
Definition 5) are basically distance functions between the current state of adherence and
the intended medication plan. Hence, these functions can be implemented in different
ways. In our proof-of-concept prototype, f1 was implemented as a model checking
function, based on weak-constraints, following Answer Set Programming (ASP) [
In order to make decisions considering all of the relevant information such as the
medication plan and the so called information variables, a multi-criteria decision
makingapproach has been chosen and more specifically, the weighted sum method (WSM) [
terion can compensate for a bad criterion is usually referred to as the full aggregation
approach [
Information availability. One of the drawbacks with multi-criterion decision making in general, is that a lot of information has to be specified. In this case however, the information should always be available in real time through sensors and internal values.
All desires (also called goals), which the MCIA has committed to achieving, are called intentions and for each intention there is a finite set of actions fa1; a2; : : : ; ang. Actions are basically different ways of achieving an intention. Actions can also be seen as the MCIA’s means of interacting with the environment. The distinction between intentions and actions is a way of handling high level reasoning (using intentions), while still being able to adapt and be sensitive to the current situation (by using an appropriate action). An intention is defined as follows:
An intention x is a pair hI D; i, where I D 2 N, be the intention to be performed.
Example 1. Let xr be the intention to send a reminder to the user. Then there is a set of actions fa1; a2; a3g related the intention xr, where a1 = use audio output, a2 = use visual output and a3 = use audio and visual output.
Before presenting the definition of decision making, a couple of related definitions are presented. Utiliy weights can be seen as the priority of the information variable regarding a given decision, and is defined in the following way.
Let wxd be the weight for information variable x regarding the decision d, then wxd 2 fL; M; H g (low, medium or high importance).
An exact numeric value of the utility weight for the different priorities, is not defined and it may have to depend on the type of decision. However, a value between 0 and 1 will always be used for each of the different levels of importance.
The utility function U (a), uses information variables with utility weights to calculate the utility of an alternative a and is defined in the following way.
Let a be an action, and 1; 2; : : : ; n be the positive information variables, then U (a) = P0n xiwi where, xi is the value of the information variable i(1 i n), wi be the weight of the information variable i.
Only positive information variables are used in the calculation. Positive, simply means that if the information variable has a high value, it should increase the utility for the given alternative. This is chosen for simplicity of the calculation, but it puts some requirements on what information variables there must be in order for the utility function to be fair. Competing alternatives should always depend on similar information variables, which means that they should have the same importance and have a similar purpose. This problem could be addressed by setting a weight which corresponds to the exact value of the importance of the variable, but it is hard to exactly define the importance of an information variable for a given decision. Instead, utility weights are merely a rough estimation of how important an information variable is.
Information which is not defined explicitly as a value (such as information in the medication plan), but may still be important when calculating the utility of the alternative, will be converted into an information variable using a separate function. This function varies depending on the type of information, but the result will be a number between 0 and 1 and will therefore be treated as a regular information variable.
The decision of choosing the best action is taken in real time by using the following definition.
Let D be a decision and a1; a2; : : : ; an be competing actions, then D = max(U (a1); U (a2); ::; U (an)) such that U (ai) is a utility function which calculates the utility of ai(1 i n).
A plan is simply a list of intentions which, if nothing changes, will be executed by the MCIA.
Let be a plan = ( ; ), = [x1; x2; : : : ; xn] where each xi(1 i n) be an intention, = [l1; l2; : : : ; ln] be a list of dependencies such that li = ( ; ), 2 fIDj(ID; ) appears_in g, fIDj(ID; ) appears_in g8 and 2= . 4
In this section, a proof-of-concept prototype of the MCIA is described. In this
proof-ofconcept prototype, the MCIA has been embodied as a smart augmented reality
(AR)mHealth application in the settings of a Microsoft HoloLens. This AR-mHealth
application was designed as a long-term experience application (LTEA) [
The reasoning loop of the MCIA can be seen in Figure 2. A plan of intentions (later referred to as plan) will be constructed using the current state, referred to as internal state (Figure 1, 2), and the long-term goals. This plan will be created on a daily basis and planning will take place over a specific time period. By practical reasons, it is assumed that this planning process will take place during night time. This means that when the user wakes up, the plan for the day has already been made and only replanning using the event-driven approach is necessary. The reason why it is referred to as an event-driven process is that events can be seen as triggers that changes the internal mental state of the MCIA. Therefore, in the case of an event, the MCIA should check for interactions with the plan and re-plan accordingly.
Proactive behavior emerges by actions executed in order to achieve the intentions of the MCIA, such as sending reminders to the patient. Reminders are also contextaware regarding time, but in our proof of concept, also by simulated information from the environment. Reactive behavior emerges by using input to directly trigger some behavior, e.g. using voice commands and information regarding the user’s vision (using the AR-headset), it is possible to display information regarding medicine boxes. Autonmous behavior emerges by reasoning about the intentions of the day. The purpose of intentions are to improve medication adherence of the user, and by evaluating the behavior of the user, the reasoning can be adapted. 5
In order to show the feasibility of our approach, a usability evaluation of the proof-ofconcept prototype of the MCIA was done. We aimed to answer the following questions: – Is there a difference, related to age, regarding if people are willing to use an ARheadset for medication management? – Is there a difference, related to experience of using smart technology, regarding if people are willing to use an AR-headset for medication management? The functionality involved displaying information about medicine boxes regarding two features namely, helping the user to use a medicine at this moment and to help the user to prepare a dispenser. The evaluation involved 15 participants who were selected by using the following criteria: a.- Different levels of management of medication on a regular basis; b.- Wide range of ages (medication management in applies to people in all ages, not just elderly); c.- Different experiences using smart technology in general. The setting was a quiet and home-like environment. The participants were able to use voice, vision and gestures to interact with the system and were presented with both visual and audible output. Table 1 summarizes the visual information presented to the participants. After the test they were asked to fill in a form. Responses were on a five-point Likerttype scale graded from 1 (strongly disagree) to 5 (strongly agree). The lower bound to agree was made at 4 (4 or 5 = agree).
The evaluation showed that of all participants 20% perceived the technology hard to use and 13% thought that they would need a lot of training before using this technology in real life. There were a couple of vast differences regarding participants over and under 70 (4 and 11 respectively), and also between experienced smart technology users and those less experienced (8 and 7 respectively). Of the participants over 70, 50% were willing to use the technology in the future but none thought other people would appreciate the technology. For the participants under 70, the corresponding numbers were 91% and 100%. For participants over 70, 0% considered themselves as experienced smart technology users, while 73% of the people under 70 considered themselves as experienced. Of all experienced participants 100% were willing to use this technology in the future and 88% thought that most other people would appreciate the technology. 6
The possibilities of using smart-glasses (AR) within a system to assist doctors and other
healthcare-personnel in emergency situations was explored in [
Mitrasinovic et al. concluded that smart-glasses have evident utility to healthcare
professionals [
A concept to send context-aware reminders to users in order to increase medication
adherence was presented in [
An article about using a humanoid robot to support elderly peoples’ everyday life
[
General tools to increase medication adherence for patients are smart dispensers
(e.g. [
Medication adherence is a healthcare issue that affects both youth and elderly patients around the world. Until now, there is no a general solution that can support the dynamic demands that each individual requires for keeping his or her self-medication management optimal. In this regard, we argue that our approach based on intelligent coaching systems and AR-headsets shows a solid and scalable solution for leading with the complex processes of tailored services on medication management. Results from our evaluation showed that participants felt comfortable using an AR-headset during medication management procedures, such as taking pills and putting pills into pill dispensers. The evaluation indicates that the MCIA embodied in an AR-headset can be a useful tool in helping patients to maintain self management and medication adherence. From a societal perspective, maintained self management is likely to delay or possibly prevent, the need for professional assistance by nurses. As most western countries suffer from lack of nurses and other health care professionals the effects of the MCIA would potentially have a high impact on sustainability of public health resources. In our future work, we aim for a complete implementation of the MCIA and a long term usability evaluation of the MCIA.
Acknowledgements: This research has been supported by Nordic Telemedic Center (EU Interreg Botnia Atlantica), SFO-V Strategic Research Area Health Care Science and CEDAR (Centre for Demographic and Aging Research).