Clinical pathways describe the treatment procedure for a patient from a medical point of view. Based on the patient's condition, a decision is made about the next actions to be carried out. Such recurring sequential process decisions could well be outsourced to a reinforcement learning agent, but the patient's safety should always be the main consideration when suggesting activities. The development of individual pathways is also cost and time intensive, therefore a smart agent could support and relieve physicians. In addition, not every patient reacts in the same way to a clinical intervention, so the personalization of a clinical pathway should be given attention. In this paper we address with the fundamental problem that the use of reinforcement learning agents in the specification of clinical pathways should provide an individual optimal proposal within the limits of safety constraints. Imitating the decisions of physicians can guarantee safety but not optimality. Therefore, we present an approach that ensures compliance with health critical rules without limiting the exploration of the optimum. We evaluate our approach on open source gym environment where we are able to show that our adaptation of behavior cloning not only adheres better to safety regulations, but also manages to better explore the space of the optimum in the collective rewards.
• Applied computing → Health care information systems.
HealthRecSys’20, September 26th, 2020, Online, Worldwide © 2020 Copyright for the individual papers remains with the authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This volume is published and copyrighted by its editors.
Our work focuses on the use of reinforcement learning to optimize
and personalize clinical pathways, illustrated in Figure 1.
Rehabilitation procedure, called „Clinical Pathway“, describes in detail
which activities are to be carried out for a patient within a course
of treatment[
The process of creating a clinical pathway tailored to an individual patient spans several stages. To adapt a clinical pathway to a patient’s needs, one starts from a disease specific blueprint and later incorporates the patients clinical picture as well as his or her individual preferences.
On an abstract level, the adaptation of a pathway can be modelled as a decision process. A number of activities must be decided upon, which in turn have interdependent efects among one another. Feedback on the efectiveness of the decisions made is often only given with a delay or in aggregated form - for example during a control visit to the doctor after a certain time. Reinforcement learning is about optimizing processes that can be described as a feedback control loop. The application of RL to the individualization of clinical pathways is therefore particularly well suited and promising.
The personalization of a clinical pathway is about identifying the optimal combination of activities and treatments in rehabilitation for an individual patient. In this context optimality can be considered from diferent viewpoints. On the one hand we see the fundamental objective of proposing rehabilitation measures that are safe from a medical perspective. On the other hand we aim to support the recovery process in the best-possible way by exploring alternative rehabilitation activities. While there are generic templates for diferent medical diagnosis - that are safe, there is the need to go beyond and adapt the clinical pathway - to provide tailored care plans to the individual patients.
In order to address the objectives described above for clinical path recommender systems, we present a safety-aware reinforcement learning approach. On a conceptual level this means that we have a state of our patient and our agent proposes an action - a rehabilitation measure - for our patient at time (Figure 1).
The agent receives a reward +1 based on the change in the
condition of our patient +1. While classical RL is based on try&error,
the healthcare application must guarantee the safety of the
patient during the proposed activity. Imitation Learning is one of
the ways in which this is pursued. Here the agent is trained to
to „imitate“ an expert’s actions, i.e., to suggest a similar treatment
activity to the one a doctor would choose faced with the same
patient profile. Current work in imitation learning [
• How can we emphasize the importance of safety in suggesting rehabilitation treatments to a reinforcement learning agent? • How can an agent explore the individual optimum and still remain within a safe and medically acceptable action space? In answering the questions within this study we contribute to the following: Contributions: • a conceptual approach to extract safety relevant behavior from expert demonstrations • an adapted conceptual method for imitation learning that emphasizes safety-critical thinking • implementation, application and preliminary evaluation of the concepts
Paper outline: After we position our work in the scientific related work in section 2, we introduce the conceptual background of our approach in section 3.1. While in section 3.2 we present the novel concepts of our approach, in section 3.3 we focus on the explicit application to optimize clinical pathways. In chapter 4 we outline our evaluation method and discuss the results achieved in chapter 5. Our work is then completed by a conclusion (section 6) and an outlook on future work in section 7. 2
Our work covers various areas of health care and machine learning, which we would like to examine in greater detail.
Research has shown an increasing interest in applying machine
learning techniques to health care related tasks. From
modelling disease progression [
Contrary to previous imitation learning techniques, our approach
focus on avoiding unsafe states while still exploring safe states
to find the optimum. We teach the agent to handle safety critical
states by imitating expert actions in similar situation. In safe states
however the agent does not need to stick exactly with the behavior
observed in expert demonstration. In fact we encourage it to search
for the best personalized clinical path possible by exploration. While
current safety RL algorithms [
At each step the agent selects an action ∈ () based on the received representation of the environment state ∈ . Applied in a health recommender system speaking about action and states relates to recommendations for therapy activity and patients clinical state respectively. The agent receives a reward, +1, which quantifies the development of the clinical condition and personal well-being of the patient and a new state +1 of the patient as a consequence of its action. (|) is the agents policy which is assigning a probability to each action at a given state and choices the most promising. This part is to be trained during the exploration or in the case of imitation learning during the expert observation. Since the new state serves as the input for the next iteration the agent keeps on interacting with the environment and creates a trajectory = ( , ) | ∈ [0, ℎ ] where is the state at a given time and the action where includes all elements from the start time 0 to the time of termination ℎ . The trajectory for an individual patient directly relates to the configured pathway (actions map to the parameterised treatment activities foreseen in the clinical pathway) and the observed reaction of the patient (states ). The objective function denoted as ( ) relates to ( ) = E [ ( )] = ( ) = ∼ ∞ Õ
=0
where ∈ [
Focusing on modelling the safety of an reinforcement learning agent we define for brevity to approximate the safety of a given state . The flexibility of the approach provides the possibility to diferentiate safety in several dimensions or to describe it as a holistic unit. In the case of Imitation Learning from sub-optimal but safe demonstrations we calculate the threshold value over the distribution of expert trajectories, such that = ( ) from the observed in the expert demonstrations . Evaluating the received expert trajectories we can now quantify how critical the diferent states were in terms of safety by defining: = {( , ) : ∈ ∧ ( ) ≥ − } (4)
By focusing on the subset to train our agent we can assure
that it knows how to handle critical situations while preserving the
freedom of exploring safe states. The collected demonstration data
set is then weighted in such a way that the training data set for
(1)
(2)
imitation learning consists of safety-relevant trajectories ( ) to
a defined extent mixed with randomly sampled trajectories from
. Trough out this paper we will refer to this weighing as safety
focus ∈ [
Our approach allows to learn efectively from demonstrations that guarantee a safe state of the environment respectively of the patient, but beyond that the actions not always show the optimal reaction to the state. Furthermore we aim to train a reinforcement learning agent with weighted expert demonstrations and thereby putting safety or other evaluation criteria in the foreground. Applying the approach on training a reinforcement learning agent to suggest and parameterize treatment activities in a clinical pathway we train the agent to explore the optimal recommendation while imitating expert recommendations when facing critical states described by constraint cost functions.
If the formal description is applied to the healthcare application, the cost function evaluates the clinical condition of the patient. In concrete terms, one could, for example, evaluate the deviation of the measured pulse from rest or optimal pulse. One now look at the expert’s demonstration, i.e. any number of pairs of the patient’s condition and the proposed therapy measure, you can evaluate for each demonstration what the cost function is, i.e. the safety assessment of the patient’s clinical condition. It is crucial that the costs are not per se included in the objective function but are used as restrictions. As a result, an increased heart rate is not interpreted as negative by our recommender, but we take care in the decisionmaking process that the safety of this attribute is within certain limits.
We are therefore aware that the heart rate drops out during a therapeutic measure, and that this is one of the undesirable efects. But we want to make sure that the proposals of our intelligent system are within the limits of the experts’ opinions. So if we see in the trajectories that the safety costs are below a certain level, we want to make sure that our proposals do not exceed this limit. To learn this, the demonstrations where the patient’s condition was particularly close to the observed limit are particularly relevant. In our approach we define a sub set of trajectories that have a defined distance from the critical limit . From this sub set we know that it is particularly relevant to learn how to avert critical states. During the training process, our intelligent system should accordingly pay special attention to adapting the expert suggestions close to the critical states. 4
Although the concept presented here was developed out of the motivation to individualize clinical pathways for patients, it can be applied to various applications of reinforcement learning. For this reason, and because clinical data was not available to the extent necessary for an analysis, the evaluation is based on common and comparable safety problems in the directive. We use the gym environment provided by OpenAI. 4.1
The gym environment ofers the possibility to run diferent task scenarios for reinforcement agent and to extend the provided framework. Especially atari games and two dimensional games such as car racing are very popular and provide an excellent baseline to compare results. Due to the parallel use of the car racing environment as a recommender in the health care sector, the car racing environment is particularly suitable to demonstrate the functionality of our approach. The car on the race track describes the condition of the patient, who changes depending on the action - steering and accelerating, or parameterization of the next treatment measure. The more critically the condition of the patient - the position of the vehicle on the track - is evaluated, the more relevant it is to behave similar to the expert demonstrations. While we have described the relevance of heart rate in the clinical environment above, safety in this environment can be quantified with a cost function based on the distance to the edge of the track. So while in the medical case we can observe how a doctor behaves when the heart rate is particularly high or exceptionally low, in this environment we can quantify how far the vehicle is from the edge of the track.
To learn how to deal with critical conditions, we then look at the demonstrations where the assessment of the condition was particularly critical, as described in the formal description. The subset used for imitation learning is selected upon those based on equation 5. 4.2
In the following we will describe the dimensions and parameters used for our evaluation in more detail: Demonstrations:All experiments were carried out on the same demonstration data set of size | | = 4692 ∗ ( , ), for further detail see Appendix A. Imitation learning was performed as supervised learning of a tensorflow model with same architecture for every experiment. The agent was trained for 2000 ℎ, pairs of ( , ) respectively. Cost function: In our evaluation we consider three cost function that quantify the cars position in the environment. Since the cars state represents a patients clinical state this can be seen as three diferent clinical parameters that are monitored during expert training. The cost functions considered quantify the cars position by evaluating the game frame received as a state representation. In the three directions , and ℎ we calculate the distance to the unsafe state - the green besides the road. Evaluating these three cost functions for each state observed during the demonstration we develop a representation of the states safety. The parameter , which indicates how early a state should be classified as safetyrelevant is set to = 5. To calculate we move from the edge of the distribution of expert examples in the dimension of a constraint - in this case the safety - to the centre of the distribution. Visually this parameter defines how wide the edge of the distribution is, which is classified as safety critical as shown in Figure 3.
Agent testing After the weights of the reinforcement learning agent were trained via imitation learning the agent is evaluated in a newly generated gym environment. Here we observe the agent for two whole episodes to collect information about its performance and its safety. Depending on the individual performance of the agent this relates to ≈ 2000 state action pairs. 5
In the following we want to present the preliminary results of applying our approach to the safety critical decision process described in 4.2. Diferent values for in equation 5 has shown significant influence on the performance of the agent with respect to the safety as well as the reward as shown in table 1.
The results show that the safety focus has a significant impact on the agent’s performance. The agent trained with the unweighted expert demonstrations achieves an average safety rating of 13.05 for its proposals, and the variation in safety over the ≈ 2000 state action pairs of 16.41 should be noted. The approach of pre-selecting and weighting the demonstrations based on the distribution of the cost function shows a positive impact. The security evaluation of the conditions caused by the agent can be raised to a level of 17.3 by a weighting of = 0.1, and by a weighting of = 0.5 it can achieve a value of 21.37. In addition, the weighting of the expert trajectories in these cases also leads to a more robust reinforcement learning agent, which is reflected in the standard deviation of safety.
To make the results presented more comprehensible, Figure 4 provides a visualization.Training the agent with diferent safety focuses results in safety and reward shown on the y-axis and the standard deviation represented by the dots size.
Taking a closer look at comparing the cost function values for two agents - one trained without safety focus (5) and one trained with a safety focus = 0.8 (6) - emphasising the safety critical trajectories in the expert demonstrations can significantly raise the safety of the actions recommended by the agent. While the performance of the agents is already reflected in the values listed in table 1, the reasons for this can be identified in Figures 5 and 6.
The non safety focus runs where not able to obtain a critical distance to the critical states„ while the safety focus runs successfully learned to avert critical states in an expert reaction manner.
While the agent without safety focus was not able to learn the correct handling of safety critical conditions during imitation learning, our approach was successful in adapting the expert’s handling of critical states. By pre-selecting the expert examples without providing any further information during the training process, the agent with safety focus was able to avert safety critical conditions similar to the expert’s behaviour. The motivation for this work is derived from the medical context, in which the objective is to adapt clinical pathways to a patient’s needs in the best possible way. while this scenario can be aptly described as a reinforcement learning problem, as discussed in the introduction, it is important to limit the exploration and thus the parameterisation of therapies and activities to a safe range of action from a medical point of view. the imitation learning approach ofers a suitable approach to imitate the behaviour of experts. However, two central questions have arisen in reinforcement learning. Firstly, the question arose as to how an agent imitating an expert can concentrate on learning safety relevant actions. Furthermore, we asked ourselves whether an agent can be given the opportunity to explore the optimum within the action space while still maintaining a focus on safety.
To answer these questions, we have developed an approach that learns from expert demonstrations and concentrates on adapting the safety-relevant behaviour of the expert by appropriately weighting the examples provided. Our approach defines two parameters that determine how to deal with the state action pairs observed among experts. On the one hand, we have parameter , which indicates how early a state should be classified as safety-relevant. On the other hand we have safety focus forcing the agent to train on a subset of expert trajectories, where of the examples are classified as safety relevant under a given value . Our approach for imitation learning was able to outperform equivalent agents trained on balanced demonstrations with regard to the safety as well as the reward. The generic conceptual approach underlying the work can be applied to a wide range of RM tasks. It is especially relevant for domains where expert knowledge is available, which defines how one should behave to be safe, but where it is not sure exactly what the optimal behaviour may look like. This is the case in the personalization of clinical pathways. while physicians can precisely advise which activities to suggest as rehabilitation under certain clinical conditions of the patient, it is not certain whether these suggestions are the optimal choice. with our approach we provide an important basis for exploring the optimum when proposing individually parameterized activities without violating the limits of the safety-relevant parameters. 7
Besides the further exploration of the parameter combinations of
and , the transfer to additional RL problems is pending. Evaluating
the approach on further 2D games in the gym environment is a
logical next step. Additionally, teaching robotics to safely interact
with their environment is relevant application [
Future research should also consider how to completely avoid
safety-critical examples that are dealt with by experts. One possible
approach to this could be the simulation of responsibilities and the
evaluation of possible reactions by an expert, using human in loop
approaches as feedback for the system, see [
This work was partially supported by the project vCare: Virtual Coaching Activities for Rehabilitation in Elderly (funded by Horizon 2020 research and innovation programme under Grant Agreement Number: 769807). Special acknowledgements are directed to the partners of the project, who have contributed valuable feedback in the specification of the research problem and by providing their expertise to this study. A
Following we show the cost function calculated for the expert demonstrations. In 7 we see the to cost functions calculating the safety for and ℎ . In addition we evaluated the safety cost function in the dimension ℎ , as shown in 8.
In the following we provide further insights on the agents performance trained on diferent levels of .
Safety Focus 0.0 To complete the report on reinforcement agent performance with no safety focus besides 5 we provide the cost function referring to the safety evaluation . Training the agent with = 0.0 results in the cost function to the front shown in Figure 9.
Training the agent with a safety focus of 0.1 results in the cost function shown below. Safety estimation to cost function sides is shown in Figure 10 and function front in Figure 11 respectively.