As the COVID-19 outbreak continues to pose a serious worldwide threat, numerous governments choose to establish lockdowns in order to reduce disease transmission. However, imposing the strictest possible lock-down at all times has dire economic consequences, especially in areas with widespread poverty. In fact, many countries and regions have started charting paths to ease lock-down measures. Thus, planning efficient ways to tighten and relax lock-downs is a crucial and urgent problem. We develop a reinforcement learning based approach that is (1) robust to a range of parameter settings, and (2) optimizes multiple objectives related to different aspects of public health and economy, such as hospital capacity and delay of the disease. The absence of a vaccine or a cure for COVID to date implies that the infected population cannot be reduced through pharmaceutical interventions. However, non-pharmaceutical interventions (lock-downs) can slow disease spread and keep it manageable. This work focuses on how to manage the disease spread without severe economic consequences.
While governments are responding to the spread of COVID19 by imposing lock-downs of varying intensity to reduce human-human contact, the situation cannot be maintained indefinitely. Each day of lock-down brings severe economic loss affecting the livelihood of billions. Thus, it is imperative to use the available resources of interventions – lock-downs, test-kits, ventilators etc., in an efficient manner. This work aims to find optimal lock-down policies based on epidemiological models and reinforcement learning.
Reinforcement learning has shown promising results on
sequential decision making tasks like Go
In addition to insufficient data and uncertainty, for our problem, it is hard to specify a single objective that one wants to achieve. It is likely that many objectives need to be met for the task to be considered successful. For example, we may want to delay the peak of infections while making sure that our hospitals are not overburdened or our economy is not affected too severely. The decision maker in this case is looking at the problem from several perspectives leading to many possible objectives that the model will be evaluated on. Thus, in this work, we incorporate multi-objective functions.
The main contributions of this work can be summarized as follows: • We formulate the problem of lock-down implementation as a Markov Decision Process (MDP). To solve this MDP, we propose a Reinforcement Learning (RL) approach that optimizes the trade-off between health objectives and economic cost. • We tackle the uncertainty in environment parameters that might arise from the noise in the data and the process of estimation by considering different robust approaches. • We analyse different robust approaches including uniform sampling and adversarial sampling during the training phase. We find that there is a trade-off relation in the average-case and worst-case between RL agents with different degrees of risk aversion. • We design different health objectives that might be of interest to decision-makers and measure our performance along these different objectives simultaneously.
With this work, we aim to address the challenging task of planning temporal resource allocation for lock-downs. The models that we use for modelling the spread of COVID are the SEIR class of epidemiological models.
Since as early as the 17th century, when Bernoulli
proposed the first mathematical epidemic model for
smallpox
Apart from epidemic modeling, the problem of
optimizing cure and control for preventing the spread of disease is
also of interest. However, most works in the computer
science literature usually assume an idealistic model, such as
every contact being known, no uncertainty in the disease
parameters or a strong cure/isolation that guarantees the
recovery of the individuals
Robust control is a branch of control theory that has a long
history. In particular, robustness toward parameter
uncertainty results in a performance drop from the model toward
its real-world application
Another important consideration for lock-down policy
makers might be to include different desirable objectives
in their decision-making. Multi-objective optimization has
tremendous practical importance in many real-life
applications
S E I R D t Ro
ties are often considered and solved when the system is easy
to describe
For modelling COVID-19, we adopt a discrete time SEIRD
model
The discrete-time dynamics equations for our epidemic model are:
St+1 − St = Et+1 − Et = (
It+1 − It = ( Rt+1 − Rt = Dt+1 − Dt =
−StIt
,
St − Et
It
Rt , , in which Ti1nf = Trec1over + Tfa1tal and the basic reproductive number can be obtained from R0 = Tinf /Ttrans. A typical SEIRD model described above starts from a population being mostly susceptible and a small fraction of infectious people. When R0 > 1, each infected individual will infect more than one susceptible individual in its lifetime on average. Each susceptible individual will eventually go through the exposed, infectious to finally recovered or deceased states. A schematic diagram of the SEIRD model is shown in Figure 1. Generally in compartmental models, Ttrans is a constant. However, in a real-world setting, the transmission time can be reduced through the deployment of non-pharmaceutical lock-down interventions. Note that there is no direct reduction in infected population as there is no cure or vaccination available. We only consider the lockdown interventions that increase the transmission time of the virus based on their strength.
Compartmental models and their variants are commonly
used in disease state forecasting and prediction. For
concreteness, in this work, state populations and numerical
values of the transmission parameters are based on the available
data for the city of Mumbai
As there is no cure/vaccine available for COVID-19 till date,
models of pharmaceutical interventions are not applicable.
To manage the rapid disease spread, different lock-down
policies can be considered to limit individual contacts. For
example,
These lock-down policies should enjoy several desiderata for real-world deployment. First, each type of intervention needs to last for a minimum duration d. Second, since the lock-down has economic cost, government bodies would expect a trade-off between the total budget B spent for planning and policy deployment and public health related gains. To model such interventions, we considered lock-downs as a series of action choices. The decision maker can plan different policies in different time periods based on the limitations mentioned above.
During the lock-down period in India, the change in
estimated transmission time as the effect of interventions has
been observed based on
In the public health domain, governments and decision makers may want to achieve different objectives when deploying a policy. One direct objective could be eliminating the disease which can be achieved by suppressing contacts so that patients recover at a rate greater than the spread of infection. This is equivalent to minimizing the area under the infection curve. However, this is not achievable in many regions, including cities in many of the developing countries due to the huge economic cost of such strict lock-downs. Thus, we focus on economically sustainable interventions that do not reduce R0 below 1. In epidemic theory, this means the disease cannot be eliminated within reasonable time no matter how the government plans the lock-down in these regions. Every susceptible individual will eventually go through the recovered or deceased state. In other words, although the infection curve will change, the area under the curve will remain the same.
To evaluate the effectiveness of lock-down policies under these circumstances, we use indirect objectives that are vital and achievable for sustainable interventions. For example, as there is limited hospital capacity, a patient’s quality of life will likely be better when the infected population does not exceed that capacity and they can receive proper treatment. Alternatively, we may want to delay the infection to the point when we have better system preparedness, medicines, resources etc. for handling the disease. These different desired objectives can be described as a family of objective functions that are variants of the Quality Adjusted Life Year (QALY) score in our model, which is elaborated below.
QALY is a popular established metric to quantify the
effectiveness of health interventions. It is often used in the
public health literature
In this work, we change the time scale from years to days to adapt to the dynamics of the disease we are facing. We mainly focus mainly on two objectives, burden and delay. We define these two objective functions as:
OBurden = X((I(t) − λ)1I(t)>λ − δBurdenl(at)) (6) t ODelay = X(tI(t) − δDelayl(at)).
t (7) where t refers to timestep. at, I(t) and l(at) refer to action, infected population fraction and cost of action at time t respectively. Also δ and λ refer to economy-health weight and hospital capacity and 1 is the indicator function. Here, we focus on optimizing a linear combination of these two objective functions, written as:
Omix(w) = wO¯Burden + (1 − w)O¯Delay, (8) where the weight w ranges from 0 to 1 and O¯ is O normalized by the absolute value of no intervention, i.e., we divide O by its absolute value in the absence of interventions.
Our lock-down control problem can be modeled as a Markov
Decision Process (MDP)
We thus consider a parameter-wise robust reinforcement learning model to solve the MDP framework. The MDP framework can be written as:
hS, A, P, Ri with state space S, action space A, and transition distribution and vector reward
P(s′|s, a) for s, s′ ∈ S and a ∈ A
r(s) ∈ R and the preference weight w ∈ Rn.
The states we consider are the fractions of population present in S,E,I,R and D compartments at the given time. Furthermore, we consider several discrete actions at each time step corresponding to different strengths of lock-down with different costs. It is natural to assume the strength to be monotonically increasing with the cost as otherwise the action choice will be dominated by actions with less cost but more effectiveness. For simplicity, we adopt a linear mapping for both cost and effectiveness, as:
Ttrans(a, e) = (1 + ec(a)) Tinf
R0 (9) and c : A → [0, 1], in which e is the lock-down effectiveness coefficient and R0 the basic reproduction number when there are no lock-down interventions. Both of these are estimated with data from the city of Mumbai, India. Ttrans and Tinf are the transmission and infection time periods.
For the remaining tuple, the transition distribution P(s′|s, a) is described as the disease transmission equation 1 to 5. The total accumulated reward is exactly the objective function O in equations 6, 7. The next section describes the distribution of individual reward signals across states.
Multiple Objectives: We have defined the state, action, transition probabilities and total reward in the MDP section. The only missing piece for a complete reinforcement learning framework is to design the reward signal at every timestep. We have designed a framework that works not only for the two example objectives we focus in this work, but on most variants of QALY.
Most variants of QALY, including our examples, are related to time and population of certain health states. We propose a function we call the QALY value V (x, t) which is a function of the population x in a certain health state and time t. We focus only on I or the Infectious state in these experiments. However, the QALY value function can be generalized to a vector form to include multiple states. For controlling the hospital capacity, the function can be formulated either as a constant penalty for x exceeding the capacity or simply as a reward for x below the threshold, since the area under the infection curve is a constant, as we elaborate in section Multi-Objective Functions. We formalize this as:
VBurden(x, t) = 1 for x < λ As for delay, we formalize this function as (10) (11) VDelay(x, t) = t T
Given that the QALY value V (x, t) of the objective function is defined, the reward signal at any given time t can be calculated by r(t) = R I(t) V (x, t)dt. We can thus apply the 0 reinforcement learning approach.
One benefit of such a proposed approach is that the QALY value function of the mixed objective can be easily calculated as:
Vmix(w) =
wVBurden OBurden(no action) +
(1 − w)VDelay
ODelay(no action)
(12)
Uncertainty: Another important aspect other than having a
multi-objective function in the lock-down application is the
uncertainty of the parameters (e, Tinf , Tinc), which are
related to the infection curve directly or indirectly. We
experiment with three approaches to analyze the effect of
uncertainty in a reinforcement learning setup:
(1)Fixed RL(FRL): Train the RL agent using only the mean
of the uncertain parameters.
(2)Distributed RL(DRL): Train the RL agent using
samples of uncertain parameters from the estimated range.
(3)Adversarial RL(ARL): Inspired by
In this section, we describe the application of our method to a specific location – the city of Mumbai, India. In subsequent subsections, we describe how we estimate the model parameters as well as the uncertainty in these parameters. We also report the results of our method when used on the estimated parameter ranges.
We fit our SEIRD model to the time-series data from the
COVID19-India API
We decided the initial search space for the model
parameters based on the estimates given by public health
experts and those cited in literature. We process the data with
smoothing techniques to reduce the effect of bulk data
entry. Then, we search over the parameter space for parameter
sets that have a small aggregated RMSE loss between
predicted numbers and actual numbers using the Hyperopt
library
We found that there are diverse parameter sets that have loss close to the best-fit parameter set. Thus, we picked all parameter sets that have a loss within a certain range of the best loss (within 10%). Among all picked parameter sets, we find the range of values taken by individual parameters. These ranges for individual parameters give us a measure of uncertainty for these parameters. We assume a uniform distribution over these ranges as our parameter distribution.
Robustness: Robustness of policy to uncertainty in parameters is an important aspect. Over the estimated uniform distribution range, we find the worst-case parameters for different methods using a fine grid-search. Then, we measure
Random FRL DRL ARL the performance of different methods on their corresponding worst-case parameters. We also find the corresponding average performance over the parameter distribution. The results are tabulated in Tables 2 and 3. As shown in these tables, the ARL helps the reinforcement learning discover risky parameters and thus performs best in its worst case scenario. For average case, however, ARL performs worse than the best method (FRL and DRL respectively). This has shown the trade-off between performance and robustness in our lockdown problem - at the cost of average performance, we can obtain better worst-case performance.
Different objectives: We use different weights between Burden and Delay objectives and compare the results to the case when we individually focus on Delay and Burden in Table 4. The objective function we use is (1 − w) ∗ Delay + w ∗ Burden for different values of w. The aim is to maximize normalized objective for both Burden and Delay.
From Table 4, we observe that, as expected, as the weight on Burden increases, the Burden objective becomes larger for all methods in general. Similar behaviour is observed for Delay as well. When applying this method for policy guidance, we can tune w to achieve the required objectives for both Burden and Delay.
We implemented reinforcement learning on the lock-down policy optimization problem for COVID-19 while considering important real-world aspects like robustness and multiobjective optimization. Robustness can be achieved by introducing an adversarial agent for parameter discovery, but at the cost of sacrificing some performance on average. For the multi-objective mixture, we study the trade-off between controlling hospital capacity and delaying the infection spread. We proposed a reward distribution framework for the reinforcement learning agent to shift from one objective to another in the lock-down problem. One point to note is that our epidemiological model (SEIRD) is a homogeneous model and is being used to optimize the policy keeping the trade-off between economy and health for the community as a whole. The model does not discriminate between two infected individuals based on their economic contribution and neither is it capable for the same. This makes sure that we generate lockdown policy as fairly as possible.
The future direction of this work is to gather more data on both cost and effectiveness of the real-world lock-down policies on community scale so that a more complex model can be used to better estimate the real-world scenarios. For example, transmission times are known to not be homogeneous and several super-spreader events have been identified in many different spreading routes. Collecting data on such cases and modifying the model to have different transmission times for different cases of spread would give us a more holistic view of the entire scenario. Another important direction of extension would be estimating the reporting rate from other sources of data and normalizing the reported numbers to estimate parameters that are closer to the real-world.
This study is made possible by the generous support of the American People through the United States Agency for International Development (USAID) and Army Research Office (ARO). The work described in this article was implemented under the TRACETB Project, managed by WIAI under the terms of Cooperative Agreement Number 72038620CA00006 and by Teamcore, CRCS, Harvard University under Multidisciplinary University Research Initiative grant number W911NF1810208. The contents of this manuscript are the sole responsibility of the authors and do not necessarily reflect the views of USAID, ARO or the United States Government.