Throughout history, epidemic outbreaks have led to spikes in human illness and mortality, with major challenges to communities and society in general. An epidemic situation requires decisions to be made about interventions that could reduce or contain the disease spread, taking into account all the information received and the projections of the current situation into the future. Decisions made by public health o cials involve determining the best sequence of actions to perform from a set of alternatives (school closure, vaccination, isolation). In order to decide which intervention strategies to implement, decision makers need to analyse a large number of scenarios and variables. This task can be overwhelming. Reinforcement Learning (RL) optimisation strategies have been proposed in the past years to automatically nd optimal intervention strategies for a disease spread in order to support decision makers. An important component in RL is the environment, which describes the task that the RL agent (solution approach) aims to optimise. This work focuses on how to design environments to represent the problem of epidemics and nding optimal interventions. We present di erent challenges that need to be addressed for environment design and provide diverse examples from the state of the art.
Infectious disease spread is a persistent threat to humankind. Emergence or reemergence of pathogens can lead to an unexpected increase in the number of infected and sick individuals in a local area (epidemic) or even a global area (pandemic). In the last decade, diseases such as Measles, AIDS, Malaria, Ebola, among others, still cause millions of deaths. The crucial question is not whether an epidemic will emerge, but when it does, Which interventions are the most e ective(e.g., social distancing, school closure, vaccination) to contain or reduce the spread? This question poses a challenge to governments, public health o cials and emergency response personnel, who must select the optimal intervention to implement from a vast number of possible alternatives.
To make analyse the possible outcomes, policy makers generally use mathematical models or simulations. In this arti cial setting, di erent types of decisions can be made without experiencing real costs or consequences. Using a simulation, the decision maker could employ intervention strategies at discrete decision points during the evolution of an epidemic. The e ect of any simulated intervention is stochastic, which increases the complexity of the decision-making process.
The space of possible interventions or combination of interventions strategies is large and nding the optimal health policy can be an overwhelming task. Generally, decision makers analyse and compare the performance of a one-time decision over a limited number of pre-selected intervention strategies. For example, in [21], authors compare the cost-e ectiveness of a pre-de ned set of health policies. Nonetheless, this approach may fail to consider the optimal intervention strategy. In addition, it does not consider that a plan of sequential interventions over time may be better than a one-shot intervention. However, nding an optimal combination of interventions over time can quickly become intractable and leads to a complex sequential decision-making problem.
A solution would be an optimisation technique that can e ectively search the space of possible health policies and identify the optimal interventions to control an epidemic. In this work, we rst frame the task of nding an optimal intervention strategy as a Reinforcement Learning (RL) problem. In RL, an agent (optimisation technique) interacts with an environment (problem de nition) to learn and nd an optimal policy in a sequential decision-making scenario. The agent and environment components can be implemented in di erent ways depending on the problem de nition. In this work, we mainly focus on how to design environments to represent the problem of nding optimal interventions during an epidemic outbreak. We present di erent challenges that need to be addressed for environment design and provide diverse examples from the state of the art. Next, we o er a concise overview of possible solution approaches to address the optimisation problem represented by the environment. Lastly, we conclude and highlight future work. 2
We frame the task of nding an optimal intervention strategy as a Reinforcement Learning (RL) problem in Figure 1. In RL, an agent within an environment executes an action at over t 2 T discrete time steps or decision points. For each action, the agent receives a reward rt. The goal is to nd an optimal policy that can indicate the best way to act for each environment state. Similarly, we assume that public health o cials employ mitigation strategies at discrete decision points during the evolution of an epidemic. The mapping of RL components to our problem setting can be summarised as follows (see Figure 1 above): Agent/Optimisation Component: the Optimisation Component aims to explore the space of possible policies to learn an optimal policy. See subsequent sections for more details.
State
Reward +1 +1
Agent
Reward Function Disease Model
Action (Intervention
Strategy) indicating \no intervention". Each element a 2 A is an action or intervention strategy. At each state st, only a 2
A(st) actions are possible. For instance, possible actions depend on the availability of resources (e.g., vaccine doses) and S Xn is the power set of interventions that can be carried out simultaneously.
: The policy is a function that indicates an action a 2 A to perform for each state s 2 S ,
: S ! A.
The main goal of this research is to design an optimisation component that can nd an optimal policy which can be used by a public health o cial, to choose an action a in each disease state s to obtain the maximum expected reward after t 2 T decision points, i.e.
= arg max E
PT t=1 rt . 3
The environment in RL characterises the world that the agent interacts with to learn and achieve its goal. We de ne two main elements for the environment: the disease model and the reward function. The disease model is a mathematical or simulation model that represents the disease dynamics. At each decision point t, the optimisation component executes an action at to test its impact over the disease spread. The outcomes generated by the disease model is represented as a state st and interpreted by the reward function, which issues a reward rt that the agent uses to measure its success. Lastly, other design decisions related to the de nition of the environment are those related to the intervention strategy (what are the actions/interventions?) and the state representation (how are states de ned and represented?). In the following sections, we will discuss in depth on how each of these components can be implemented and provide di erent types of examples.
Throughout this section, we will use a running example to introduce several
topics. This example is called the Boarding school scenario. In 1978, an
outbreak of the in uenza virus in a boys boarding school was reported [
In order to understand the large-scale dynamics of a disease spread, we use a mathematical model structure. The models allow prediction of the spread of the disease through a population using a description of the infectious disease dynamics at the individual level.
For infectious disease we understand the e ective colonisation of a host by a
micro-organism like bacteria, viruses, parasites or fungi, producing a subnormal
functioning of the host, which can lead to disease [
Transmission The propagation of a pathogen in the population depends on
several factors. The World Health Organisation (WHO) suggests a list of
measures to quantify the transmissibility of the disease [20] such as the number of
symptomatic cases, the basic reproduction number (R0), and the clinical attack
rate. A measure that is frequently used in modelling is the basic reproduction
number [
Virulence This is the ability of the pathogen to cause damage or death in the
host.The WHO list of measures for virulence are found in [20]. Pandemic Severity
Assessment Frameworks [
How are diseases modelled? The value of using mathematical models in
epidemiology has been in existence since 1766 when Bernoulli published a model
that shows the increases in people's life expectancy if a portion is inoculated
against smallpox [
Mathematical models are a representation of the reality that describes the evolution of an infectious disease over time. The models are useful to explore the di erent hypothesis, analyses and assumptions on a variety of scenarios. In consequence, the model should be as complex as needed to ful l its purpose.
The modelling of the disease's dynamics is based in the division of the population into categories or compartments that represent a speci c stage of the epidemic (e.g., susceptible, infected, recovered). The transition of individuals between compartment depends on the equation that describes the system ( e.g., di erential equation, transition probability) that considers the characteristics of the disease (e.g., virulence, transmissibility), the population dynamics (e.g., population size, e ective contact networks) and the applied intervention strategies.
The model structure should consider the distinctions of disease transmission,
with examples shown in Fig.2 The simplest is SI (Susceptible Infected), the
susceptible individuals have no immunity, when they get infected they remain in
this compartment their entire life. It is usually used to model HIV [
The SIR model has been used to describe the Boarding school use case. Since
we only have a dataset about the disease spread, di erent methods can be used
to implement the SIR model to build a simulation, for instance in [28] authors
used a Markov chain and in [
Generally, epidemic models can be classi ed into two categories: deterministic
and stochastic models. In a deterministic model, the outcome is fully determined
by initial conditions, parameter values and underlying equations. Deterministic
model simplify the dynamics of the disease spread and the population
composition enabling relatively fast computation of model outputs. However, these
models do not capture the complexity of human interactions and behaviours. A
stochastic model has some randomness in that the same initial conditions and
parameter values leads to a di erent result each time the model is executed.
Infectious disease transmission is a stochastic process. To create more
realistic models, it is necessary to incorporate random elements at some level. For
this reason, stochastic models are preferable to study a small population such as
households [30]. In local social networks (e.g., schools, works places, households),
the social structure is very important in how the disease is transmitted because it
is based on the interactions between the individuals. Disadvantages of stochastic
models are the high computational cost of simulating a disease spread in a large
population. There are many types of stochastic epidemic model: network [
Di erent approaches to model the Boarding school scenario can lead to a
deterministic or a stochastic epidemic model. On the one hand, the Markov chain
used in [28] has stochastic transition probabilities between states. On the other
hand, using di erential equations the authors of [
Note that using a deterministic or stochastic model will make the environment deterministic or stochastic, which in turn a ects the way an agent should be evaluated. If the environment is deterministic, the interaction results between the agent and environment will always provide the same results. If the environment is stochastic, then it is necessary to run the interaction between the agent and the environment several times to evaluate how the agent performs in expectation. 3.2
Interventions are a set of measures that aim to mitigate the impact of a disease spread by preventing disease propagation, by reducing the severity or duration of an existing disease, or by restoring functions lost as a result of the disease.
When an epidemic arises, the attention of decision makers shifts to questions such as: What should be the target population of the intervention be? What material and human resources are needed, and how much? What are the indicators of success (e.g. vaccine coverage, number of people in treatment, number of people infected)? In this way, the decision makers start analysing which mitigation measures may be more e ective. In the research literature, some authors divide the interventions into pharmaceutical interventions (PHI) and non-pharmaceutical interventions (NPI). PHI includes treatment, vaccines and antiviral drugs. NPI include quarantine, isolation, travel restrictions and school and workplace closure
Another classi cation of interventions can be found in [
In an epidemic model, when interventions are added they are implemented to reduce the number of susceptible individuals (aka preventive interventions), reduce the length of time an individual is infectious (aka treatment of disease) or reduce the transmissibility by limiting contacts or introducing sanitary measures (aka reduce-transmission interventions), among other. In general, the application of an intervention implies a change to the epidemic model parameter values. For example, reduce-transmission interventions such as isolation (restricts the movement of infected individuals and separates them from those who are healthy) decrease the transmission rate parameter in the model. Note that, depending on the level of granularity of the model, several interventions of the same type can be applied to the epidemic model by modifying the same parameter. For instance, in the Boarding school scenario which is described by the SIR model, modifying the transmission rate parameter can have several meanings in terms of interventions such as isolation of individuals, school closure or wearing masks. 3.3
The state S is de ned as the set of all possible states fs1; s2; : : : ; smg where the size of the state space is jSj = m. The state completely encapsulates the status of the system and includes all the information necessary for the agent to make a decision, discarding other irrelevant aspects of the data. In a chess game, a state can be all possible chessboard con gurations.
In [
We can nd examples of state representation for disease dynamics in the
following works: In [
Imagine that you are teaching your dog to do tricks. You will provide treats as a reward every time the dog does what you are teaching it. But If it fails to do the trick, the punishment is not giving any treats. Therefore, your dog will gure out the action that made it receive treats and repeat that action.
In the RL environment, the term reward is used to denote any type of feedback presented to the agent, which may be positive (reward) or negative (punishment). This feedback provides the agent with an idea of the actual value of being in a state. Therefore, the agent which receives the reward/punishment will improve itself.
The reward function in our problem represents the decision maker's goals. It can be single or multi-objective, and the reward function structure should re ect this. If a con ict between the objectives is generated, we need to prioritise the components of the function. The reward function punishes bad performance which is commonly considered in term of costs. It provides positive reinforcement for good performance which we will call bene t.
To understand how a reward function is de ned, let's use simple examples with the boarding school scenario. The goal is to minimise the number of individuals infected during the epidemic; a simple reward is utilising the number of new infected in each time step. rt = N ewInf ected. The problem with this reward approach arises when the agent learns that an intervention that reduces the infection, without evaluating the social and economic costs, is always the best.
Another goal can be to minimise the total expenditure cost during the epidemic. The reward function is represented as the sum of all cost incurred during each time step. rt = CostSick + CostIntervention + CostHospitalization.
In this case, the challenge is to determine the value of each cost. In interventions such as school closure it is di cult to calculate the cost because it involves multiple variables such as tutors absent from work. An example using bene t is the maximisation of the health of the population. The reward function can utilise health units such as QALY (Quality-Adjusted Life Year ), DALY(Disability-Adjusted Life Year ) to represent the gain of health in the population. Thus, a reward function can be a combination of these cost and bene ts components. It is widely used in health economic evaluation.
In [
There are di erent methods to tackle the optimisation problem that is represented by the environment (previous section). Although we have focused on environment design up to now, in this section we o er a concise overview of di erent solution approaches, with a focus on RL-based solutions.
A possible solution approach is a brute force exhaustive search. This
approach is computationally infeasible due to the large solution space (discussed
above). Testing the cost-e ectiveness of a pre-de ned set of health policies, as in
[21], could miss the optimal solution. Exploration-Exploitation methods could
provide better solutions. These initially carry out a broader search and slowly
focus on exploring policies with the most potential of being optimal. In [
In this paper, we described the di erent components needed to build environments to support epidemic control. First, we framed the problem of nding optimal policies as a reinforcement learning problem. Next, we focused on how to design environments, which are the components that characterise the epidemic problem setting in RL. We describe environments using the following components: disease model, intervention strategy, reward function, and state representation. Also, we mentioned challenges that need to be tackled for environment design and provided some examples.
We have shown that each of the environment components can be de ned
in a variety of ways. Components can also be combined in di erent ways to
express di erent types of problems. Concretely specifying a component requires
setting values to its parameters. This exibility allows us to represent numerous
epidemic scenarios but also brings a challenge related to reproducibility. If all
the details of how the environment was built is not shared, it is very hard to
replicate evaluation results. However, it is common for published works to miss
certain details of their environment design as there are many details that need
to be shared to entirely describe environments. A possible solution, that to our
knowledge has not been explored in the case of epidemic control, is to share the
exact environment de nition in a platform such as OpenAI Gym [
As a next step, we aim to review the di erent types of agents that can be developed to solve our problem of interest. Our overall goal is to propose a new solution to nd optimal health policies that can be used by decision makers. The work described in this paper is an attempt to de ne the environmental model suitable for representing an epidemic outbreak and the potential interventions that are available to decision makers. We hope that this work is useful to others that are also working on the same eld.
Acknowledgement. This publication has emanated from research conducted with the nancial support of Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 P2, co-funded by the European Regional Development Fund. 20. Organization, W.H., et al.: Pandemic in uenza risk management: a who guide to inform and harmonize national and international pandemic preparedness and response. Tech. rep., World Health Organization (2017) 21. Perlroth, D.J., Glass, R.J., et. al.: Health outcomes and costs of community mitigation strategies for an in uenza pandemic in the united states. Clinical Infectious Diseases 50(2), 165{174 (2010), http://dx.doi.org/10.1086/649867 22. Qualls, N., Levitt, A., Kanade, N., Wright-Jegede, N., Dopson, S., Biggersta , M., Reed, C., Uzicanin, A., Group, C.C.M.G.W., Group, C.C.M.G.W., et al.: Community mitigation guidelines to prevent pandemic in uenza|united states, 2017.
MMWR Recommendations and Reports 66(1), 1 (2017) 23. Rahmandad, H., Sterman, J.: Heterogeneity and network structure in the dynamics of di usion: Comparing agent-based and di erential equation models. Management Science 54(5), 998{1014 (2008) 24. Reed, C., Biggersta , M., Finelli, L., Koonin, L.M., Beauvais, D., Uzicanin, A., Plummer, A., Bresee, J., Redd, S.C., Jernigan, D.B.: Novel framework for assessing epidemiologic e ects of in uenza epidemics and pandemics. Emerging infectious diseases 19(1), 85 (2013) 25. Siettos, C., Anastassopoulou, C., Russo, L., Grigoras, C., Mylonakis, E.: Modeling the 2014 ebola virus epidemic{agent-based simulations, temporal analysis and future predictions for liberia and sierra leone. PLoS currents 7 (2015) 26. Stinnett, A.A., Mullahy, J.: Net health bene ts: a new framework for the analysis of uncertainty in cost-e ectiveness analysis. Medical decision making 18(2 suppl), S68{S80 (1998) 27. Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018) 28. Yaesoubi, R., Cohen, T.: Dynamic health policies for controlling the spread of emerging infections: in uenza as an example. PloS one 6(9), e24043 (2011) 29. Yaesoubi, R., Cohen, T.: Identifying cost-e ective dynamic policies to control epidemics. Statistics in Medicine 35(28), 5189{5209 (Dec 2016) 30. Yarmand, H., Ivy, J.S.: Optimal intervention strategies for an epidemic: A household view. Simulation 89(12), 1505{1522 (2013)