Simulation is a useful tool for evaluating behavioral interventions when the adoption rate among a population is uncertain. Individual agent models are often prohibitively expensive, but, unlike stochastic models, allow studying compliance heterogeneity. In this paper we demonstrate the feasibility of evaluating behavioral intervention policies using large-scale data-driven agent-based simulations. We explain how the simulation is calibrated with respect to real-world data, and demonstrate the utility of our approach by studying the effectiveness of interventions used in Virginia in early 2020 through counterfactual simulations.
In the absence of pharmaceutical interventions, behavioral interventions are often the first line of
defense early in an epidemic. However, the effect of behavioral interventions can be unpredictable,
especially when the adoption rate of the population is unknown. While stochastic models are often
used to evaluate the costs and benefits of such interventions [
In this work, we demonstrate the feasibility of using large-scale data-driven agent-based
simulation for evaluating the effectiveness of behavioral interventions. In addition, we address
the concern of generalization of a population to the ‘standard human’ [
Between March and June 2020, nine subsets of the interventions shown in Table 1 were implemented in the form of Executive Orders (EOs) issued by the Governor of Virginia to mitigate the spread of COVID-19. The interventions are parameterized by the individuals affected and circumstances under which they apply. The timeline of these interventions is shown in Figure 1.
In previous work, we developed a large-scale data-driven normative agent-based simulation to
study the effects of compliance with these interventions [
A contagion simulator called PanSim simulates the progression of COVID-19 based on agents’
collocation [
To ground the model in real-world data, both the behavior model and the disease model were calibrated. For the behavior model, the parameters that were calibrated were the mean of the Beta distribution from which the agents’ trust is sampled, a discount factor termed fatigue and the start time at which fatigue comes into effect. Fatigue reduces the trust of all agents linearly starting from the start time, and models the gradual increase in mobility after the first intervention which is not associated with the deactivation of norms. For the disease model, the parameters that were remove goal each norm is applied to each trigger
modifies find plan for trigger
planschemes
For each plan application rule contexts conditions put plan on plan base deliberation start triggers / goals for each trigger & norm no applies yes
yes regimented no transform
trigger attitude >= θ < θ remove plan execute each adopted plan adopted plans external actions yes plan finished
plan produces yes goal achieved trigger & no belief conditions yes match? collected by
Sim-2APL after deliberation calibrated were the probability of symptomatic and asymptomatic infected agents infecting their susceptible contacts.
The behavior interventions had—by design—significant impact on mobility patterns. This was used to calibrate the behavior model using the mobility index: the percentage change in Cuebiq mobility data for each county compared to pre-pandemic levels. The mobility index of each of the four simulated counties in our calibrated simulation is plotted against those reported by Cuebiq in Figure 3a.
The disease model was calibrated using the cumulative number of reported cases. To account for under-testing at the start of the pandemic, we multiplied the number of reported cases by 30. While this scale factor is arbitrary, it can be changed without affecting the methodology of our work. The simulated case counts are plotted against the reported case count in Figure 3b.
For both calibration processes, Nelder-Mead [
As can be seen in Figure 3b, the disease model was not able to accurately reproduce either our estimate of the total number of infections or the time at which infections start to increase significantly. Other approaches to estimating the real number of cases, such as the number of 3Initial simulations indicated that the standard deviation is sufficiently small for a reasonable level of confidence; the purpose of using more than one simulation is to avoid a statistical outlier as a global minimum hospitalizations or deaths, may give better results. However, while the number of hospitalizations or deaths is generally assumed to be more accurate, like the number of positive tests, they are only a proxy for actual disease spread, and the effect of disease spread on these metrics needs to be known or calibrated as well.
In this section, we brieyfl show how our simulation framework can be used to help policy makers evaluate interventions through counterfactual simulations. For the interventions implemented in Virginia, we ran 10 experiments E0 . . . , E9. In E0, we disabled all interventions, in E1 we included only the interventions from EO0 (n1 and n4); in E2 we included the interventions from the first two EOs, in E3 the first three EOs, and so on up to E9 in which we simulate all interventions reflecting the actual policy implemented in Virginia. Note that, in each experiment, all activated EOs still follow the timeline from Figure 1. For each experiment we ran 10 simulations to account for stochastic variation. Figure 4 shows the average progression of the disease in each of the 10 experiments plotted against the reported (and scaled) number of cases. Compared to no interventions at all (E0), the policy implemented in Virginia (E9) can be considered effective in the sense that it reduced the number of infections in the first four months from ∼ 50% of the population to only ∼ 7%. Moreover, the results show that each additional set of interventions both reduced the number of infections and delayed the exponential spread compared to the previous set of interventions. The largest gain appears to be made by going from E3 (in yellow) to E4 (max group size of 10, stay home when sick, reduce business capacity to 10), and from E4 to E5 (Closure of all non-essential businesses, everyone asked to stay home as much as possible, restaurants closed except for take-away).
We showed the feasibility of using data-driven multi-agent simulation to evaluate the effectiveness of behavioral intervention policies. We explained how both the behavior and disease model components can be calibrated with real-world data, and demonstrated the utility of such a model through a simple experiment in which we quantify the effect of the 9 different sets of behavioral interventions that were implemented in Virginia in the early months of the COVID-19 pandemic through counterfactual simulations. Given the availability of suitable data, such as a synthetic population, the model can be calibrated and applied to different regions with different interventions or varying rates of compliance.
Our model is not able to correctly approximate the spread of disease, both temporally and in magnitude. In future work, we plan to both incorporate age-stratified infectivity and calibrate against better estimations of true infections, which may help improve the calibration and modeling of the spread of disease. This may also prove useful for studying the effects of interventions in regions with different age distributions.
We also plan to investigate the use of the Stringency Index [
We thank Cuebiq; mobility data is provided by Cuebiq, a location intelligence and measurement platform. Through its Data for Good program, Cuebiq provides access to aggregated mobility data for academic research and humanitarian initiatives. This first-party data is collected from anonymized users who have opted-in to provide access to their location data anonymously, through a GDPR and CCPA compliant framework. To further preserve privacy, portions of the data are aggregated to the US Census block group level.
PB and SS were supported in part by NSF Expeditions in Computing Grant CCF-1918656. policies (Oxford COVID-19 government response tracker), Nature Human Behaviour 5 (2021) 529–538.