Computer systems that automate the making of decisions about people must be accountable to regulators. Such accountability requires looking at the operation of the software within an environment populated with people. We propose to use agent-based modeling (ABM) to model such environments for auditing and testing purposes. We explore our proposal by considering the use of ABM for the regulation of ad targeting to prevent housing discrimination.
The use of agent-based models (ABMs) can improve software accountability by representing populations of actors (e.g., house buyers, job seekers, college and insurance applicants) afected by software. Unaccountable software can embed and exacerbate societal biases or have other pernicious efects. ABMs can be used to explore the societal efects of software systems to aid the design and enforcement of regulations. Currently, software accountability measures are largely defined very narrowly, as mechanical compliance with regulations, written without a systematic way of predicting societal impact. ABMs are a way to model the societal impact of software and to correct or design regulations accordingly. We argue, in essence, that agent-based models (ABMs) of the interactions between software systems, those systems’ social environments, and applicable regulations can help to improve software accountability. We focus on software systems that employ personal data, have significant impacts on individuals, and are subject to regulation or used within regulatory agencies.
For example, anti-discrimination regulations in the housing context focus on societal goals such as fairness in housing or the reduction of residential segregation. Some of these regulations nEvelop-O LGOBE https://sbenthall.net (S. Benthall); https://www.icsi.berkeley.edu/~mct/ (M. C. Tschantz) are applicable to the software systems that target online housing ads. These systems, typically, are written not by regulators but by private sector software designers. The resulting deployed advertising systems may produce efects that are inconsistent with – even subverting – the regulatory intent. One possible reason for such an outcome is that the social environment in which the software is deployed is absent from the analysis of software impact. We want to include it and make it accessible to the regulators. If the social environment were represented, auditors could better anticipate whether the software is likely have the intended social efect. We propose to use ABMs to fill this gap, making software more accountable in this sense.
This sort of transformation has occurred in other fields. Infectious disease modeling is one. Faced with a novel pathogen, like Swine Flu or SARS-CoV-2, it is crucial to have some way to estimate how fast it may spread, how much vaccine to produce, whether to ban international travel or close schools and workplaces. Before the advent of infectious disease transmission models, doctors and public health oficial were operating in the dark. Now, we have disease simulation models at scales from the local to planetary for forecasting and mitigation. They are part of the fabric of public health decision making, and are used to inform policy at the CDC, NIH, WHO, and many national governments. They are not crystal balls and do not always agree, as in weather forecasting. But collectively, they can bound our uncertainties, estimate sensitivities, explore tradeofs, and ofer headlights in uncertain settings. Agent-Based Models specifically, which can include social networks and cognitive factors, are increasingly used. ABMs have the enormous advantage that they are also visual and rule-based (not equationbased) allowing non-technical audiences and domain experts to understand their results and, indeed, to participate in their construction. At the same time, they can be calibrated to epidemic data. The net efect of all this is that ABMs can have higher impact than complex diferential equation models. We are proposing to do this for software accountability.
We see potential in this method because ABMs are legible to software engineers, social scientists, and regulators. We have also identified key challenges to this approach, which are the potential politicization of model choice, the selection of appropriate robustness metrics, and the design of the interface between the ABM and audited software.
Software’s importance as an object of regulation is increasing as ever more individually and
socially significant private sector activities are automated. Controversial examples include the
use of automated decision tools in employment [
Applying regulatory standards created for human actors to automated systems may fail to
further the policy goals of the regulatory regime. Software poses critical challenges to traditional
policy and regulatory approaches to accountability, both as an object and as a tool of regulation
[
We posit that there are three elements to the design of accountable software systems. The ifrst two are the software system itself and the regulatory environment, including regulations, regulators, and their enforcement mechanisms. Our key insight is that the third, the social environment in which the system operates, is neglected in current software accountability techniques. The appropriateness of the software given a regulation, and in light of regulatory goals, depends not just upon the software’s behavior but upon its impacts on people in the social environment. Accountable software regulation thus requires expertise relating to all three elements and collaboration between software engineers, regulators, and domain scientists, as well as mechanisms for oversight by the legislature and ultimately the public.
ABM is already part of the policy-making toolkit [20]. ABMs may have additional advantages when software is a regulatory object or tool. Coglianese and Lehr [ 21] argue that as “regulation by robot”, or regulations designed and implemented using machine learning, become more common, other analytic techniques will need to be used to test them. In particular, they recommend the use of ABMs or multi-agent systems (MAS) to test the impact of regulations on the complex social environment.
By modeling the social environment, ABMs can capture regulatory concepts dificult to express in terms of software alone, such as people’s beliefs and purposes, and how regulations are often designed to be open to ongoing reinterpretation as new circumstances arise. We suggest that this method of testing the social impact of a software system can be more intuitive to regulators and domain experts than other means for projecting likely impact. ABMs can also more accurately account for social complexity and feedback loops, including unintended ones not readily apparent from the behaviors of individuals or the software acting in isolation. Furthermore, ABMs are implemented as explicit programs, making them intelligible to software engineers, thereby facilitating dialogue between them and regulators about policy goals and the interpretation and meaning of regulations. The dificulty of expressing the accountability goals of software in the traditional language of software testing, test suites expressed as the approved outputs for a given input, inspires the turn to ABMs.
As a concrete example, the goal of contact tracing is to slow the spread of a disease by identifying people who have likely been exposed to it and notifying them for testing, treatment, and/or isolation. Contact tracing may also be subject to policy making by public health departments, various privacy regulations, and protections of confidentiality. In the COVID-19 pandemic, many countries controversially turned to smartphone based contact tracing, despite public outcry in places about inappropriate surveillance. To understand whether an app is compliant with this regulatory environment requires information about the social environment. Policymakers may also seek to encourage or mandate the use of contact tracing. Software designed to be responsive to privacy concerns might increase the public adoption of the apps. ABMs can model this complex social environment, calibrated to empirical data to predict social outcomes, and provide information necessary to assess compliance. Because ABMs are software objects, they may be coupled directly with regulated software systems or model their social and behavior impact.
We picture, for example, a regulator that has been entrusted by the public and legislature (the “agenda setters”) with social goals in a particular domain. The regulator works with domain scientists, who provide a descriptive ABM that includes the entities important to regulations, such as actors and social outcomes. The regulator introduces normative elements into the model, which may be hard rules on agent behavior or objective functions to be tracked and optimized. This model can be used to analyze the intended and unexpected consequences of the regulation.
We propose a regulatory design process that uses ABMs to enhance accountability by spanning the boundaries between law, software, domain expertise, and ultimately, the public (Fig. 1 & 2). ABMs can be developed to simulate the social efects of such a private software system and thus evaluate its implications for regulatory goals. Regular reports to government agencies on the social impact of software systems (“Impact Assessments”), and reports responding to investigations, are an unsettled requirement of many laws governing software systems including the CCPA/CPRA, the GDPR, and the E.U.’s more recently proposed technology regulations such as the Digital Services Act, Digital Markets Act and Artificial Intelligence Act. There remain many open questions about the legal and technical feasibility of software that allows regulatory users to interact with and visualize results from the model. A key aspect of this workflow are methods for testing the robustness of the system’s compliance to social changes, modeled as both endogenous change within an ABM and as variation of the ABM’s performance over variations in its parameter space. These will be truly new capabilities.
The Fair Housing Act and related regulations prohibit certain discriminatory housing practices,
including ad targeting that would “deny particular segments of the housing market information
about housing opportunities” based on “race, color, religion, sex, handicap, familial status,
or national origin.”1 Prior to 2019, Facebook’s ad targeting options included targeting based
on Facebook-constructed attributes, including “ethnic afinity” (later renamed “multicultural
afinity”). Though based on users’ activities on the platform, “ethnic afinity” could obviously
have the efect of targeting ads unevenly across racial groups. In fact, Speicher et al. [ 22]
demonstrated that, even without using an attribute such as “ethnic afinity,” it was not dificult
to obtain biased targeting, intentionally or unintentionally, using Facebook’s targeting tools.
Related work has shown similar issues with Google’s advertising platform for race [23] and
gender [
One well-studied area of AI accountability is machine learning fairness, which has been developed in part to support the accountability of software to nondiscrimination laws. A result is that diferent fairness metrics are appropriate under diferent assumptions about bias in the data generation process [25, 26, 27]. There are also concerns about the “ripple efects” of automated decision-making, such that an automated decision becomes involved in a feedback loop with the social environment it’s acting upon [28]. With just access to the inputs and outputs of an ad targeting program, an auditor can merely detect bias in the presentation of advertisements (e.g., [23, 24]). Adding a calibrated ABM of housing choice, lets the auditor additionally predict how such biases play out in society, including feedback loops, as represented in the ABM. For example, the ABM could show how the biased ads increase segregation. We are not aware of any auditors combining software and ABMs in this way.
ABMs have, however, long been used to study housing segregation, beginning with models based on ethnic homophily and over time introducing models of market dynamics [29, 30, 31]. The Schelling Segregation Model [29] uses a square grid of residences, most of which house an agent. The main counterintuitive result of this model is that the relationship between the level of in-group preference and the amount of segregation in aggregate is non-linear. Even a weak preference for in-group neighbors can generate strong segregation. For example, with 2 groups and a 30% preference for in-group neighbors, the model will converge to near 75% of all neighboring pairs being of he same group. There are many variations on the Schelling segregation model, including more groups, variable preference rates, more realistic land topologies, and market dynamics [32].
We will develop a modified Schelling segregation model that includes the efects of advertising which may influence of the residential behavior of agents and the socioeconomic characteristics of neighborhoods.
The new modified model can be designed to include parameters representing the degree to which advertising is targeted based on (a) race and ethnicity, (b) location, and (c) socio-economic status. Measurements of interest will include metrics for segregation along similar axes. A contribution of the model will be that it can reveal the extent to which targeting based on, for example, socio-economic status contributes to segregation based on race and ethnicity. Rather than rely on point estimates to predict these efects, through multiple simulations the model can be tested over its entire parameter space. The results of this simulation can then be analyzed to determine efects and phase transitions on the dependent variables of interest (such as positive and negative rates on groups; positive and negative error rates; and so on).
We have identified several key challenges to the proposed approach. These pain points are each opportunities for future technical and policy research.
One challenge is designing and selecting the ABM for testing. Modeling the full complexity of the agents interacting with or afected by software may be beyond an auditor’s capabilities and knowledge. The auditor will have to make simplifying assumptions to produce an ABM. If the ABM is consequential for regulation, its details will entail diferent outcomes for diferent stakeholders. What process can improve the scientific accuracy and objectivity of the simulation results?
A related challenge is identifying the key metrics for translating ABM output into policyrelevant information. The output of an ABM can be very sensitive to the calibration of its input parameters, choices about its rules, as well as its own endogenous dynamics. Policy decisions should be based on robust outcomes of the model, rather than on brittle point estimates or single examples. The use of ABMs for software accountability requires well chosen metrics (e.g.,statistical indices of spatial segregation) for outcome robustness.
Lastly, we have identified a challenge at the technical interface between the audited software and the ABM. This part of the accountability process is likely to be least transparent to regulators, as it will be informed by the technical expertise of both the domain scientists and the audited system engineers. The choice of which system outputs are entered into the ABM is a sensitive one for which there is little preexisting guidance. For example, in our housing advertising example, the auditing mechanism requires measurements of the degree to which the advertising system discriminates with respect to race, location, and socioeconomic status. While it is possible to take these measurements using black box testing techniques with synthetic web user profiles, the resulting measurements are likely to vary with the technical specifics of the measurement technique. It would be better to have general guidelines and standards for how to design these interfaces between the audited software and the ABM.
Beyond these scientific and technical challenges is the fact that, in an area as politically fraught as fair housing, the politicization of model design (e.g., behavioral assumptions regarding diferent social groups) is a distinct possibility. The fact that all assumptions are explicit and all runs are replicable will help minimize this risk.
Agent-based models are a promising solution to one significant obstacle to software accountability: the modeling of the social environment of the software system. We envision a regulatory process that embeds ABMs to bridge the expertise of software engineers, regulators, and domain social scientists. In this article, we have developed one scenario for the use of ABMs for software accountability: the testing of advertising system’s efects on housing segregation. By developing this example, we have identified several key challenges to our proposed approach. We will address these challenges in future research.
This material is based upon work supported by the National Science Foundation under Grant Nos. 2131532, 2131533, and 2105301. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. One of the authors of this article is supported by the NYU Information Law Institute’s Fellows program, which is funded in part by Microsoft Corporation. [19] M. C. Tschantz, A. Datta, A. Datta, J. M. Wing, A methodology for information flow experiments, in: 2015 IEEE 28th Computer Security Foundations Symposium, IEEE, 2015, pp. 554–568. [20] S. Benthall, K. J. Strandburg, Agent-based modeling as legal theory tool, Frontiers in
Physics 9 (2021) 337. [21] C. Coglianese, D. Lehr, Regulating by robot: administrative decision making in the machine-learning era, Geo. LJ 105 (2016) 1147. [22] T. Speicher, H. Heidari, N. Grgic-Hlaca, K. P. Gummadi, A. Singla, A. Weller, M. B. Zafar, A unified approach to quantifying algorithmic unfairness: Measuring individual &group unfairness via inequality indices, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 2239–2248. [23] L. Sweeney, Discrimination in online ad delivery, Communications of the ACM 56 (2013) 44–54. [24] A. Datta, M. C. Tschantz, A. Datta, Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination, in: Proceedings on Privacy Enhancing Technologies (PoPETs), De Gruyter Open, 2015, pp. 92–112. [25] S. A. Friedler, C. Scheidegger, S. Venkatasubramanian, On the (im) possibility of fairness, arXiv preprint arXiv:1609.07236 (2016). [26] S. Yeom, M. C. Tschantz, Avoiding disparity amplification under diferent worldviews, in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, pp. 273–283. [27] S. A. Friedler, C. Scheidegger, S. Venkatasubramanian, The (im)possibility of fairness: Diferent value systems require diferent mechanisms for fair decision making, Commun.
ACM 64 (2021) 136–143. URL: https://doi.org/10.1145/3433949. doi:10.1145/3433949. [28] A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, J. Vertesi, Fairness and abstraction in sociotechnical systems, in: Proceedings of the conference on fairness, accountability, and transparency, 2019, pp. 59–68. [29] T. C. Schelling, Dynamic models of segregation, Journal of mathematical sociology 1 (1971) 143–186. [30] I. Benenson, I. Omer, E. Hatna, Entity-based modeling of urban residential dynamics: the case of yafo, tel aviv, Environment and Planning B: Planning and Design 29 (2002) 491–512. [31] A. Sahasranaman, H. J. Jensen, Ethnicity and wealth: The dynamics of dual segregation,
PloS one 13 (2018) e0204307. [32] E. Hatna, I. Benenson, The schelling model of ethnic residential dynamics: Beyond the integrated-segregated dichotomy of patterns, Journal of Artificial Societies and Social Simulation 15 (2012) 6.