Making Sense of Collaborative Challenges in Agent-based Modelling for Policy-Making

Abstract

The aim of this study is to analyze collaborations including agent-based modellers and policymakers to identify potential challenges that need to be overcome to facilitate simulation-based policy-making. To achieve this, we examined 18 publications reporting on joint projects where Agent-based modelling (ABM) was carried out in conjunction with modellers, policymakers, and other stakeholders to support policy-making. This study focuses on the challenges that modellers experienced during their collaboration e.g., disagreement about model specification, political obstacles, unrealistic expectations regarding the insights provided by ABM as well as the limitations of the models, and impatience of stakeholders when waiting for results. We identified and categorized these challenges into five themes: Challenges of Scope, Politics, Management, Understandability, and Credibility. These challenges were analyzed and used to formulate five recommendations, which are presented as a single approach that takes ethical considerations of policy modelling into account. So that these insights can be used to facilitate future simulation-based policy collaborations.

1. Introduction

Using Agent-based Models (ABMs) to simulate policy interventions – which is sometimes referred to as policy modelling – together with policymakers has become increasingly applied. Modelling and simulation collaborations can facilitate empirical calibration of ABMs as stakeholders often collect and store descriptive data concerning the target system, allowing local conditions and priorities to be considered. This allows policymakers to leverage ABMs as digital laboratories, where diferent policy interventions can be tested and compared. Recently, literature on ABMs has sought to inform modellers on how to work more efectively together with policymakers. To this end, warnings of pitfalls and advice for policy collaborations have been identified to help modellers [ 1, 2, 3 ]. However, much of this advice is derived from theoretical work, historical examples, or personal experiences. Therefore, consulting the literature could both improve and expand the evidential base for the guidelines used in simulation-based policy collaborations. The current literature review aims to address this by identifying reported challenges in simulation-based policy collaborations that apply ABM.

ABMs have traditionally had high scientific value but have often been less useful for practitioners due to inappropriate assumptions or overly complex models. To combat this, an increasingly applied strategy is to include stakeholders with a common cause to solve a specific problem. However, questions remain about how to best structure modelling and simulation collaborations between stakeholders and modellers[ 4 ]. In policy modelling, collaborative aspects have often been overlooked. However, more recently there has been some exceptions where the collaboration has been the main focus [ 5, 6 ]. This literature indicates a need for a deeper understanding of how to use ABMs in policy-making. Some relevant branches of research include defining formal requirements for policy modelling [ 7, 8 ], ethical practices [ 2 ], and communication strategies for presenting model results to policymakers [ 9, 10 ]. Policy modelling collaborations can be resource-intensive and have the potential for significant societal impact, making insights from such collaborations valuable. While successful collaborations have the potential to facilitate evidence-based policy-making – where scientific methods are integrated into the policy-making process – to formulate tailored policy solutions [ 11 ], scientific policy collaborations have also caused detrimental societal and environmental repercussions by producing erroneous evidence [ 3 ]. Thus, documenting the collaborative challenges and cementing best practices should be a top priority within the field of policy modelling.

The objective of the present study is to systematically categorize collaborative challenges of policy modelling through a comprehensive literature review and provide recommendations to address them. With the aim of considering how ethical responsibilities should be divided between modellers and policymakers, and how simulation-based policy collaborations could be executed to avoid challenges. The following research questions are posed: What are the potential collaborative challenges that may arise when utilizing Agent-based Models in policymaking? What ethical considerations should be taken into account when constructing and simulating policy models for the purpose of policy-making? What strategies can be employed to overcome the collaborative challenges in policy modelling? The methodology, search strategy, and selection criteria for the study will be described in the following section. Followed by a thorough examination of each of the identified challenges in the coming sections: Challenges of Scope, Politics, Management, Understandability, and Credibility. Then there will be a discussion analyzing all of the reported challenges to propose five recommendations in section eight. In the next section (nine), we propose a collaborative approach which takes all the previous recommendations into consideration. The study ends with concluding remarks in section 10.

2. Methodological approach

We conducted a literature review that examined the collaborative challenges of simulationbased policy projects involving Agent-based modellers and policymakers. Using the Scopus database, we searched for articles using a three-layer search string that focused on the model, the collaborative activity, and its intended policy focus.1 This study aimed to identify collaborative challenges in simulation-based policy collaborations by conducting a literature review. The search generated a list of 199 articles, from which 168 were excluded for not applying ABM or not including policymakers in the project. We also excluded articles that included multiple 1(TITLE-ABS-KEY ( agent-based ) OR TITLE-ABS-KEY ( individual-based ) OR TITLE-ABS-KEY ( policy- model ) OR TITLE-ABS-KEY ( computational-model ) AND TITLE-ABS-KEY ( collaboration ) OR TITLE-ABS-KEY ( support ) OR TITLE-ABS-KEY ( participatory ) OR TITLE-ABS-KEY ( companion ) AND TITLE-ABS-KEY ( stakeholders ) OR TITLE-ABS-KEY ( decision-makers ) AND TITLE-ABS-KEY ( policy ) OR TITLE-ABS-KEY ( policies ) ) AND ( LIMIT-TO ( LANGUAGE , ”English” ) ) simulation applications such as project reports or literature reviews. It is important to note that the use of field-specific umbrella terms like ”stakeholders” could result in type-II errors, where projects that include policymakers may be excluded. To mitigate this, surveys or interviews with modellers could be conducted in future studies. The remaining 31 articles were read in full to establish whether they fulfilled the inclusion criteria, which required that the article report any collaborative challenges. A total of (n=18) articles met these criteria. The process is illustrated in the flowchart below.

We refer to collaborative challenges as reported issues which could impede a simulation-based policy project. Thus, purely technical challenges are excluded as these may be very specific in nature, relating to modelling choices, programming platforms, and data. This decision is justified on the basis that the purpose of the current literature review is to report collaborative challenges, which could be reformulated to useful guidelines for other modellers seeking to engage in simulation-based policy collaborations. We extracted and categorized all the challenges reported in each paper into five diferent categories: Scope, Management, Politics, Understandability, and Credibility. These categories do overlap and are interconnected, for instance, understandability is often described to directly afect credibility. While we could combine these categories into a broader category called ’interpretation,’ doing so would result in the loss of valuable information. Therefore, we have strived to strike an appropriate balance between abstraction and specificity, in order to make the categories easily understood and communicated. To facilitate communication and reduce repetitiveness, each of the challenges is abbreviated accordingly [CoC:t-ID] to indicate the challenge of the class, the type of challenge and its ID number.

3. Challenges of Scope

Challenges of Scope have implications on the width of model scope and length of the project: what aspects to include or omit in the model; and the duration of the project are critical in any simulation-based policy collaboration. This literature review reports that many challenges concern this type of delineation problem. We have categorized these challenges as challenges of scope.

Occam’s razor provides a good starting point, meaning that a model should not be more complicated than necessary [ 12 ]. While modelling collaboratively with stakeholders is a step towards the KIDS (keep it descriptive stupid) approach, contrary to less complicated models and the KISS (keep it simple stupid) approach. The line between what is necessary and superfluous is seldom clear-cut and relates to the stakeholder’s perceptions of the target system [ 13 ] and the purpose of the model [ 14 ]. There are two reported types of challenges that serve to determine what is included and omitted in a simulation-based policy collaboration, negotiations, and resource constraints. These two types of challenges influence two diferent aspects in the collaboration: the model and the project. These could also be conceptualized as afecting the width of the scope i.e., what is included or not, and the length of the scope i.e., the time-frame of the project.

There are several reasons why diferent stakeholders may have difering perceptions of the target system (CoS:n1). In the absence of scientific evidence, there could be many diferent legitimate competing explanations for how a system operates [ 15 ]. Also, policymakers might agree on how the system operates descriptively speaking; but still have radically diferent perceptions of the role that the system ought to fulfil. Van Berkel and Verburg provide an example of how Dutch agricultural policymakers are split about the role that the local ecological system should fulfill: “One segment of the workshop participants viewed agriculture (n = 9) as key to future rural functionality while another saw nature services and high quality living as more important (n = 5)” [ 16 ]. The disagreement between the policymakers is not necessarily found in how the system functionally operates but in diferent perceptions of it, and the conclusions drawn from them.2

These diferences in perception could permeate to other aspects of the simulation project, such as (CoS:n2) having diferent priorities and political agendas, or (CoS:n3) disagreement about the research objective. Something that Ahrweiler and colleagues report during their work with the European Commission on research and innovation policy: “Stakeholders neither shared the same opinion about what questions should be in the final sample and how potential questions should be ranked in importance, nor shared the same hypotheses about questions in the final sample” [ 5 ]. This resulted in time-consuming and demanding negotiations (CoS:n4) which could potentially be even more challenging to moderate in the face of political contention and controversy (CoS:n5). Thus, perceptions and views afect the scope of the project and the model – something that is reafirmed in the following section. This can also be exacerbated by the following challenge, where modellers and policymakers can have dificulties communicating with each other (CoS:n6). The diferent contexts in which each vocational group operates are quite distinct from one another, meaning that negotiations of scope can be quite laborious for both parties.

There is documented disagreement about essentially all modelling aspects of simulation-based policy projects. This is not surprising given that the formalization of an ABM can be understood as a design problem. This means that there is inherent flexibility in each model specification and that each modelled system can be formalized in multiple ways [25]. Diferent values and perceptions of the target system can afect how the problem is formulated and the scope of the investigation, such as determining which parameters to include (CoS:n7). For instance, in the Dutch study with agricultural policymakers, one of the participants advocated for more efective and innovative interventions (CoS:n8) in the region, while others prioritized maintaining the agricultural landscape as a part of the region’s identity [ 16 ]. Delmotte and colleagues report a similar experience concerning scenarios (CoS:n9): “The stakeholders disagreed on the scenarios to be assessed and did not share interest for the others’ scenarios. We therefore choose to work in parallel with each stakeholder to assess their scenarios of interests” [22].

The development of models for policymaking is a complex process that involves various negotiation challenges. Diferent stakeholder perceptions of the system’s purpose may lead to varied approaches to development, interventions, model modifications (CoS:n10), scenarios, agent behaviour (CoS:n11), and the metrics used to evaluate policy objectives (CoS:n12). The last negotiation challenge is unfeasible model requests (CoS:n13), again from Ahrweiler and colleagues “here again, we encountered the diversity of stakeholder preferences. Diferent members of the DG INFSO Steering Committee opted for diferent changes and modifications of the model. Some were manageable within given time constraints and financial resources; some would have outlived the duration of the project if realized” [ 5 ]. Another challenge (CoS:rc1) relates to the resource constraints faced by policymakers. This can put modellers under considerable pressure to deliver within short time frames. Something which can make collaboration intensive approaches like companion modelling dificult. 2There are ample of examples like this from literature outside of the scope of the current literature review. For example, free-marketers and anti-consumerists can agree that capitalism are driven by self-interest, while disagreeing on how the system ought to operate [23]. Similarly, the perceptions of cleanliness of using vultures for carcass disposal stations is tied to the perceptions of death, nature, and the vultures themselves, rather than how the carcass disposal works functionally speaking [24].

4. Challenges of Politics

Simulation-based policy collaborations can be highly efective tools for policymakers to evaluate and address complex policy issues. However, such collaborations could also be subject to significant political challenges that should be acknowledged. These challenges pose questions like: how do the political institutions influence the outcome of the project; and what problems can arise when reformulating the model results in formal policy? Hence, these are challenges that are all tightly interlinked with the core concept of policy modelling [ 2 ]. The first type of challenge involves understanding how the influence of political institutions can afect the outcome of the project. This includes considerations of how formal institutions and norms within the political setting can impact collaborations [26]. The second type of challenge relates to how model results can be practically applied to the policy problems. These challenges reportedly afect two aspects of the collaboration: the project itself or the public policy to be formulated.

The first three political challenges arises from a conflict between the expectations of the traditional policy advisory role and new, exploratory techniques – also pointed out in other literature [ 2, 9 ]. As observed in one of the projects where modellers proposed continuing participatory modelling meetings with policymakers, but the policymakers preferred to end the meetings and stick to the traditional expert-driven approach, where results are presented during public meetings (CoP:i1) [27]. Additionally, elected oficials have been reported to be unlikely to directly interact with the models, instead relying upon others to communicate this information (CoP:i2). This can be a challenge for participatory simulation approaches, as the insights generated from these sessions need to be communicated to the decision-makers before being implemented. Hoch and colleagues reported that policymakers are more accustomed to using evidence derived from traditional statistical frameworks and tend to favor model predictions (CoP:i3): “Many found it dificult to consider using a model that did not ofer expert predictions about the ‘facts’, and required them instead to test assumptions and expectations interactively” [28].

The fourth political challenge (CoP:i4) is related to ineficiencies in the existing administrative system and emphasizes the significance of the political context. In water management in Spain, numerous administrative overlaps that result in conflicts and hinder the achievement of common goals [ 17 ]. Similarly, incorporating farmers in an agricultural simulation-based policy collaboration in Ethiopia was challenging due to the top-down approach to policy-making [29]. Another reported challenge, stemming from a bottom-up approach to modelling and simulation, is the loss of endorsement from policymakers due to the end of their mandate (CoP:i5). Also, policymakers can ask modellers to provide policy advice (CoP:i6). Ahrweiler et al write: ”The stakeholders wanted the study team to communicate the results as ‘recommendations’ rather than as ’findings’” [ 5 ]. This can put modellers in an awkward position as they are not only asked to provide endorsements for certain policy prescriptions (which the modellers themselves underscore). While modellers should be free to ofer policy advice according to their own judgement. If modellers fulfil a facilitative role to aid stakeholders with the framing the model and the choice of policy interventions. This practically means that the modellers are choosing to endorse policy interventions based on the cumulative modelling decisions of their stakeholders.

The final three political challenges of modelling arise when model results are to be converted into public policy, which can be met by several impediments. The first challenge is unrealistic expectations when modellers are asked to fix a task that is not solvable. Put diferently, the viewpoints of the participants concerning the problem can be incompatible no matter how sophisticated the method might be (CoP:r1) [ 18 ]. The second challenge is a lack of public support for policies that may have promising results from the model (CoP:r2). For example, in water management Zellner and colleagues note: “[…] most stakeholders recognized that individual water users would need to change their behaviour and consume less water (self-interest) to preserve the resource for the region (collective interest). However, when asked if they, as residents of the county, would pay for water, some of the respondents resisted: ‘I don’t know. You would have to convince me that the worth of water is such that I should have to pay’” [27]. Similar observations have also been documented regarding the attempts to curb climate change, pollution, and congestion by increasing the costs of car use [ 19 ].

In addition to policy interventions being publicly unpopular, they can also be politically inconvenient (CoP:r3). This is reported from another water management project in Chicago: “[…] participants exhibited resistance to using ABM that challenged the conservation strategies in the groundwater plan. The participants closed ranks as many came to realize that the simulated efects of favoured policies would not meet the plan sustainability goals” [ 28]. Another case reports similar observations: “stakeholders actively distorted the practical meaning of the simulation results and thus reinforced the existing viewpoint rather than transformed it. For instance, instead of using the simulation results—which had been collectively validated—to reconsider their commitment to continued growth and expansion, two stakeholders shifted temporal scales to argue that had WRAP measures been applied decades earlier, current depletion risk would be much less” [27]. In summary, political challenges can significantly influence the outcome of a simulation-based collaboration, even when interactive aspects of policy collaborations and challenges of scope are put aside.

5. Challenges of Management

The management challenges in simulation-based policy collaborations refer to the questions of how to efectively manage the collaboration process itself: such as managing stakeholder interests; determining meeting locations; and managing dissenting stakeholder groups. Some of these challenges are purely administrative, such as how and where meetings between modellers and stakeholders should take place. Others involve managing stakeholder engagement to ensure the success of the project. Emphasis is put on the project itself here, as all challenges of management directly afect the practical aspects of the project. Challenges concerning engagement arise from deviations in balanced stakeholder involvement throughout the simulation project. While administrative challenges concern how and where meetings between modellers and stakeholders will be held.

Maintaining stakeholder engagement can be challenging, as indicated by several reports (CoM:e1). This can be especially true in political contexts where policymakers are used to scientific policy collaborations to the extent that it becomes routine. “In the Dutch context, local policymakers are often required and/or frequently requested to join diferent (sciencepolicy) workshops as stakeholders of their policy field […] Repeated interaction with nature organisations, scientists and other policy bodies in these exercises can stimulate innovation, but also result in a situation where workshops become a routine for participants. Combating apathy caused by common workshop procedures and results is an important consideration in workshop design” [ 16 ]. This highlights the need for careful planning and efort to ensure the successful execution of policy collaborations.

Stakeholder engagement can decline not only in general, but also occur within specific groups (CoM:e2). This decline have been reported in stakeholder groups that are less afected by the policy problem [32]. This can also afect the unrepresentativeness of stakeholder composition (CoM:e3) which may have diferent requirements depending the specific project. The challenges of inclusion and commitment of higher organizational levels (CoM:e4-5) further hinder successful collaboration. Both these observations originate from bottom-up approaches to modelling and simulation: “[…] decision makers at district, province and national levels, were not easily mobilised to contribute to participatory processes of land-use planning in which local communities were the key players” [33], and “An important methodological challenge is to get more commitment of stakeholders from higher organizational levels. The objective of companion modelling is to build a bottom-up process of institution development that goes beyond the expert-government” [30]. Thus, while the bottom-up approaches can legitimize the policy project, they may also impede engagement from policymakers.

Determining appropriate meeting arrangements between policymakers and modellers to facilitate collaborative modelling presents the first administrative challenge (CoM:a1). Another challenge is scheduling conflicts, which can make collaborations dificult (CoM:a2): “While care was taken in the selection of stakeholders, scheduling conflicts and interest level limited our flexibility in dictating stakeholder composition” [ 16 ]. Additionally, as the project progresses, it may be necessary to identify and include additional stakeholders (CoM:a3). Working with dissenting stakeholder groups in parallel was the last reported administrative challenge (CoM:a4) in policy modelling projects, as the groups had conflicting interests in the scenarios to be investigated [22]. While splitting the groups into two is a potential solution, it requires significant planning, labor, and resources. Thus, both stakeholder engagement and administrative aspects can pose challenges to project management and afect the success of collaborative modelling eforts.” project.

6. Challenges of Understandability

The challenge of understandability poses several important questions like: how technically involved should stakeholders be; how can models and results be communicated to policymakers in a understandable way. The second question is tightly interlinked to aspects of explainability where the aim is to open the black box to understand the inner operations of a model from an end-user perspective [34]. While the developers of simulations should understand how their model functions (except for ABMs using truly opaque methods), meaning that the operations or equations of the system are not hidden, policymakers may perceive the model as a blackbox. The essential aspect of policy modelling is to ensure that policymakers understand the conclusions that can safely be derived from a specific model. If policymakers make conclusions that go beyond the scope of the model, they should be able to recognize that these conclusions are based on their assumptions. Furthermore, understandability challenges also afect the model’s perceived credibility – as briefly noted in the methodological discussion. The challenge of understandability is widely reported and often stems from technical aspects of the modelling and simulation project.

The first statement concerns the first two challenges, stakeholders being uncomfortable using computers (CoCo:t1) and programming (CoCo:t4): “Stakeholders expressed discomfort using computers, let alone manipulating the code, and relied heavily on the moderators to guide them and make changes to the models” [27]. This suggests a possible mismatch between the technical depth of the activities and the stakeholders’ competencies.

Similarly, the challenges of understanding the software interface (CoCo:t2) and the model (CoCo:t3) are captured by a statement by Hoch and colleagues: “Some participants experienced continued dificulty using the interface and understanding how the diferent ABM functioned” [28]. However, understanding the model (CoCo:t3) is one of the most commonly reported challenges, which is closely related to the topic of the next section, credibility. As Ahrweiler and colleagues point out: “The stakeholders put considerable demands on the study team concerning understanding and trusting the simulation findings. The first and most important is that the clients want to understand the model. To trust results means to trust the process that produced them” [ 5 ]. Similarly, concerning optimization routines (CoCo:t5), stakeholders found it dificult to understand how a utility function accounting for diferent combinations of land in an agricultural model afect the trade-of curves for diferent pieces of farmland. Delmotte and colleagues write: “stakeholders find their optimizing routines dificult to understand, which therefore limited in some cases stakeholders’ confidence in validity of the results” [ 22]. Hence, the stakeholders’ competence in technical aspects of the project could impact the perceived quality of a model.

7. Challenges of Credibility

The challenge of credibility is important for the step between model output to policy prescription. If policymakers do not find the model or its results credible it is unlikely that it will be applied to inform public policy – rightfully so. Therefore, it is essential to address questions such as; what factors contribute to a model being perceived as less credible; and how well the model represents the target system [35]. While many modellers use the term trust to refer to stakeholder confidence in a model and its results, we use the term credibility as trust implies the possibility of reciprocity. Accordingly, in this line of reasoning, agents can trust each other but objects such as models are credible. Credibility is not only a matter of validity but also of end-user perception, as a model might be valid but still be perceived to lack credibility, or vice versa. While many challenges of credibility arise from the model, the project itself also plays an important role. This was partly underscored in the previous section, where a lack of understandability was reported to negatively afect the credibility of the model.

The first two credibility challenges relate to the quality of the data used in the model (CoCr:m1) and the calibration of the model (CoCr:m2). Modellers have reported that stakeholders can be sceptical of metrics (CoCr:m3) and tools (CoCr:p2) in the project. Indicating the importance to critically reflecting on the weaknesses and appropriateness of these aspects in collaboration with stakeholders. The following challenges concern model output. If stakeholder’s perceive the results of the model to be inadequate or inaccurate the collaboration can grind to a halt (CoCr:m4). In a water management project from 2005, stakeholders expressed scepticism about promising model predictions because they were not convinced about the capability of models to produce predictions (CoCr:m5) [36]. More recently, models are often used for exploratory purposes rather than prediction as recent work has demonstrated the dificulty of this feat [ 37, 3 ]. Indiferent of the purpose, being clear about the uncertainty of model cannot only help to convey its limitations, but also help to identify potential areas of improvement. Being clear in communication is also important to ensure that modellers take their ethical responsibility and are nuanced in how information is relayed to their partners.

The next challenge emphasizes the importance of understandability in establishing the credibility of a model (CoCr:m6). Ahrweiler and colleagues note that: “The stakeholders put considerable demands on the study team concerning understanding and trusting the simulation ifndings. The first and most important is that the clients want to understand the model. To trust results means to trust the process that produced them” [ 5 ]. Late stakeholder involvement in a project (CoCr:p1) can also pose a challenge to achieving this level of understanding and credibility. Another challenge, providing returns for end-users (CoCr:m7), can be dificult to assess. Not only are the project returns judged by the policymakers and other stakeholders involved in the project, but they are also contingent on their expectations of the project. While insights which could help to better formulate policy prescriptions should be understood as an important return. If policymakers in accordance with the challenge (CoP:i8), expect that modellers will provide them with policy advice, such insights could simply be understood as a by-product. Thus, making sure that expectations align can also give policymakers a better understanding the intended returns of the project.

8. Discussion

This section highlights the documented challenges that can arise when constructing and simulating policy models. It also ofers five recommendations to mitigate these challenges. While some of the recommendations are already practiced, our aim is not to innovative but to establish what works, meaning that this to some extent re-afirms the conclusion of others. These recommendations are intentionally designed to be general in purpose and we believe that they could be well applied to any simulation-based policy collaboration. Furthermore, while some modellers have suggested that policy modelling is an ethically and politically neutral process [ 2 ], the applied literature indicates otherwise. Disagreements can arise from all aspects of modelling in simulation-based policy collaborations, such as perceptions of the target system (CoS:n1), priorities (CoS:n2), research objectives (CoS:n3), interventions (CoS:n8), scenarios (CoS:n9), model modifications (CoS:n10), agent-behaviour (CoS:n11), and metrics (CoS:n12). It is evident that policy modelling becomes an integral part of the political process when performed for the purpose of policy-making. It is important to underscore this for two reasons: democratic accountability on the part of the policymakers but also ethical responsibility for modellers as scientific advisors. However, modellers must still be involved in deliberations to play a facilitative role and make sure that requests are technically feasible within the limitations of the project (CoS:n13) so that the collaboration does not grind to a halt.

Recommendation 1: Modellers can adopt a facilitative role by prioritizing policymaker input during the model design, while minimizing their own influence .

Many of the reported challenges often stem from uncertainties and diferences in expectations between modellers and policymakers. To avoid such challenges, it is recommended to establish a project plan early on. An initial plan can be proposed, renegotiated, and agreed upon to align the expectations and prevent misunderstandings. As policymakers tend to prefer traditional scientific advisory roles (CoP:i1-3), an overview of the project plan can help gain acceptance for deviations from the norm. Key aspects such as project duration, number of planned meetings, scheduled times and locations (CoM:a1-2), and project output (CoCr:m7) should be included in the project plan to align the expectations of both modellers and policymakers. Something that can also serve to make commitments explicit so that issues of engagement are reduced (CoM:1-5). Clear division of responsibility and shared goals could improve communication between modellers and policymakers (CoS:n6), while avoiding unaligned expectations that produce situations where modellers are expected to recommend policy prescriptions (CoP:i6).

Recommendation 2: Agree on a project plan as early as possible so that expectations are aligned and commitments are made.

When planning collaboration activities, it is essential to consider their technical depth and alignment with the project’s purpose. As stakeholder understandability is crucial for the model’s perceived credibility, these activities should be purposefully planned to ensure stakeholder involvement fulfills a predefined goal. Policymakers reportedly feel uncomfortable with many technical aspects of modelling and simulation (CoCo:t1-5), making it all the more important for projects focused on policy formulation to ask stakeholders for their preferences to ensure the activities are useful for them. Modellers can simultaneously improve model understandability and credibility (CoCr:m7) by being open to feedback, promoting open discussions of the model’s strengths and limitations after each update, and communicating results clearly and nuanced. It is vital not to include policymakers too late in the project (CoCr:p1) as this can hinder their learning process and limit their impact on the model’s scope, making it less relevant for them. Thus, an important aspect for future research of policy modelling is to explore if, and under which conditions, predefined models could be useful. Here we re-afirm recommendations provided elsewhere, see for example Seifu and colleagues [20].

Recommendation 3: Include policymakers early in the project and tailor activities to their needs.

Recommendation 4: Communicate model capabilities, limitations and results – comparatively between each model version if possible – in a clear and nuanced way.

Something else that is concluded elsewhere is that policy modelling projects benefit from a quick and agile methodological approach [ 5, 2 ]. Heavy budget and time restrictions of policymakers can make collaboration-intensive approaches like companion modelling more dificult to apply (CoS:rc1). While bottom-up modelling approaches have the potential to legitimatize policy prescriptions and democratize decision-making processes, they pose challenges for including and maintaining policymaker engagement (CoM:e4-5), as well as navigating institutional changes such as ending mandates (CoP:i5).

While broad stakeholder involvement is desirable, potential complications (CoM:a3-4) should not make the projects less useful for policymakers. Some policies require a top-down approach as the stakeholder groups are too difuse to be meaningfully involved e.g., transportation systems, global warming, epidemiological modelling etc. However, this conclusion is not agnostic to the political context and the policy issue at hand. Something that needs to be considered on a case-to-case basis so that policy modelling does not become a tool to disenfranchise the very people that it seeks to serve - meaning the general public. While modellers have limited influence over inefective pre-existing administrative systems (CoP:i4). Their existence should motivate not discourage modellers, as they are one of the motivations for why policy modelling is needed. Furthermore, there are also situations where the model results can be less useful for purely political reasons (CoP:r1-3), something which is out of the control of scientific advisors. To ensure a successful collaboration, it is important to recognize and respect that each party plays a critical role in fulfilling their respective responsibilities throughout the process.

Recommendation 5: While broad stakeholder participation is worth striving for, the collaborations needs to remain quick and agile to maximize its usefulness for policymakers.

9. Towards a Unified Approach for Simulation-based Policy

Collaborations

To address the previously accounted for challenges, we suggest the following approach by combining the recommendations from the previous section for simulation-based policy collaborations: It is recommended to invite policymakers as early as possible in the project and to prepare an initial draft of the project plan. Involving policymakers early serves to ensure that they get the opportunity to learn and understand the model which is important for its credibility. Updated model versions provides a good opportunity to efectively communicate and compare the previous model version to the new. Something which can facilitate communication of model capabilities, limitations and results in a nuanced and clear manner.

The initial project plan should not be thought of as a static document, but something to be developed and agreed upon collaboratively. This activity is aimed at aligning the expectations of both parties and ensuring that commitments are made. The project plan could include information like project duration, number of planned meetings, scheduled times and locations, the output of the project, and be agreed upon as soon as possible. This ensures that both parties are working towards the same goal and can facilitate communication. Planning the scope of the model should be given ample time as it involves political consideration. Thus, allowing stakeholders to negotiate internally should be seen as an important ethical aspect of policy modelling. The modeller’s role is to facilitate and maximize policymaker influence during model design, while limiting the scope only due to technically unfeasible requests.

Policymakers have reported disliking the technical aspects of modelling and prefer the traditional scientific advisory role. The project plan could allow modellers to make potential deviations from status-quo while avoiding confusion about the project process or outcome. Furthermore, while broad stakeholder participation is worth striving for, the project should be fast and agile to maximize its usefulness for policymakers. However, the political context and the policy issue at hand must always be considered on a case to case basis. As the main purpose of policy modelling should always remain to serve the good of the people. 10. Concluding Remarks We have in this study gathered and synthesized the reported challenges from 18 articles which have sought to apply ABMs for policy-making. These challenges were categorized into the ifve following themes: Challenges of Scope, Challenges of Politics, Challenges of Management, Challenges of Understandability and Challenges of Credibility. To address these challenges, we proposed five tentative recommendations and suggested an approach for simulation-based policy collaborations. Additionally, we propose including any and all sound advice ofered by other modellers with the insights ofered here to increase the probability of successful outcomes. We do, however, stress the importance of the ethical considerations that the suggested approach above takes into consideration. Our hope is that these insights can aid both Agent-based Modellers and policymakers in avoiding dificult challenges that may arise during simulationbased policy collaborations.

During times of increasing uncertainty, which stifle the possibility to produce experimental data, policymakers face increasingly dificult challenges in making informed decisions that benefit society. By providing policymakers with the necessary tools, resources, and training, Agent-based Models can be incorporated into the policy-making process so that some of these challenges can be overcome. This can lead to the development of evidence-based policies that are better suited for complex systems and produce more favorable outcomes for us all.

Acknowledgments

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program – Humanities and Society (WASP-HS) funded by the Marianne and Marcus Wallenberg Foundation and the Marcus and Amalia Wallenberg Foundation.

We would like to express our gratitude to the reviewers at Agent-based Modelling & PolicyMaking for their valuable feedback and insightful comments on our work. Their constructive criticism helped to enhance the overall quality of our paper. [20] L. Seifu, C. Ruggiero, M. Ferguson, Y. Mui, B. Y. Lee, J. Gittelsohn, Simulation Modeling to Assist with Childhood Obesity Control: Perceptions of Baltimore City Policymakers, Journal of Public Health Policy 39 (2018) 173–188. [21] P. G. Johnson, Agent-based Models as “Interested Amateurs”, Land 4 (2015) 281–299. [22] S. Delmotte, J.-M. Barbier, J.-C. Mouret, C. Le Page, J. Wery, P. Chauvelon, A. Sandoz, S. L. Ridaura, Participatory Integrated Assessment of Scenarios for Organic Farming at Diferent Scales in Camargue, France, Agricultural Systems 143 (2016) 147–158. [23] A. Goldberg, S. K. Stein, Beyond Social Contagion: Associative Difusion and the Emergence of Cultural Variation, American Sociological Review 83 (2018) 897–932. [24] H. Dupont, J.-B. Mihoub, S. Bobbé, F. Sarrazin, Modelling Carcass Disposal Practices: Implications for the Management of an Ecological Service Provided by Vultures, Journal of Applied Ecology 49 (2012) 404–411. [25] A. R. Hevner, S. T. March, J. Park, S. Ram, Design Science in Information Systems Research,

Management Information Systems Quarterly 28 (2008) 6. [26] D. C. North, Institutions, ideology, and economic performance, Cato J. 11 (1991) 477. [27] M. L. Zellner, L. B. Lyons, C. J. Hoch, J. Weizeorick, C. Kunda, D. C. Milz, Modeling, Learning, and Planning Together: An Application of Participatory Agent-based Modeling to Environmental Planning, Urisa Journal 24 (2012) 77–93. [28] C. Hoch, M. Zellner, D. Milz, J. Radinsky, L. Lyons, Seeing is Not Believing: Cognitive Bias and Modelling in Collaborative Planning, Planning Theory & Practice 16 (2015) 319–335. [29] W. Daré, J.-P. Venot, C. Le Page, A. Aduna, Problemshed or Watershed? Participatory

Modeling towards IWRM in North Ghana, Water 10 (2018) 721. [30] K. Worrapimphong, N. Gajaseni, C. Le Page, F. Bousquet, A Companion Modeling Approach

Applied to Fishery Management, Environmental Modelling & Software 25 (2010) 1334–1344. [31] T. Berger, C. Schilling, C. Troost, E. Latynskiy, Knowledge-brokering with Agent-based

Models: Some Experiences from Irrigation-related Research in Chile (2010). [32] C. Simon, M. Etienne, A Companion Modelling Approach Applied to Forest Management

Planning, Environmental Modelling & Software 25 (2010) 1371–1384. [33] J.-C. Castella, P. H. Verburg, Combination of Process-oriented and Pattern-oriented Models of Land-use Change in a Mountain Area of Vietnam, Ecological Modelling 202 (2007) 410–420. [34] T. Miller, Explanation in Artificial Intelligence: Insights from the Social Sciences, Artificial

Intelligence 267 (2019) 1–38. [35] B. S. Onggo, L. Yilmaz, F. Klügl, T. Terano, C. M. Macal, Credible Agent-based Simulation– An Illusion or Only a Step Away?, in: 2019 Winter Simulation Conference (WSC), IEEE, 2019, pp. 273–284. [36] A. López-Paredes, D. Saurí, J. M. Galán, Urban Water Management with Artificial Societies of Agents: The FIRMABAR Simulator, Simulation 81 (2005) 189–199. [37] T. Martin, J. M. Hofman, A. Sharma, A. Anderson, D. J. Watts, Exploring Limits to Prediction in Complex Social Systems, in: The 25th International Conference on World Wide Web, 2016, pp. 683–694.

References

[1] B. Edmonds , C. Gershenson , Modelling Complexity for Policy: Opportunities and Challenges , in: R. Geyer, P. Cairney (Eds.), Handbook on Complexity and Public Policy , Edward Elgar Publishing, 2015 , pp. 205 - 220 .

[2] N. Gilbert , P. Ahrweiler , P. Barbrook-Johnson , K. P. Narasimhan , H. Wilkinson , Computational Modelling of Public Policy: Reflections on Practice, Journal of Artificial Societies and Social Simulation 21 ( 2018 ).

[3] B. Edmonds , L. ní Aodha , Using Agent-based Modelling to Inform Policy-What Could Possibly Go Wrong? , in: Multi-Agent-Based Simulation XIX: 19th International Workshop , MABS 2018, Stockholm, Sweden, Springer, 2019 , pp. 1 - 16 .

[4] R. Seidl , A Functional-Dynamic Reflection on Participatory Processes in Modeling Projects , Ambio 44 ( 2015 ) 750 - 765 .

[5] P. Ahrweiler , D. Frank , N. Gilbert, Co-designing Social Simulation Models for Policy Advise: Lessons Learned from the INFSO-SKIN study , in: 2019 Spring Simulation Conference, IEEE, 2019 , pp. 1 - 12 .

[6] A. Melchior , F. Dignum , M. Ruiz , A Closer Look at Dutch Policy Development, in: P. Ahrweiler, M. Neumann (Eds.), Advances in Social Simulation, Complexity , Springer, 2019 , pp. 383 - 395 .

[7] M. Calder , C. Craig , D. Culley , R. De Cani, C. A. Donnelly , R. Douglas , B. Edmonds , J. Gascoigne , N. Gilbert , C. Hargrove , et al., Computational Modelling for Decision-Making: Where, Why , What, Who and How, Royal Society Open Science 5 ( 2018 ) 172096 .

[8] B. Edmonds , Review of The Aqua Book: Guidance on Producing Quality Analysis for Government , Journal of Artificial Societies and Social Simulation 19 ( 2016 ).

[9] P. Barbrook-Johnson , C. Schimpf , B. Castellani, Reflections on the Use of Complexityappropriate Computational Modeling for Public Policy Evaluation in the UK , Journal on Policy and Complex Systems 5 ( 2019 ) 55 - 70 .

[10] L. Freebairn , J.-A. Atkinson , P. M. Kelly , G. McDonnell , L. Rychetnik , Decision Makers' Experience of Participatory Dynamic Simulation Modelling: Methods for Public Health Policy, BMC Medical Informatics and Decision Making 18 ( 2018 ) 1 - 14 .

[11] P. Cairney , The Politics of Evidence-based Policy Making , Springer, 2016 .

[12] G. Polhill , D. Salt , The Importance of Ontological Structure: Why Validation by 'Fit-todata' is Insuficient , in: B. Edmonds , R. Meyers (Eds.), Simulating Social Complexity: A Handbook , Springer, 2017 , pp. 141 - 172 .

[13] B. Edmonds , S. Moss , From KISS to KIDS-an 'Anti-simplistic'Modelling Approach, in: Multi-Agent and Multi- Agent-Based Simulation: Joint Workshop MABS 2004 , New York, USA, Springer, 2005 , pp. 130 - 144 .

[14] B. Edmonds , C. Le Page , M. Bithell , E. Chattoe-Brown , V. Grimm , R. Meyer, C. MontañolaSales, P. Ormerod, H. Root , F. Squazzoni , Diferent Modelling Purposes, Journal of Artificial Societies and Social Simulation 22 ( 2019 ).

[15] P. d'Aquino , C. Le Page , F. Bousquet , A. Bah , A Novel Mediating Participatory Modelling: The Self-Design Process to Accompany Collective Decision making , International Journal of Agricultural Resources, Governance and Ecology 2 ( 2002 ) 59 - 74 .

[16] D. B. Van Berkel , P. H. Verburg , Combining Exploratory Scenarios and Participatory Backcasting: Using an Agent-based Model in Participatory Policy Design for a Multifunctional Landscape , Landscape Ecology 27 ( 2012 ) 641 - 658 .

[17] J. D. Tabara , E. Roca , C. Madrid, P. Valkering, P. Wallman , P. Weaver , Integrated Sustainability Assessment of Water Systems: Lessons from the Ebro River Basin , International Journal of Innovation and Sustainable Development 3 ( 2008 ) 48 - 69 .

[18] R. Rojas , J. Castilla-Rho , G. Bennison , R. Bridgart , C. Prats , E. Claro, Participatory and Integrated Modelling under Contentious Water Use in Semiarid Basins, Hydrology 9 ( 2022 ) 49 .

[19] L. Whitmarsh , B. Nykvist , Integrated Sustainability Assessment of Mobility Transitions: Simulating Stakeholders' Visions of and Pathways to Sustainable Land-based Mobility , International Journal of Innovation and Sustainable Development 3 ( 2008 ) 115 - 127 .