An agent-based model (ABM) is a computational model for simulating autonomous agents' actions and interactions to understand a system's behavior and what governs its outcomes. When the data or number of agents grow or multiple runs are necessary, agent-based simulations are generally computationally costly. Therefore, adopting diferent computing paradigms, such as the distributed one, is essential to manage long-running simulations. The main problem with this approach is finding a way to distribute and balance the simulation field so that the agents can move from one machine to another with the least amount of synchronization overhead. Based on our experiences, we present a Rust-based ABM engine capable of distributing models on high-performance computing resources, gaining remarkable speedup against the sequential version.
Modeling real-world phenomena is incredibly challenging due to the intricate interactions among
numerous interconnected elements. Understanding these systems is nearly impossible when they are
viewed in isolation. Consequently, such systems are often referred to as complex systems, though a
precise definition of complexity remains elusive [
ABMs are also a recurring theme in High-Performance Computing (HPC), since these models are
designed to mimic social interactions, the global economy, or natural phenomena. In addition, they can
help predict potential outcomes involving numerous entities. However, when the number of agents
consistently grows, traditional ABM engines fail because the computational power required by a single
execution becomes an unbearable limitation. The literature states that ABMs can intersect with HPC
through two distinct ways: the outer and inner loops [
This paper presents the early stages of implementing a distributed version of krABMaga, an ABM
engine written entirely in Rust. We have employed our experiences in ABMs and the distributed
computing field to enhance the capabilities of the krABMaga engine, assessing the possibilities and
opportunities for deploying a high-optimized tool to an HPC cluster with minimal efort and achieving
good results [
ABMs have been extensively studied and applied across various domains, providing valuable insights
into complex systems through simulations [
Generally, distributed ABMs involve partitioning the simulation across multiple computational nodes
to handle larger-scale models or to achieve higher performance. This approach can significantly improve
the scalability and eficiency of simulations by leveraging parallel processing and distributed computing
resources. Despite its potential, the distributed version of ABMs presents additional challenges, such
as managing communication between nodes, ensuring data consistency, and efectively balancing the
computational load. Several frameworks have explored a distributed approach, such as:
Mason [
Flame [
Flame-GPU [
Pandora [13]. Pandora is an ABM framework for large-scale distributed simulations. It provides two identical programming interfaces in diferent programming languages, one of them including the automatic generation of parallel and distributed code.
Repast-HPC [14]. Repast-HPC is a component of the Repast suite, specifically designed for largescale simulations on C++-based systems. It is tailored for execution on large computing clusters and supercomputers.
This work revolves around a distributed version of krABMaga1, an open-source discrete events
simulation engine written in Rust for developing ABM simulations [
Rust stands out by performance similar to C, which can shorten the duration of a single simulation and by a unique programming approach that ensures no memory-related bugs occur throughout long-running experiments, hence enabling simulation reliability [17]. Every ABM engine needs each simulation to have at least two important components: a state and an agent. The state describes the environment of the simulation, which contains diferent elements, such as a field and its dimensions, the number of steps, the initial number of agents and others, while the agent contains the behavior of the population inside the simulation. Thanks to this decoupled structure, the simulation field can be modified without impacting other parts of the simulation, such as the agent behavior or the simulation parameters. Finally, the framework provides additional functionalities, such as monitoring, reproducibility, and visualization tools.
A key aspect of ABM computation is the communication between agents. Typically, each simulation agent needs to identify its neighbors to exchange information and perform its tasks. In sequential execution, this process is routine and has a moderate impact on performance. However, in distributed execution, where neighbors may be located on diferent machines, discovering and interacting with them can become a significant performance bottleneck. Moreover, when an agent moves between partitions or is removed from the simulation, the process must proceed seamlessly as if all agents were in a single field. Addressing these challenges is crucial when developing an eficient distributed system that can manage multiple partitions of the same field across remote machines.
To facilitate the distribution of the simulation, we began by modifying an existing field in our framework. Our first attempt takes as a reference the Field2D implementation on the krABMaga repository2, which is the standard field where an agent can move on a continuous 2D space. Sequential structure. The Field2D uses a two-dimensional toroidal grid as a simulation field characterized by an origin point (x;y), a width, and a height. Each grid coordinate identifies a cell in which an agent can reside and interact with other agents.
Distributed structure. K-Dimensional Tree (K-D Tree) data structures are widely used in distributed computing, particularly for tasks like clustering and closest neighbors search when a scalable solution is required [18]. A K-D Tree is a tree structure in which each node has exactly two children and can be split until the desired number of nodes is reached. For each pair of children created, the parent node keeps references to them, allowing us to reach any leaf, starting from the root, in a short time [19]. We implemented a customized version of a K-D Tree, where each child node stores references to all other nodes created in the tree, referred to as blocks. Each block represents a segment of the simulation field and includes an ID corresponding to the process rank it will be assigned to, its origin point (x, y), as well as its width and height, as shown in Figure 1. By maintaining references to every child node, each node gains access to comprehensive information about all nodes, facilitating eficient neighbor search and synchronization operations. Although this modification increases the physical space required, the resulting eficiency in communication justifies this trade-of. Moreover, this approach remains practical since the growth rate of machines — and thus partitions — does not scale as rapidly as the number of agents.
After the main process has computed each block, it sends a reference to each block to all other processes. These processes then receive and store the reference in local memory, allowing them to communicate using the associated ID when necessary. 2https://github.com/krABMaga/krABMaga/blob/main/src/engine/fields/field_2d.rs
When an agent changes position during the simulation and exits from a block’s border, it must be sent to another block. Since every block knows the exact size and ID of all the others, when an agent moves outside its border, it can easily calculate the ID of the processor that will be responsible for the agent based on its position. To make this exchange possible, the object is saved into an array whenever an agent moves outside the border of its process field. At the end of every step, every process sends all the agents who have moved of their partition to their respective neighbors and receives all agents sent by their neighbors. This phase is preceded by a message exchange phase, where each process sends the number of agents it will exchange to each neighbor and receives the value from all its neighbors. When this process is complete, each process allocates the bufer with adequate slots for the incoming agents. This communication phase is handled using MPI collective operations, such as scatter and gather, combined with non-blocking send and receive operations to facilitate data exchange across processors eficiently.
Additionally, in many simulations, an agent needs to be aware of nearby agents within a specific area of interest (AOI), usually defined by a fixed-size radius (see Figure 2). This can be particularly challenging in distributed simulations because an agent’s AOI may be divided across diferent processes. To accomplish this task, it is essential to identify first agents that could be neighbors of agents from other processes. This process is facilitated by Halo Regions, of which an example is illustrated in Figure 2. A Halo Region is a fixed-length zone located near the borders. When an agent moves inside the field and enters a Halo region, a copy of the agent is placed inside an array of agents, indexed by the Halo Region that keeps it. At the end of every step, these agents are sent to the potential neighbors and received from each one. This operation uses the same principles of the exchange between processors. Once all agents have been received, the process can calculate the neighbors of each agent, considering both the local agents and the received agents that are in the AOI.
To evaluate the proposed implementation, we have chosen Flockers as the main benchmark model [20]. Tests are based on known parameters for the model [15]. The number of steps for each simulation is fixed to 200. The model is evaluated using the configurations for the number of agents and field dimensions listed in Table 1.
The code to reproduce the model is available on the GitHub repository3. We have built a cluster on Microsoft Azure4 to make the benchmark as fair as possible, eliminating the noise from background 3https://github.com/krABMaga/examples/tree/main/flockers_mpi 4https://azure.microsoft.com activities on local machines. Each virtual machine on the Azure cluster was created within a Proximity Placement Group5 and has the following specifications: • Size: standard_DS1_v2 • Number vCPU : 1 • CPU family: Intel Xeon Platinum 8370C (or similar6). • Memory: 3.5 GiB • S.O.: Ubuntu Server 22.04 LTS • Disk: 8 GB Standard SSD
All numerical results obtained with an average execution time of 10 runs for each setup are displayed in Table 2. The curve trend of this model is displayed when varying the number of processors in Figure 3. It is evident that when computation is the main task of the distributed system, every configuration performs eficiently, closely approaching the optimal curve. However, it is also apparent that the curves tend to slow down at specific thresholds and can sometimes deteriorate. The primary culprit is the communication phase, which becomes a bottleneck when many halo regions are filled with agents and need to exchange information at each step. The simulation with 1M agents running on 8 processing nodes reveals a speed-up exceeding 8, which is an unexpected result that warrants further investigation. This anomaly could be due to near-perfect system load balancing or optimal memory alignment with the machine’s cache. However, these explanations seem improbable, considering the benchmark was executed multiple times with randomized seed values. These results highlight the need for load balancing to optimize the size of each partition, thereby reducing the number of agents that need to be exchanged each step.
The results demonstrate that the current implementation achieves significant speed-up with an increasing number of processors, outperforming the sequential algorithm even with just 2 processes. Additionally, this implementation allows each simulation to utilize the modified K-D Tree with minimal adjustments required to the sequential model. These changes are related to the state and the launching parameters of the simulation and involve the necessity, until a new version of the library is released, to define the Equivalence trait on the agent to make the communication with MPI possible.
This work comes with two main limitations. First, the field structure does not implement any loadbalancing system. The lack of this feature explains the degradation of the model’s performance as the number of agents increases. An example of how the load balance should work is shown in Figure 4. Second, the tests have been evaluated only on the krABMaga engine, not against diferent engines. The next step should include a more extensive study comparing it with other distributed ABM engines.
Future work should concentrate on balancing the distribution of agents across partitions, ensuring each processor shares a fair amount of work. To provide a more comprehensive assessment, we will evaluate and compare the engine’s performance against that of current state-of-the-art open-source engines with available distributed versions. Finally, after a more polished version of the framework is released, we should focus on abstracting the distributed field as much as possible from the modeler. This will make it easier to write a new model from scratch without dealing with the complex layer of the distributed computing paradigm.
This work has been partially supported by the spoke “FutureHPC & BigData" of the ICSC – Centro Nazionale di Ricerca in High-Performance Computing, Big Data and Quantum Computing funded by European Union – NextGenerationEU. [13] X. Rubio-Campillo, Pandora: a versatile agent-based modelling platform for social simulation,
Proceedings of SIMUL (2014) 29–34. [14] N. Collier, M. North, Repast HPC: A Platform for Large-Scale Agent-Based Modeling, Large-Scale
Computing (2011) 81–109. [15] A. Antelmi, G. Cordasco, G. D’Ambrosio, D. De Vinco, C. Spagnuolo, Experimenting with Agent
Based Model Simulation Tools, Applied Sciences 13 (2023). [16] W. Bugden, A. Alahmar, Rust: The programming language for safety and performance, arXiv preprint arXiv:2206.05503 (2022). [17] R. Jung, J.-H. Jourdan, R. Krebbers, D. Dreyer, RustBelt: Securing the foundations of the Rust programming language, Proceedings of the ACM on Programming Languages 2 (2017) 1–34. [18] A. Chakravorty, W. S. Cleveland, P. J. Wolfe, Scalable -d trees for distributed data, arXiv preprint arXiv:2201.08288 (2022). [19] R. A. Brown, Building a Balanced -d Tree in ( log ) Time, Journal of Computer Graphics
Techniques (JCGT) 4 (2015) 50–68. [20] C. W. Reynolds, Flocks, herds and schools: A distributed behavioral model, in: Proceedings of the 14th annual conference on Computer graphics and interactive techniques, 1987, pp. 25–34.