Modeling real-world phenomena is incredibly challenging due to the intricate interactions among numerous interconnected elements. Understanding these systems is nearly impossible when they are viewed in isolation. Consequently, such systems are often referred to as complex systems, though a precise definition of complexity remains elusive [1]. These systems typically share features such as non-linearity, decentralized control, and feedback mechanisms. In recent years, Computational Science has leveraged data-intensive computing and analysis to study such issues. ABMs offer a bottom-up approach for analyzing complex systems, allowing modelers to design the behaviors of individual agents (e.g., members of a population) and the environments in which they operate. The interactions among these agents in the simulated environment produce emergent properties and phenomena that the modeler aims to examine and understand. These three components (behavior, environment, and interactions) are the building blocks of an ABM and have been proven to be very effective in modeling different scenarios across a vast corpora of fields [2].

ABMs are also a recurring theme in High-Performance Computing (HPC), since these models are designed to mimic social interactions, the global economy, or natural phenomena. In addition, they can help predict potential outcomes involving numerous entities. However, when the number of agents consistently grows, traditional ABM engines fail because the computational power required by a single execution becomes an unbearable limitation. The literature states that ABMs can intersect with HPC through two distinct ways: the outer and inner loops [3]. The former usually describes optimization techniques such as model parameters sweeping [4]. The latter, which is also the focus of this paper, typically involves distributing a model across different computational nodes using de facto standards such as MPI (Message Protocol Interface) [5] or OpenMP (Open Multiprocessing) [6].

This paper presents the early stages of implementing a distributed version of krABMaga, an ABM engine written entirely in Rust. We have employed our experiences in ABMs and the distributed computing field to enhance the capabilities of the krABMaga engine, assessing the possibilities and opportunities for deploying a high-optimized tool to an HPC cluster with minimal effort and achieving good results [4].

ABMs have been extensively studied and applied across various domains, providing valuable insights into complex systems through simulations [7,8]. However, while there is a substantial body of work focused on ABMs in general, research into their distributed versions is relatively limited, with tools outdated or not supported anymore.

Generally, distributed ABMs involve partitioning the simulation across multiple computational nodes to handle larger-scale models or to achieve higher performance. This approach can significantly improve the scalability and efficiency of simulations by leveraging parallel processing and distributed computing resources. Despite its potential, the distributed version of ABMs presents additional challenges, such as managing communication between nodes, ensuring data consistency, and effectively balancing the computational load. Several frameworks have explored a distributed approach, such as:

. Mason is a Java-based ABM simulation toolkit. A distributed version of this library, known as D-Mason [10], was developed to enhance performance. It uses a Quad-Tree data structure to manage the simulation field.

Flame [11]. Flame is an ABM system designed to run on a wide range of heterogeneous computing platforms. It offers a formal framework for model creation using the XXML language, a variant of XML, along with automatic parallel code generation. [12]. Flame-GPU extends Flame to support GPU translation. It simplifies GPU programming by using an API that leverages the FLAME template to generate CUDA code for target GPU devices, eliminating the need for users to engage directly with GPU programming languages or optimization techniques.

Pandora [13]. Pandora is an ABM framework for large-scale distributed simulations. It provides two identical programming interfaces in different programming languages, one of them including the automatic generation of parallel and distributed code. [14]. Repast-HPC is a component of the Repast suite, specifically designed for largescale simulations on C++-based systems. It is tailored for execution on large computing clusters and supercomputers.

This work revolves around a distributed version of krABMaga 1 , an open-source discrete events simulation engine written in Rust for developing ABM simulations [4,15]. The distributed engine, as described in the next sections, simplifies the simulation development process by abstracting the complexities involved in distributed computing. Therefore, this implementation allows modelers to focus on the core logic of their simulations without getting bogged down by technical issues and communication layers. KrABMaga aims to be an intuitive toolbox inspired by the popular MASON library, particularly its modular design that separates the simulation and visualization subsystems. The Rust programming language underpins this approach [16], thanks to its main principles:

• Performance: Rust offers both speed and memory efficiency. Its memory model eliminates the need for a garbage collector at runtime, making it well-suited for critical services, embedded devices, and seamless integration with other languages.

• Reliability: It exploits its ownership and borrowing system to guarantee memory and thread safety, removing many classes of bugs at compile time. • Productivity: The language is shipped with great documentation, a friendly compiler with useful error messages, and a fast-growing community that has written a large corpus of high-optimized crates (libraries in the Rust ecosystem).

Rust stands out by performance similar to C, which can shorten the duration of a single simulation and by a unique programming approach that ensures no memory-related bugs occur throughout long-running experiments, hence enabling simulation reliability [17]. Every ABM engine needs each simulation to have at least two important components: a state and an agent. The state describes the environment of the simulation, which contains different elements, such as a field and its dimensions, the number of steps, the initial number of agents and others, while the agent contains the behavior of the population inside the simulation. Thanks to this decoupled structure, the simulation field can be modified without impacting other parts of the simulation, such as the agent behavior or the simulation parameters. Finally, the framework provides additional functionalities, such as monitoring, reproducibility, and visualization tools.

A key aspect of ABM computation is the communication between agents. Typically, each simulation agent needs to identify its neighbors to exchange information and perform its tasks. In sequential execution, this process is routine and has a moderate impact on performance. However, in distributed execution, where neighbors may be located on different machines, discovering and interacting with them can become a significant performance bottleneck. Moreover, when an agent moves between partitions or is removed from the simulation, the process must proceed seamlessly as if all agents were in a single field. Addressing these challenges is crucial when developing an efficient distributed system that can manage multiple partitions of the same field across remote machines.

To facilitate the distribution of the simulation, we began by modifying an existing field in our framework. Our first attempt takes as a reference the Field2D implementation on the krABMaga repository 2 , which is the standard field where an agent can move on a continuous 2D space.

Sequential structure. The Field2D uses a two-dimensional toroidal grid as a simulation field characterized by an origin point (x;y), a width, and a height. Each grid coordinate identifies a cell in which an agent can reside and interact with other agents. Distributed structure. K-Dimensional Tree (K-D Tree) data structures are widely used in distributed computing, particularly for tasks like clustering and closest neighbors search when a scalable solution is required [18]. A K-D Tree is a tree structure in which each node has exactly two children and can be split until the desired number of nodes is reached. For each pair of children created, the parent node keeps references to them, allowing us to reach any leaf, starting from the root, in a short time [19]. We implemented a customized version of a K-D Tree, where each child node stores references to all other nodes created in the tree, referred to as blocks. Each block represents a segment of the simulation field and includes an ID corresponding to the process rank it will be assigned to, its origin point (x, y), as well as its width and height, as shown in Figure 1. By maintaining references to every child node, each node gains access to comprehensive information about all nodes, facilitating efficient neighbor search and synchronization operations. Although this modification increases the physical space required, the resulting efficiency in communication justifies this trade-off. Moreover, this approach remains practical since the growth rate of machines -and thus partitions -does not scale as rapidly as the number of agents.

After the main process has computed each block, it sends a reference to each block to all other processes. These processes then receive and store the reference in local memory, allowing them to communicate using the associated ID when necessary. When an agent changes position during the simulation and exits from a block's border, it must be sent to another block. Since every block knows the exact size and ID of all the others, when an agent moves outside its border, it can easily calculate the ID of the processor that will be responsible for the agent based on its position. To make this exchange possible, the object is saved into an array whenever an agent moves outside the border of its process field. At the end of every step, every process sends all the agents who have moved off their partition to their respective neighbors and receives all agents sent by their neighbors. This phase is preceded by a message exchange phase, where each process sends the number of agents it will exchange to each neighbor and receives the value from all its neighbors. When this process is complete, each process allocates the buffer with adequate slots for the incoming agents. This communication phase is handled using MPI collective operations, such as scatter and gather, combined with non-blocking send and receive operations to facilitate data exchange across processors efficiently.

Additionally, in many simulations, an agent needs to be aware of nearby agents within a specific area of interest (AOI), usually defined by a fixed-size radius (see Figure 2). This can be particularly challenging in distributed simulations because an agent's AOI may be divided across different processes. To accomplish this task, it is essential to identify first agents that could be neighbors of agents from other processes. This process is facilitated by Halo Regions, of which an example is illustrated in Figure 2. A Halo Region is a fixed-length zone located near the borders. When an agent moves inside the field and enters a Halo region, a copy of the agent is placed inside an array of agents, indexed by the Halo Region that keeps it. At the end of every step, these agents are sent to the potential neighbors and received from each one. This operation uses the same principles of the exchange between processors. Once all agents have been received, the process can calculate the neighbors of each agent, considering both the local agents and the received agents that are in the AOI.

To evaluate the proposed implementation, we have chosen Flockers as the main benchmark model [20]. Tests are based on known parameters for the model [15]. The number of steps for each simulation is fixed to 200. The model is evaluated using the configurations for the number of agents and field dimensions listed in Table 1.

The code to reproduce the model is available on the GitHub repository 3 . We have built a cluster on Microsoft Azure 4 to make the benchmark as fair as possible, eliminating the noise from background Figure 2: An example of halo region. In this example, the central square represents the main actor. The red-highlighted areas are the Halo regions of its neighboring agents, while the green areas indicate the portions of the field shared with those neighbors. Agents located within these green regions are locally stored by the processor managing them. At the end of each step, processors sharing borders exchange information about the agents in the red halo regions and, if needed, transfer agents between processors.

Size of the examples.

Field size Density

activities on local machines. Each virtual machine on the Azure cluster was created within a Proximity Placement Group 5 and has the following specifications:

• Size: standard_DS1_v2 • Number vCPU : 1 • CPU family: Intel Xeon Platinum 8370C (or similar 6 ).

• Memory: 3.5 GiB • S.O.: Ubuntu Server 22.04 LTS • Disk: 8 GB Standard SSD All numerical results obtained with an average execution time of 10 runs for each setup are displayed in Table 2. The curve trend of this model is displayed when varying the number of processors in Figure 3. It is evident that when computation is the main task of the distributed system, every configuration performs efficiently, closely approaching the optimal curve. However, it is also apparent that the curves tend to slow down at specific thresholds and can sometimes deteriorate. The primary culprit is the communication phase, which becomes a bottleneck when many halo regions are filled with agents and need to exchange information at each step. The simulation with 1M agents running on 8 processing nodes reveals a speed-up exceeding 8, which is an unexpected result that warrants further investigation. This anomaly could be due to near-perfect system load balancing or optimal memory alignment with the machine's cache. However, these explanations seem improbable, considering the benchmark was executed multiple times with randomized seed values. These results highlight the need for load balancing to optimize the size of each partition, thereby reducing the number of agents that need to be exchanged each step.

The results demonstrate that the current implementation achieves significant speed-up with an increasing number of processors, outperforming the sequential algorithm even with just 2 processes. Additionally, this implementation allows each simulation to utilize the modified K-D Tree with minimal adjustments required to the sequential model. These changes are related to the state and the launching parameters of the simulation and involve the necessity, until a new version of the library is released, to define the Equivalence trait on the agent to make the communication with MPI possible. This work comes with two main limitations. First, the field structure does not implement any loadbalancing system. The lack of this feature explains the degradation of the model's performance as the number of agents increases. An example of how the load balance should work is shown in Figure 4. Second, the tests have been evaluated only on the krABMaga engine, not against different engines. The next step should include a more extensive study comparing it with other distributed ABM engines.

Future work should concentrate on balancing the distribution of agents across partitions, ensuring each processor shares a fair amount of work. To provide a more comprehensive assessment, we will evaluate and compare the engine's performance against that of current state-of-the-art open-source engines with available distributed versions. Finally, after a more polished version of the framework is released, we should focus on abstracting the distributed field as much as possible from the modeler. This will make it easier to write a new model from scratch without dealing with the complex layer of the distributed computing paradigm.