A distributed simulation of P/T nets requires partitioning the overall model into modules that run on multiple simulators. Failures in distributed systems can compromise consistency, resulting in incorrect outcomes. Ensuring resilience in such simulations is essential for maintaining correctness, especially in long-running executions. This research develops a concept for resilient simulators in distributed P/T net simulations. A prototyping approach grounded in constructivist principles enables the detection and recovery of failures in P/T net simulations. The evaluation follows a summative ex-post methodology, applying a criteria-based assessment to validate efectiveness. Experimental validation demonstrates the feasibility of the proposed concept of resilience mechanisms for distributed P/T net simulations. The system maintains simulation consistency despite failures by integrating state-saving and recovery techniques. The results confirm improved fault tolerance and reliability in distributed P/T net simulations. The introduced concept for resilient P/T net simulations enhances the robustness of distributed simulations. Preventing inconsistencies ensures accurate analysis and reliable execution over extended periods. The findings contribute to the development of fault-tolerant simulation frameworks, supporting more reliable distributed computing environments.
The distribution of complex simulation models across several independent computing nodes is becoming
increasingly important in software engineering, particularly system simulation. [
The present work lies at the intersection of distributed systems and the simulation of Petri nets, particularly in the context of resilience, i.e., the ability to recover system states after faults occur. Particularly in application areas such as process automation, workflow management, or complex, longrunning simulations of critical systems, simulations must continue to run resiliently and consistently despite errors. Despite considerable progress in the distribution and scaling of P/T net simulations, there are still significant research gaps regarding suitable strategies for fault detection, state safety, and systematic recovery from faults. The development of resilient simulators is not only a technical challenge but also opens up new methodological perspectives for the reliability of distributed simulations. The state safety and recovery mechanisms developed and validated in this work are expected to enhance fault tolerance and provide a comprehensive understanding of the causes and efects of inconsistencies in distributed Petri net simulations.
Therefore, the central research question of this work is: How can distributed simulators of P/T nets be designed to guarantee consistent simulation results even in the event of system failures? This research question is based on the hypothesis that implementing structured statefulness procedures and liveness-based monitoring strategies can significantly increase the resilience of distributed simulations of P/T nets, allowing simulators to continue operating consistently despite failures.
This contribution adopts a constructivist approach to addressing the research question, developing
a prototype simulator grounded in resilient design principles. [
Within the Foundations (Section 2), the topics of Renew (Section 2.1), Distributed P/T Nets (Section 2.2), Failures (Section 2.3), Resilience (Section 2.4), and Kubernetes (Section 2.5) are systematically introduced. Subsequently, the Problem Description (Section 3) and the design of the Distributed System (Section 4) are presented. The ensuing section details the prototypes developed during this work, specifically Detecting Failures of Simulators (Section 5) and Recovering Failures of Simulators – DPTNResiliency (Section 6). The overall system is then evaluated using a classical case study in computer science, the Producer-Storage-Consumer scenario (Section 7). A critical discussion (Section 8) of the advantages, disadvantages, and limitations of the proposed concept follows. Finally, the article concludes with an overview of Related Work (Section 9) and the Conclusion (Section 10).
This section introduces the relevant concepts and technologies that are the focus of this paper. Firstly, we
present Renew as a Java-based multi-formalism editor and simulator for especially reference nets (2.1)
and further introduce our set-up for distributed P/T nets (2.2). Then, we proceed to diferent possible
failure types, including classification, detection, and recovery (2.3). We finish with a definition of
resilience (2.4) and an overview of the relevant terms and concepts within the Kubernetes environment
(2.5).
2.1. Renew
Renew [
Renew is implemented in Java 17 [
For each supported Petri net variant, Renew provides a dedicated formalism plugin, the most
prominent being the reference net formalism according to Kummer [
The distributed P/T nets used in this contribution are the same as in [
The informal extension of distributed synchronous channels ensures that P/T nets can communicate with each other by implementing rendezvous synchronization. Transitions are labeled with signatures
so that they can synchronize if they have the corresponding signature. These labeled transitions with matching signatures can only fire together.
The signature of a synchronous channel, which is used here, consists of type, identifier, and parameter.
There are two diferent types: the downlink and uplink, which can be denoted as and ,
respectively. [
Each of the distributed P/T nets can be considered as an individual module, whereby the consideration
of a module based on [
The overall system architecture for the distributed P/T nets [
The communication between P/T nets across simulator boundaries is facilitated through distributed
synchronous channels. The event-based communication medium Apache Kafka is employed for this
purpose. Apache Kafka is an open-source, distributed event-streaming platform designed to deliver
scalability and high performance [
An illustrative example is provided by Clasen et al. [
(a) Producer and Consumer Net
In every computer system, it is unavoidable that, at some point, some failure can occur. [
Internal or in-process errors occur within an application and can be classified into diferent types.
Logical errors [
Closely related are semantic errors, which are often used synonymously. Semantic errors [
In addition to the mentioned types of errors that result in faulty program behavior, runtime errors may
also occur despite a "correct" program implementation. For example, excessive nested calls of a recursive
function typically lead to a stack overflow, where the call stack of a program exceeds the allocated
memory space. If a program continuously consumes memory without releasing it correctly, this leads
to a memory leak. Both stack overflows and memory leaks fall into the category of memory-related
errors. [
Problems in the parallel execution of (sub-)processes can cause synchronization errors, such as
deadlocks or race conditions. A deadlock occurs when a cyclic waiting situation arises between the
involved processes, with each waiting for the release of a system resource that is exclusively held by
another. A race condition, on the other hand, describes an unintended fault where multiple operations
influence the final result due to their timing behavior. [
We can group logical and semantic errors, as well as synchronization and memory errors, into a
broader category of software errors to fit into the categorization of Schroeder et al. [
When disruptions occur during data transmission or exchange between distributed components, they are referred to as network or communication errors. These can result from issues such as packet loss or connection failures, leading to inconsistencies in the state of distributed systems. Such inconsistencies can adversely afect the synchronization and correctness of the simulation.
System and hardware errors originate from physical defects or failures in the components of the distributed system, such as the CPU or memory. These errors are often dificult to predict and can cause abrupt system crashes or faulty data processing.
In addition to internal errors, there are also external error types that depend on external factors. Undesired user inputs or environmental influences can impair the correctness of the simulation or even disrupt the intended functionality of the entire system.
In the context of our contribution, a relevant structure of error types emerges, as shown in Figure 2. We distinguish between internal and external errors that may afect our distributed simulation. Among internal errors, we further diferentiate between software-based errors (such as logical, semantic, synchronization-specific, and memory-related errors) and infrastructure-based errors, which depend on the system architecture. The latter includes communication and hardware errors.
To ensure the robustness and reliability of a distributed simulation, the early detection of occurring failures is essential. These errors can occur in both deterministic and non-deterministic ways, which complicates their identification and reproducibility. Typical deterministic errors include logical flaws or faulty algorithms, whereas synchronization errors, such as deadlocks and race conditions, are considered non-deterministic.
The first method for detecting failures is through static analysis. Although they primarily serve
compile-time checking, formal analysis techniques can detect potential runtime errors before execution.
Static analysis tools such as SonarGraph [
On the other hand, there are dynamic methods to detect failures at runtime. This involves monitoring
the system during execution. Techniques such as assertion checking, instrumentation, or the use
of debugging tools like Valgrind [
Monitoring technologies such as Prometheus [
Since we cannot entirely prevent failures, we must at least design our systems to tolerate or recover
from them when they occur. It is worth noting that, in some situations, it can be better to tolerate small
failures by not addressing them if an automated recovery system might do more harm than good [
In general, we can categorize fault tolerance methods into two main groups: reactive and proactive
methods. Static analysis methods act as a proactive method. Proactive methods take action preemptively
to try and limit failures as much as possible, while reactive methods rely on failure detection and start
acting when a failure has occurred. [
Since proactive methods cannot entirely prevent failures, we do not elaborate on them further in this
paper. More information can be found, e.g., in [
An obvious reactive method is checkpointing, sometimes also referred to as checkpoint-restart. [
Another reactive method, which can also be considered a hybrid method, is replication. Here,
individual components can be replicated, potentially even on diferent physical machines, resulting in
multiple instances of each component. If one instance fails, a replica can take its place to ensure smooth
execution, and the original instance gets restarted to serve as another replica. [
According to Laprie et al. [
For this paper, we use the definition from Pradhan et al. [
Kubernetes [
On each node, a service called kubelet acts as the "primary node agent" for Kubernetes, manages everything that Kubernetes runs on that node. If some application crashes on a specific node for example, the set of kubelets of all nodes would be responsible for restarting it.
Additionally, various resource types allow applications to scale automatically according to demand, making it suitable for all kinds of applications and workloads.
Containerisation can be defined as the act of bundling a software application and all necessary
dependencies and system libraries into a single container. [
A key property of containers is that they are stateless by default. That means every time a container is started, it has no recollection of past instances of itself or other state.
A Pod is the smallest deployable entity in Kubernetes and consists of one or more containers [
Container lifecycle hooks are part of the OCI runtime specification [
Kubernetes ofers a few hooks (more commonly called probes) related to Pod lifecycles, most
importantly liveness probes. As the name suggests, liveness probes allow Kubernetes to check that our Pods
are still active and have not failed or crashed [
In Kubernetes, one usually does not directly create Pods themselves. Instead, there are a number of resources types that create and manage a set of Pods, each having unique use cases, advantages and disadvantages.
A StatefulSet owns a set of Pods and maintains a unique identity for each one, as well as an ordering over all its Pods. The Pod identity it provides includes a network address, (if configured) a dedicated storage mount, a name and and index label. If any Pod that belongs to a StatefulSet fails, for example by crashing or because the associated Liveness Probe fails, a new Pod will be created with the same identity of the failed Pod.
Some applications do require Pods to have some form of persistent storage, or memory, even between restarts. For this purpose, Kubernetes implements the concept of Persistent Volumes (PVs) and Persistent Volume Claims (PVCs).
A PV is a resource in a cluster that provides a piece of storage. Their lifecycle is independent from Pods. If a Pod needs some persistent storage, it can request some via a PVC. In this case, the Pod can only start when a fitting PV binds to its PVC. The volume will then mount in the corresponding container(s) filesystem.
PVs can either be created manually by cluster administrators, or by a StorageClass that is configured to automatically provision PVs when a PVC requests storage from it.
Ceph is a mature distributed file system that is developed for performance, scalability and reliability
[
The Cloud Native Computing Foundation [
All simulators collaboratively execute a single simulation. To this end, each simulator is capable of concurrently executing multiple distributed P/T nets. The global state of the simulation comprises the complete set of distributed P/T nets along with their respective markings. The local state of a simulator is defined as the subset of distributed P/T nets it executes, including the corresponding markings. Thus, each simulator is responsible for a partition of the overall system and manages the associated segment of the simulation state.
In the context of complex systems or processes, simulations often run for extended periods of time. Prolonged runtimes increase the probability of individual component failures. In the worst-case scenario, such failures can result in inconsistencies that necessitate a complete restart of the simulation. Since failures may stem from a variety of causes—most notably hardware faults—they cannot be entirely precluded.
It is, therefore, imperative to develop robust mechanisms for detecting and mitigating failures in distributed simulations. Failures are detected through continuous monitoring of the simulators’ operational status. Once a failure is identified, the afected simulator is reinitiated on a functional computing node.
To ensure correct recovery, appropriate techniques must be employed to reconstruct the simulator’s state, as each simulator retains a distinct portion of the global state. This is achieved by periodically persisting the simulator’s state under predefined conditions to highly available and durable storage. In the event of a failure, the simulator can resume execution from the most recent consistent state, ensuring continuity of the simulation.
The concept introduced in this work is validated in the context of distributed simulation of P/T nets. Given the distributed nature of the simulation, the overall system is inherently decentralized. The ifrst objective of this work is the conceptual design and technical realization of the distributed system (Section 4).
A subsequent objective involves developing mechanisms for fault detection in simulators (Section 5). Since simulators may be inaccessible in the event of failure, direct fault detection within the simulators is infeasible. Consequently, fault detection must be implemented as part of the surrounding distributed system infrastructure.
Recovery techniques must be tailored to the specific architectural design of the simulators, which are themselves responsible for ensuring their resilience (Section 6). Simulators must be capable of persistently storing their state at semantically meaningful intervals and resuming execution from that state following a failure.
Finally, the proposed concept is evaluated within a representative scenario (Section 7), followed by a critical discussion of its advantages, limitations, and potential drawbacks (Section 8).
The distributed system comprises simulation components, a highly available and distributed memory architecture, and the overarching distributed environment. A core requirement of this environment is the ability to detect failures in the participating simulation components and initiate appropriate recovery measures, such as restarting a failed simulator instance on an alternative computational node.
The simulation components operate within the context of distributed simulations of P/T nets. This setup necessitates the presence of the simulation components themselves and a reliable communication medium through which simulators can interact and coordinate during distributed execution.
Given the distributed nature of the simulation of these P/T nets, the system requires at least two simulators operating concurrently. By the proposed recovery mechanism, a highly distributed and persistent memory system is essential for maintaining the simulator states. As long as a simulator’s state is preserved within this memory, it remains accessible even after a failure, enabling efective recovery and continuation of the simulation process.
communication medium synchronisation service distributed storage physical machine ...
distributed environment physical machine simulator 1 P/T net with distributed synchronous channel(s) physical machine simulator n P/T net with distributed synchronous channel(s) distributed storage
distributed storage communication medium communication medium
The container orchestration platform Kubernetes [
The distributed architecture (Section 2.5) further necessitates a storage system that is not only
distributed and persistent but also highly available to maintain the simulators’ states reliably. To this
end, the open-source software Rook [
While there is no predefined upper limit on the total number of nodes, a minimum number of nodes is essential for the distributed storage system to function correctly. Specifically, a minimum of three nodes is required to ensure high availability and consistency. Operating with only two nodes introduces the risk of a split-brain scenario, whereas a single-node configuration constitutes a single point of failure.
The simulation components are developed in the context of distributed P/T nets (Section 2.2). The simulators must incorporate efective failure recovery mechanisms to support fault tolerance, necessitating a design emphasizing extensibility. The simulator Renew (Section 2.1) is particularly well suited for this purpose, as it not only facilitates the distributed execution of P/T nets but also features a modular plugin architecture that supports straightforward extension.
Moreover, Renew requires a dedicated synchronization service to coordinate the distributed simulation of P/T nets (Section 2.2). This service is responsible for determining which distributed synchronous transitions may fire together. For this coordination, the synchronization service employs Renew’s unification algorithm.
Finally, an event-driven communication infrastructure (Section 2.2) is essential for inter-simulator messaging. Kafka is employed as a communication medium by integrating within the Renew simulation framework to meet this requirement. In addition, Kafka itself is provided as a highly available communication medium in order to create resilience for the communication medium as well.
Reliable detection of simulator failures is a fundamental prerequisite for ensuring the overall resilience and correctness of the simulation framework. This prototype outlines the systematic approach adopted to address this challenge, beginning with a detailed analysis of the requirements (Section 5.1) that such a failure detection mechanism must satisfy. Based on these requirements, we then provide a specification (Section 5.2) of the detection logic, followed by a discussion of the design (Section 5.3) that guided the development of the solution. The corresponding implementation (Section 5.4) is subsequently described. Finally, the efectiveness of the proposed approach is assessed through a comprehensive evaluation (Section 5.5).
To achieve a simulation of a distributed P/T net that is as error-free and correct as possible, it is all the more important to eficiently detect as many types of occurring errors during the simulation as possible. This enables their subsequent elimination using appropriate recovery mechanisms in the following prototype (Section 6), thereby ensuring the resilience of the simulators.
For this purpose, various error detection methods described in Section 2.3 are employed, with dynamic runtime analysis, as well as monitoring and logging, playing a key role in identifying potential runtime errors within and between individual simulators. The focus of this prototype is the requirement that it should be possible to detect failures within a simulator.
Besides possible logical and semantic errors—which we already try to detect and resolve within our quality assurance process—the most critical errors to identify are fail-stop errors. These are the kinds of errors that can halt the entire simulation and potentially compromise its results.
A mechanism is therefore required that can identify the fail-stop. For this purpose, the liveliness, i.e., the availability, of the simulator is to be checked at regular short intervals.
Particular attention is given to infrastructure-related faults. Thus, if communication errors occur during the simulation—i.e., errors in data transmission between the distributed components caused by, for example, connection losses or packet loss—they must be detected and reported. Similarly, if system or hardware faults arise due to processor or memory failures or malfunctions, these errors must also be identified for recovery. Furthermore, all external errors, including those due to Force Majeure or specific user inputs, are also known but not relevant to our context of distributed simulations.
In order to detect errors in a simulator, we need an infrastructure that can check the availability of the simulators and, if necessary, start new simulators on other nodes. For this purpose, a container-based infrastructure is built that can recognize failing nodes and control a container’s lifecycle.
In addition, a corresponding HTTP endpoint is required in Renew to check the availability of the simulator. The CloudNative plugin is used to provide this endpoint.
We implement our containers with Docker [
With this implementation, we can detect failures of physical machines, as Kubernetes detects node failures and reschedules Pods on healthy nodes. Additionally, we can detect failures of the simulator Pods directly if they disturb the availability of the HTTP endpoint of the CloudNative plugin.
This means we can detect fail-stop errors using this method, including crashes, node failures, network errors, and other similar issues. However, Silent errors, like deadlocks in the internal Renew-internal simulation thread pool, would remain undetected.
This section presents the DPTNResiliency prototype, which addresses the challenge of recovering from simulator failures within distributed simulation environments. We begin by outlining the requirements (Section 6.1) that guide the development of a resilient recovery mechanism. Based on these requirements, we then provide a specification (Section 6.2) of the failure recovery behavior. This is followed by a detailed description of the design (Section 6.3) that shape the structure and coordination logic of DPTNResiliency. Subsequently, we describe the concrete implementation (Section 6.4) of the proposed mechanism within our simulation framework. Finally, the section concludes with a thorough evaluation (Section 6.5) of this prototype.
Our overarching objective is to develop a fully resilient DPTN simulation. However, the present contribution explicitly addresses the resilience of the simulators themselves.
In this context, it is imperative that fail-stop failures afecting individual simulators do not compromise the overall functionality or correctness of the distributed system. To this end, resilience must be ensured through a reactive failure recovery mechanism, as proactive strategies alone are insuficient to eliminate the occurrence of fail-stop failures.
The prototype developed in this work is designed to facilitate reactive recovery from such simulator failures. Specifically, when a simulator process crashes or experiences a fail-stop event, it must be automatically restarted without impairing the operational integrity of the distributed simulation. While a minimal delay associated with the recovery process is unavoidable, it remains functionally inconsequential to the system as a whole.
In accordance with the requirements outlined in Section 6.1, a reactive recovery mechanism is necessary to mitigate the efects of fail-stop events. Given that the distributed P/T nets simulated within the Renew framework are serializable, we adopt a checkpoint-based recovery strategy. This approach entails periodically saving and, if required, reloading the simulation state within Renew.
Each checkpoint must encapsulate the complete state of the simulation, including both the internal markings of all distributed P/T Net (DPTN) instances and the communication state with the synchronization service. This includes metadata on distributed transitions—such as whether a firing request has been issued—and a consistent record of which communication events have been processed up to the checkpoint.
To ensure the durability of checkpoints beyond the occurrence of a fail-stop event, these must be stored in a fault-tolerant, highly available storage system. This storage must not reside within a single container or be bound to a single physical node. Reliance on a single container is inherently fragile, as container failure leads to complete data loss. Similarly, tying checkpoint persistence to a single node is inadequate since a node-level failure would result in the irrevocable loss of all checkpoint data. Consequently, a resilient, distributed storage solution is imperative—one that can withstand partial system failures without compromising checkpoint integrity.
Our distributed simulation system can be viewed as a distributed database system that executes
distributed transactions, in the sense that each simulator processes events coming in from the event
broker. We can design our system in a way similar to how database systems manage transactions [
Upon initialization, a simulator will check if it finds an existing checkpoint, in which case it must assume a previous failure and recover from that checkpoint. Using the list of events from the event broker as a log, it can recover from the failure by replaying uncommited events on top of the latest checkpoint. In order for this to work, we must only commit events (i.e., mark them as processed) once a checkpoint has been created that includes the implications of the events.
To facilitate resilient simulations, we developed a dedicated Renew plugin named DPTNResiliency.
This plugin relies on other essential Renew components—specifically, the Simulator and GUI
plugins—to enable functionalities such as saving and loading simulation states. It is employed by the
DPTNFormalism plugin, introduced in [
The DPTNResiliency plugin introduces the console command startResilientSimulation, which initiates the simulation of a specified distributed P/T net. If a checkpoint is available, the simulation automatically resumes from the most recent one. This command serves as the entry point for determining the current simulation state when launching a simulator instance.
Furthermore, the event-streaming mechanism within the DPTNFormalism plugin has been extended to notify the DPTNResiliency plugin after the successful processing of each event. This notification mechanism is essential for persisting and tracking the accurate communication state throughout the simulation. Additionally, the DPTNResiliency plugin manages Kafka commits to make sure events are only marked as read once a checkpoint for them has been created.
The responsibility for checkpoint creation lies with the DPTNResiliency plugin, which utilizes the current marking and communication status. To ensure fault tolerance, especially in the event of a simulator crash, each checkpoint is initially written to a temporary location. Only upon successful creation is it copied to its final destination. Subsequently, the corresponding event is marked as consumed, ensuring that it cannot be processed again.
Checkpoints generated by the DPTNResiliency plugin must be stored in a resilient, highly available
storage system. This storage operation is triggered after each Kafka event has been successfully
processed. For this purpose, the distributed storage system Ceph [
The creation of checkpoints can also be regarded as a transaction in a (multi-)database system, making it efectively a sub-transaction in the context of the triggering of a distributed synchronous channel. As its write operations conform to ACID principles, Ceph ensures the correctness and durability of stored simulation checkpoints. Additionally, being a distributed system itself, Ceph also underlies the constraints of the CAP theorem. Being designed to prioritize Consistency and Partition Tolerance, in the event of network partitioning, it may temporarily compromise the availability of specific components to uphold global consistency guarantees, matching the specification of our system.
The DPTNResiliency plugin is implemented as an additional modular Renew plugin. The simulators, as well as the synchronization service, run in Docker containers within Kubernetes (Section 2.5), deployed via StatefulSets that include liveness probes. Kafka (Section 2.2), our event broker, is deployed with high availability on Kubernetes. Attached to each Renew container is a Persistent Volume provided by Rook, the Kubernetes Deployment of Ceph, on which the checkpoints are stored.
Our recovery mechanism is backed by the Kafka event history that acts as a highly available and distributed log. Additionally, our checkpoints are stored on a highly available distributed storage system as well. After processing an event, a simulator creates a checkpoint, using an atomic copy operation when putting it into the right place to prevent checkpoint corruption. Only after the checkpoint is created, the simulator commits its Kafka ofset, marking the event as consumed in the log and making sure it’s not consumed again. Furthermore, any simulator that fails is automatically restarted, which triggers the recovery mechanism that upholds the simulations integrity.
If a failure occurs before a checkpoint for an event is written, the simulator restores from the previous checkpoint and executes the event again, thus recovering successfully. If a failure occurs after a checkpoint is written, but before the Kafka commit has been completed, the simulator restores from the new checkpoint but is unable to execute the event again. This will lead to a global event timeout and the event will be attempted again, in which case it will now succeed, thus completing the recovery. Failures during the recovery process also fall into one of these two categories.
When drawing a parallel to database transactions, it becomes evident that our solution adheres to the ACID properties. If we view the processing of each Kafka event as a single transaction, it becomes clear that our methods guarantee atomicity and durability, in a similar way to how databases implement transaction logic. This is because we either process a single event fully or not at all (in which case we later recover and do process it), and we make the changes durable by writing them to our storage medium. The non-resilient Distributed P/T Net implementation already ensures consistency and isolation and remains unafected by our enhancements.
Altogether, these guarantees fulfill all functional requirements for our prototype, thereby confirming the resilience of the simulators. Nonetheless, one limitation persists: unresolved race conditions exist within the simulation thread pool of Renew. Although a delay mechanism has been introduced to temporarily mitigate this issue—and occurrences are exceedingly rare—it nonetheless imposes a slowdown on the distributed simulation.
To validate the functionality of our prototypes, as outlined in the preceding sections, we experimented within our Kubernetes cluster to substantiate this claim. The structure of the experiment is presented as follows: Section 7.1 details the experimental setup, Section 7.2 describes the execution process, and Section 7.3 discusses our observations.
The experimental scenario is based on a classical problem in computer science: the ProducerConsumer–Storage example. In this context, we employ a modeling approach based on distributed P/T nets (Section 2.2), featuring distributed up- and downlinks in place of standard synchronous communication channels, as depicted in Figure 1.
The scenario comprises three components: a producer, a consumer, and a storage unit. The producer operates within a cyclic process in which a message is first generated and subsequently transmitted to the storage via a distributed downlink. Conversely, the consumer follows a cyclic process in which messages are actively retrieved from storage—again via a distributed downlink—and subsequently consumed. This configuration implies the presence of two active entities: the producer and the consumer. In contrast, the storage component, which exclusively features distributed uplinks, remains entirely passive. 7.1. Setup As the foundation for this experiment, we employ the distributed system described previously in Section 4, comprising three physical machines. Within our Kubernetes cluster, we deploy a highly available Kafka cluster to serve as the communication medium. Furthermore, we deploy four components—each a Renew instance—using Kubernetes StatefulSets: the synchronization service, Producer, Storage, and Consumer. Each of these components consists of exactly one Container in a Kubernetes Pod with a replica size of one. All StatefulSets are configured with liveness probes that target endpoints exposed by the CloudNative Renew plugin. Moreover, each component is provisioned with 5 GiB of highly available persistent storage via Rook/Ceph.
Prior to initiating the experiment, we ensure that all system components are returned to their default state. To this end, we delete the four StatefulSets, if present, and erase the data stored in their associated volumes. Additionally, we remove all Kafka topics and consumer ofsets to prevent residual data from afecting communication in the upcoming simulation. This reset procedure is critical, as remnants of prior experiments could otherwise influence the outcomes.
ScriptCommand: Try to load file startscript_storage.txt Opening gui...
Passing args to gui...
Initialising CheckpointStorageServiceImpl with no previous checkpoint Starting Simulation... ...
Simulation initialized.
INFO: Consumed event UpdateUplink from topic receiveMessage.
INFO: Consumed event UpdateDownlink from topic sendMessage.
INFO: Consumed event RequestFiring from topic sendMessage.
INFO: Record ConfirmUplink sent successfully on topic sendMessage.
INFO: Consumed event ConfirmUplink from topic sendMessage.
INFO: Consumed event ConfirmDownlink from topic sendMessage.
INFO: Consumed event ConfirmFiring from topic sendMessage.
INFO: Fired Confirm Transition of channel: sendMessage in Storage.
INFO: Record UpdateDownlink sent succcessfully on topic receiveMessage.
INFO: Consumed event UpdateDownlink from topic sendMessage.
INFO: Consumed event UpdateDownlink from topic receiveMessage.
INFO: Consumed event UpdateDownlink from topic sendMessage.
INFO: Consumed event RequestFiring from topic sendMessage.
INFO: Record ConfirmUplink sent successfully from topic sendMessage.
INFO: Consumed event ConfirmDownlink from topic sendMessage.
INFO: Consumed event RequestFiring from topic receiveMessage.
INFO: Record ConfirmDownlink sent successfully on topic receiveMessage.
INFO: Fired Request Transition of channel: receiveMessage in Storage.
INFO: Record UpdateDownlink sent successfully on topic receiveMessage.
INFO: Consumed event ConfirmUplink from topic receiveMessage.
INFO: Consumed event ConfirmUplink from topic sendMessage.
INFO: Consumed event UpdateDownlink from topic sendMessage.
INFO: Consumed event ConfirmFiring from topic sendMessage.
...
Once the setup has been completed, the experiment can be initiated. The simulation commences with the deployment of the four system components. To verify correct execution, the logs of each Pod are inspected to ensure that the simulation is actively running.
Following confirmation of execution, a brief waiting period is introduced. This period is suficiently long to allow the simulation to make measurable progress, yet not so extensive that it reaches completion during this interval.
Subsequently, a fail-stop fault is emulated by deliberately terminating one of the simulator Pods. Upon automatic restart, the process resumes, and the simulation continues until it reaches completion.
k8user@artpc17:~$ kubectl get pods -n dptn NAME READY STATUS RESTARTS AGE consumer-statefulset-0 1/1 Running 0 60s producer-statefulset-0 1/1 Running 0 60s storage-statefulset-0 1/1 Running 0 60s syncservice-statefulset-0 1/1 Running 0 60s k8user@artpc17:~$ kubectl delete pod storage-statefulset-0 -n dptn pod "storage-statefulset-0" deleted k8user@artpc17:~$ kubectl get pods -n dptn NAME READY STATUS RESTARTS AGE consumer-statefulset-0 1/1 Running 0 115s producer-statefulset-0 1/1 Running 0 115s storage-statefulset-0 1/1 Running 0 21s syncservice-statefulset-0 1/1 Running 0 115s
ScriptCommand: Try to load file startscript_storage.txt Opening gui...
Passung args to gui...
Initialising CheckpointStorageServiceImpl with checkpoint Storage-232.rst Starting Simulation... ...
Simulation initialised.
INFO: Record RegisterDownlink sent successfully on topic RegisterTopic.
INFO: Subscribed to topic: receiveMessage.
INFO: Subscribed to topic: sendMessage.
INFO: Consumed event ConfirmDownlink from topic receiveMessage.
INFO: Consumed event ConfirmFiring from topic receiveMessage.
INFO: Fired Confirm Transition of channel: receiveMessage Storage.
INFO: Consumed event UpdateDownlink from topic receiveMessage.
INFO: Consumed event UpdateUplink from topic receiveMessage.
INFO: Consumed event UpdateDownlink from topic receiveMessage.
INFO: Consumed event RequestFiring from topic receiveMessage.
INFO: Record ConfirmDownlink sent successfully on topic receiveMessage.
INFO: Fired Request Transition of channel: receiveMessage in Storage.
INFO: Record UpdateDownlink sent successfully on topic receiveMessage.
...
Upon initiating the simulation, the logs of all simulator instances consistently indicate that the
simulation is actively running. As illustrated in Figure 4, Kafka events are successfully transmitted and
received, confirming the correct operation of the simulation. In this context, the UpdateUplink and
UpdateDownlink events represent communication with the synchronization service regarding updates
to the up- and downlinks of the registered distributed synchronous channels. The RequestFiring
event is employed to coordinate a distributed firing across all relevant simulators. Upon successful
execution, the ConfirmUplink, ConfirmDownlink, and ConfirmFiring events confirm the resulting
state changes. Detailed specifications of these event types are provided by Clasen et al. [
Following the deletion of a simulator Pod, Kubernetes automatically initiates a replacement Pod to restore the simulation topology. As depicted in Figure 5, one simulator Pod exhibits a delayed startup relative to the others, corresponding to the previously deleted instance.
Subsequently, once the newly instantiated Pod begins to receive Kafka events, Figure 6 demonstrates that it resumes participation in distributed transitions, including up- and downlink operations. Moreover, an examination of the logs from the remaining simulator Pods verifies that interaction with the restarted Pod is functioning as expected.
This experiment can be replicated with either the Consumer or the Producer Pod, yielding analogous results. These observations collectively substantiate the resilience of our simulator architecture.
A key advantage of the proposed concept lies in the inherent resilience of the simulators. This resilience ensures that the failure of individual simulators during runtime does not compromise the integrity or continuity of the overall simulation process.
Furthermore, the system enables the execution of resilient distributed simulations over extended durations—potentially spanning several weeks or even months. This capability markedly enhances the system’s usability, as it alleviates user concerns regarding potential simulator failures during long-term simulations.
A notable drawback of the proposed system concerns the complexity of the underlying execution infrastructure. Deployment necessitates the orchestration of multiple physical nodes within containerized environments such as Kubernetes. These infrastructural requirements impose substantial demands on the system’s architectural and operational design.
Another disadvantage, albeit with a very low probability of occurrence, is the potential for deadlocks arising from the interaction between the DPTNResiliency plugin and the internal Renew simulation thread pool. Specifically, to utilize Renew’s existing functionality for saving simulation states, the simulation must be paused. However, there is no guarantee that all queued events will be processed before the pause takes efect. Given the highly asynchronous architecture of Renew, a rapid succession of pause and resume operations—especially in conjunction with checkpoint creation—can, under rare circumstances, cause the internal thread pool to enter a deadlock state. If such a deadlock occurs, it halts the simulator and remains undetectable. This issue is currently the subject of ongoing investigation.
One limitation of the current system design is the non-resilient nature of the synchronization service. Ensuring system stability under failure conditions necessitates the development and integration of supplementary recovery mechanisms for this component.
The checkpointing-based recovery strategy, while critical for fault tolerance, introduces additional computational overhead. This overhead negatively impacts the performance of individual simulators and the overall simulation, resulting in increased execution times. In particular, delays may occur in the processing of events transmitted via Kafka, further contributing to reduced simulation eficiency.
A further limitation stems from the current checkpointing policy, which creates a checkpoint for every Kafka event consumed. This leads to the generation of a substantial number of checkpoints during the simulation. Future work will focus on optimizing checkpoint frequency to minimize this overhead while maintaining resilience.
Research conducted by Moldt et al.[
In contrast, the studies presented in [
A related yet distinct approach is proposed in [
This contribution builds on the article by Clasen et al. [
After introducing the foundational concepts—Renew, distributed P/T nets, failure semantics, resilience strategies, and Kubernetes (Section 2)—this work delineates the central research problem (Section 3): the design of a resilient simulation framework for P/T nets within a Kubernetes-based cloud environment. The distributed system described in Section 4 comprises a communication medium, highly available storage, and simulation components for distributed P/T nets.
Subsequently, we present the developed prototypes. In Section 5, Detecting Failures of Simulators, we examine strategies for reliably identifying faults within simulation components. Building upon these insights, Section 6, Recovering Failures of Simulators, introduces the novel Renew plugin DPTNResiliency, which facilitates the automated recovery of failed simulation instances based on checkpointing.
To evaluate the proposed system, we employ the well-established Producer-Storage-Consumer scenario (Section 7). During simulation, a simulator instance is intentionally terminated. Recovery is demonstrated as a new instance seamlessly resumes execution using state checkpoints generated by its predecessor. Finally, Section 8 ofers a critical assessment of the approach’s strengths, limitations, and potential trade-ofs, followed by a contextualization within the scope of related research (Section 9). 10.2. Future Work The immediate next steps involve systematically resolving existing workarounds and eliminating current race conditions. This efort is expected to reduce the overhead associated with the current approach significantly. Furthermore, we intend to eliminate the need for checkpoint creation on every consumed event, thereby further enhancing performance.
Concurrently, a more advanced and fully automated testing infrastructure is required—one that can realistically emulate a broad range of failure scenarios, including those studied in the field of chaos engineering. Such a framework is essential to bolster the reliability and credibility of the proposed solution.
The synchronization service must also be extended to ensure fault tolerance not only at the individual simulator level but across the entire simulation system. Using the aforementioned improved testing infrastructure, the proposed concept should be rigorously validated against real-world models to substantiate its empirical robustness.
Additional refinements are conceivable concerning the health metrics reported by Renew. Increasing the granularity and precision of these metrics could facilitate more nuanced operational responses and may enable the early detection of system-level anomalies such as deadlocks.
A further avenue for research pertains to the scalability of the simulation framework. To date, a ifxed number of simulators has been employed, resulting in uneven load distribution during simulations. Exploring dynamic scaling strategies could mitigate this ineficiency and unlock new levels of performance.
• . . . Bing Translate in order to: Translate Text. • . . . DeepL in order to: Translate Text. • . . . ChatGPT in order to: Rephrasing.
• . . . Grammarly in order to: Grammar and spelling check, Repharsing.
After using these tool(s)/service(s), the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.