With the advent of the computational technologies (Graphics Processing Units - GPUs) and Machine Learning, the research domain of crowd simulation for crisis management has flourished. Along with the new techniques and methodologies that have been proposed all those years, aiming to increase the realism of crowd simulation, several crisis simulation systems/tools have been developed, but most of them focus on special cases without providing users the ability to adapt them based on their needs. Towards these directions, in this paper, we introduce a novel multi-agent-based crisis simulation system for indoor cases. The main advantage of the system is its ease of use feature, focusing on non-expert users (users with little to no programming skills) that can exploit its capabilities a, adapt the entire environment based on their needs (case studies) and set up building evacuation planning experiments with some of the most popular Reinforcement Learning algorithms. Simply put, the system's features focus on dynamic environment design and crisis management, interconnection with popular Reinforcement Learning libraries, agents with diferent characteristics (behaviors), fire propagation parameterization, realistic physics based on a popular game engine, GPU-accelerated agents training and simulation end conditions. A case study exploiting a popular reinforcement learning algorithm, for training of the agents, presents the dynamics and the capabilities of the proposed systems and the paper is concluded with the highlights of the system and some future directions.
• Computing methodologies → Multi-agent systems; Reinforcement learning.
With the advancements of the recent years in computing
capabilities, Artificial Intelligence and web technologies the research
domain of Crowd Simulation (CS) has gained more and more
interest. This field has grown a lot the last decade (and keeps growing)
and as a consequence, there are more and more techniques and
methods proposed. For example, crowd behaviors simulation [
The domain of crowd management and analysis had seen
interest as early as 1958 [
The research domain of Crowd Simulation for Crisis
Management (CSCM) has experienced an increasing interest the past years.
Crowd simulation is the process of simulating how a number of
entities (commonly large) move inside a virtual scene with a specific
setting [
The most suitable approach of crowd and crisis simulation
systems is the simulation of multiple individual entities [
In this paper, a novel crisis simulation system is introduced, focusing towards the creation of a prototype system, that takes advantage of the plethora of simulation and performance enhancing capabilities of a well-known game engine. The system’s key features are: (1) Ease of use: users with little to no programming skills or experience can setup a crisis scenario and simulate it through a user-friendly Graphical User Interface (GUI). (2) Dynamic environment design: a feature that allows the users to create their own building and environment based on their needs and case studies. (3) Interconnection with popular Reinforcement Learning libraries: allowing researchers to exploit popular RL algorithms for the training of the agents or to try their own algorithms. (4) Dynamic crisis management: allowing the user to model a specific structured pipeline of a crisis, for example two fires starting from diferent places. (5) GPU-accelerated agents’ training and simulation, for the support of large multi-agent systems. (6) Simulation end options, which allows the user to specify when a simulation will automatically end.
Additionally, we tested the new introduced system with a state-ofthe-art Deep Reinforcement Learning algorithm (DRL), resulting high accuracy in training and quite well evacuations of agents in an indoor environment. It should be highlighted that GPU accelerated training of the DRL was much faster than the CPU based training approaches. which boosted the hyperparameters tuning process (time consuming process) of the implemented case study.
Crisis simulation systems have several social and scientific impacts focusing, mainly, in the development of the civilization, by helping humans design safe buildings that gather many visits throughout the day. Additionally, such systems help humans to be prepared for various crisis situations (e.g. evacuation planning in indoor fire cases) by exploiting Artificial Intelligence technologies that simulate human behaviors. In addition to those, this kind of systems, provide persons the ability to design their own environments based on their need and monitor/see how these kinds of scenarios are unfolded.
The rest of the paper is organized as follows: Section 2 briefly presents some crisis simulation platforms and the research that has been done on CSCM, Section 3 introduces the prototype of the new introduced crisis simulation system, followed by Section 4 presenting a case study. Lastly, Section 5 concludes the paper by highlighting some key features of the systems and presents some future work towards the enhancement of the prototype system. 2
As mentioned before, nowadays there has been a plethora of
systems developed for the simulation of diferent crisis scenarios. For
example, Becker-Asano et al. presented a multi-agent system
focused on first-persona perception and signs, taking dynamically
changing occlusions into account [
ESCAPES, a multi-agent evacuation simulation system, presented
in 2011, which incorporated diferent agent types with emotional,
informational and behavioral interaction [
In 2013, De Oliveira Carneiro et al. presented a simulation system
to study the crowd’s behavior while evacuating a soccer stadium [
Several literature reviews and surveys have been published in
the last years, presenting advancements and important
observations regarding the direction and parts that require focus in the
domain of CSCM. Some of the most important stages when
developing a simulation tool can be derived by reading some of those
reviews. For example, S. Abar’s et al. [
The presented system is developed taking into account observations/highlights from the aforementioned related works. It has numerous advantages compared to the aforementioned systems, with the most important ones being its ease of use and its usefulness as a research tool for future studies. The development and design of the system focuses on its usability and how easily a user, without any programming skills can setup an environment and test it based on its needs. Moreover, the system can be used as a research tool by scientists to test diferent RL algorithms/models and apply them on a plethora of diferent crisis scenarios. This is achieved via the interconnection of Metis with popular RL libraries through the game engine. Additionally, by using a game engine to develop such a system, scalability, physical interactions (physics) and exploitation of GPUs is a given, as games take advantage of those things with very realistic results. The combination of these advantages, can highlight it as a unique tool for scientific communities and the general public. 3
In this section we present a prototype version4 of a novel multiagent crisis simulation system, developed over the Unity game engine, called Metis56. The main structure diagram of Metis is shown in Figure 1, which consists of three major layers: Dynamic Environment Development (DED), Scenario Design (SD) and Evacuation Simulation (ES). The first layer (DED) allows the user to design and setup the entire environment and building to be evacuated, dynamically. The second layer (SD), follows the concept of dynamic design of evacuation scenario, giving the user the ability to place pedestrians (various number of agents) in diferent parts 4The assets currently used are from “Standard Assets (for Unity 2017.3)” (https://assetstore.unity.com/packages/essentials/asset-packs/standardassets-for-unity-2017-3-32351) and the “Snaps Prototype | Ofice” (https://assetstore.unity.com/packages/3d/environments/snaps-prototype-ofice137490) Unity packages. 5The name comes from Metis, one of the elder Okeanides and the Titan-goddess of good counsel, planning, cunning and wisdom. Counsel, planning and wisdom are also required when a building is designed. https://www.theoi.com/Titan/TitanisMetis.html 6https://sites.google.com/view/metissimulationsystem of the building, designate which doors are exits, mark areas of the environment in which the pedestrians will be safe and place fires in diferent places. The last layer (ES), handles the evacuation process and the modules responsible for the simulation. The ES layer, exploits machine learning models (interconnection with popular RL libraries), includes the management of the spreading of the fires, handles the ending of simulation and gives the user the ability to manage a dynamically changing crisis. Dynamic crisis management is the ability to model the pipeline of a crisis, in our case two fires from diferent places. In the future, it could be a tsunami followed by an earthquake etc. A fist-view screenshot of the main component of the system is depicted in Figure 2. The main components of the interface are consisted of 2 User Interface Layers (UIL) and from those, one is also split into two sub-UILs (number 1 and 2 in Figure 2). These UILs focus on the design of the environment, the experiment and in conducting the final evacuation experiments, without the need of expert skills. Also, the main structure of the UILs is based on the framework presented in the Figure 1. Each UIL is consisted of multiple interactive buttons with the following functionalities (left to right, top to bottom): (1) This UIL1 has four main buttons and a scroll view content which includes additional interactive buttons with labels. The three buttons on the top-left change the category of objects that could be placed in the environment, they appear in the scroll view after a category is selected. The first button it will show all the static objects that can be placed in the environment, the second all available types of pedestrians and the last one all the sample simulation buildings (areas) that can be placed. Sample buildings are buildings created beforehand and provided for the user, with each building including placed objects and having diferent layout and number of rooms. Clicking on an object in the scroll content will allow the user to place the specific object into the environment. The last button with the magnifier icon in top-right, allows the user to filter the list of objects through a text field. (2) This UIL2 includes functionalities that can change the mode of the mouse. On the left column the button allows the user to assign a safe area in the environment. On the middle column the buttons allow the user to place fires, walls, floors and doors. In the last column the buttons reset the mouse to default and does nothing and the last button to grab and place already placed objects.
(3) This UIL3 includes buttons regarding the simulation process. The play button starts the simulation process and the gear button shows all the available options regarding the simulation ending conditions. 3.1
As mentioned above, the DED layer of the system allows the user
to design the layout of the building to be evacuated, by placing
the building’s walls. This layer can be characterized as the layer
responsible for the content generation of the environment,
commonly used in games [
The SD layer is responsible for designing the scenarios, meaning
where the fire will start, how many pedestrians will have to
evacuate the building (multi-agent approach) and where their starting
positions will be, which exit they will try to reach and where they
will be safe. The fires’ positions can be chosen with the fire
placement tool of the UIL2, which, during the design process, allows the
user to specify from which position the fire will start to spread. The
ifres start spreading when the simulation starts. Diferent “types”
of pedestrians can be placed and each type has diferent attributes,
like speed, size, color, health points etc. giving the ability to
simulate diferent human behaviors. A door is marked as an exit door
by right clicking on it. At least one door has to be marked as an
exit due to the way the pedestrians were trained. Lastly, safe areas
are used to mark a pedestrian as safe from the crisis during the
simulation, where they are trying to escape to.
3.3
The ES layer is responsible for all the functions running during
the simulation of an evacuation procedure. Starting with the fire
propagation, a very simple algorithm is employed. The fire is firstly
placed in a point in the building, with a specific maximum area and
is represented by a particle emitting object, which damages any
object that touches it. Then, when the simulation starts, the area
grows periodically and multiple fire objects are created at random
places inside the area. Simply put, the propagation of the fire,
currently, works with a random speed and direction, while the contact
of the fire with the pedestrian is enabled with collisions interfaces.
Having control over when the simulation automatically ends is
an important feature. The prototype version currently supports
end conditions like when all or a specific number of pedestrians
are safe/dead. The pedestrian’s evacuation can be done by training
the agents with RL algorithms. By exploiting the capabilities of
ML-Agents toolkit [
In this section we present and analyze how the pedestrian agents were trained, the features that the agents gather in each step, the actions the agents take to evacuate a building and the environment in which it was trained. Figure 3 depicts an indoor environment created using the Metis system and then saving it as a single area object for ease of use during the training environment setup and to keep the environment setup for further analysis. The highlighted areas with light green, inside the building, are the possible areas that an agent could spawn when an episode begun. Initially, all agents spawned inside the room marked as “1” and each one unlocked the next area (light green) one by one. An agent could unlock an area once the mean reward of the RL in the last 20 episodes was equal or higher to 0.925, as consequence of a good training for a specific area. Every time an episode begins, the agent chooses randomly between ifve possible spawn areas. Those five areas are the most recent areas the agent unlocked. Eventually, when the agent has a mean reward of over 0.925 in average of all areas, it can spawn in any area. This was done so that the agent learned gradually and all the areas were unlocked so that it can generalize correctly (without learning only 7https://docs.ray.io/en/master/rllib.html 8https://github.com/openai/baselines to escape from the specific point of the building). The red cubes inside the building are dummy fire objects which, when touched, reset the agent’s episode and set its reward to -1. The reasoning behind the use of dummy objects instead of the actual fire was to check if the agent, using the aforementioned raycast components, would eventually learn to avoid those fires. While the intense green is considered to be a safe exit for the pedestrians. The features gathered by each agent during the training/learning were 70 in total, from which 64 were gathered using three “Ray Perception Sensor 3D”9 components and 6 were calculated manually: (1) The first raycast component detected objects (static objects and fires) and is blocked by walls and doors, it casts 20 rays of 15 length (by default 1:1 meters in Unity), in a 140 degrees arc in front of the agent, responsible for detecting objects that have to be avoided (inside the room the agent currently is). (2) The second raycast component detects doors, safe exit doors and walls, with 20 rays of 25 length in an 80 degrees arc in front of the agent, responsible for detecting, doors and safe exit doors that are close. (3) The third raycast components also detects doors, safe exit doors and walls, with 24 rays of 50 length in a 140 degrees arc in front of the agent, responsible for detecting doors and safe exit doors that are far away. (4) The manually calculated features were: (a) The normalized x and z values of the safe exit door (b) The agent’s position and (c) The normalized direction from the agent to the exit door. During the training, the agent gets −0.4/ reward for each step (action) taken, −0.3/ if collides with something (static objects, walls and closed doors) and small positive rewards depending on its distance from the exit door (Eq. 1). Additionally, when the agent reached the safe area the reward was set to +1. = ( ( ), ( )) ∗ 0.3 (1) Where ( ) the normalized exit’s position, ( ) the normalized agent’s position, maxStep the maximum number of 9Rays that are cast into the physics world, and the objects that are hit determine the observation vector that is produced. actions per episode (equal to 10.000 during the training). Distance() calculates the distance between the exit and the agent.
It should be noted that the positions of the objects are normalized according to their relative position inside the building. Additionally, the reasoning behind the choice of the negative rewards is to make the agent reach the exit as soon as possible and to collide with as less objects as possible. Lastly, the reward and episode reset from touching the fire makes the agent avoid the fires and not touch them, by assigning the reward of the episode to -1.
The available actions for each agent are provided through two output branches, each with two possible actions. The first branch is responsible for the agent’s horizontal movement (left or right) and the second branch for the vertical movement (backward or forward). With this setup the agent is able to navigate to a safe exit.
Based on these parameters the agents can be trained with many diferent RL algorithms. By having access to the source code of the Metis system, all these parameters and many others can be adjusted based on the need of the experiment. At this point it should be highlighted that the Metis system will be provided under open source licensing. 4
For the sake of clarity, a case scenario was setup to present the procedure of setting up a building for a crisis evacuation planning and to also evaluate the quality of the RL based model that was trained in the previous section at escaping a diferent building in a fire crisis scenario. The creation of the building was done with the UILs and its architecture was quite simple, consisted of three main rooms and one hall. The hall was empty and was connected to the other three rooms (east, north and west rooms). Each room was decorated with many static objects, such as desks, small and large cabinets, small and large shelves and plants. The objects were placed in such way to make the evacuation of the building harder, as it can be seen in the north room in Figure 5. Four doors were placed, three connecting the rooms with the hall and one being the safe exit on the south. The safe exit door was designated as an exit by right clicking on the door. The next step, when setting up a building in the Metis system, is to designate a safe area, which when the pedestrians touch, are considered as being safe. In our case study, we designated a safe are just outside the exit. The following step is to place the pedestrians into the building. We placed a total of 25 pedestrians, scattering them around all the rooms, placing some on dificult areas. Lastly, fires were placed on all rooms, in such way that some pedestrians’ paths are partly blocked during the evacuation. Figure 5 shows the layout of our designed building for the case study, along with all the objects, agents and fires placed. During the training procedure, the environment spawned 60 agents during and each one individually started training. This is a common methodology to speed up the training process. It should be noted that, due to the fact that there were multiple agents in the same environment, they ignored each other, both physically and featurewise. For the training procedure the PPO algorithm was exploited, which is considered to be one of the most efective RL algorithms for agents’ adopting raycast observations.
In our experiment the setup of the chosen PPO algorithm was
based on a neural network which approximated the ideal function
that mapped the agent’s observations to the best action an agent
could take in a given state. The neural network set up was, input:
70, hidden layers: 512 and output: 4, with discount factor for future
rewards set to = 0.995 and the learning rate set to = 0.0003.
Figure 4 depicts the agent’s training results. The agent was trained
for 14.55 million steps (actions). The reward initially was set to
[
Figure 6 shows a snapshot at the beginning of the evacuation after the training, when the pedestrians started running towards the exit. It can be seen that most of the pedestrians find their way towards it immediately. Despite that, some of them can be seen struggling, with some being stuck running into a corner of a room. Note that to demonstrate how easy it is to setup a scenario and to better understand the whole process of creating a building and setting up the evacuation and simulation procedure, we have created a demo video10. When a simulation procedure ends, for any reason, a results window pops up to inform the user about the evacuation and the statistics. Figure 7 shows a snapshot of the info-window, highlighting the results of our case study. The info-window informs the user about the total pedestrians that survived and died at the end of the simulation. In this case study, from the total 25 pedestrians, 17 survived (agents which reached the green safe area) and 8 died (died or didn’t evacuate successfully in case of manual end). 5
Game engines have become more and more popular and have been exploited for many diferent applications besides their main target, the development of games. In this paper we present a prototype of a novel crisis simulation system called Metis. Metis is developed using a very popular game engine, Unity, and exploits many of its optimizations such as physics, particle efects, cross platform development etc. In addition to that, the Metis system can make use of trending Reinforcement Learning algorithms, to improve the simulation realism and the evacuation planning. Its interconnection 10https://tinyurl.com/MetisMABCSSDemo with popular RL libraries and its dynamic content (environment) development can establish it as a powerful research tool for basic and applied high-level research. Due to the fact that it is developed over a game engine that supports cross-platform development, it can be considered as a system that can run in multiple operating systems. As mentioned above, the most important key features of the system are its ease of use for scenario design and simulation, the ability to build case study environments dynamically, dynamic management of the crisis situation (multiple crisis situations), exploitation of various RL algorithms and well known libraries, inherent GPUaccelerated agent simulations, agents with various characteristics and behaviors and, lastly, the ability to specify simulation end conditions.
Furthermore, although the system is in Alpha version, the presented experimental results are encouraging and promising. To sum-up, by using the Metis system one can design their own building layout, place a variety of objects, agents and fires, towards the development of personal evacuation plan. Due to its simplicity the Metis system can be used by everyone, even from users without special programming skills. The aforementioned features of Metis focus on a key concept of a dynamic and general system, especially due to the dynamic crisis management. The user can start a crisis at diferent moments during the simulation, creating unique scenarios and allowing them to observe the pedestrians’ reactions.
From the results it is obvious that there is room for improvement. First and foremost, not all pedestrians found their way towards the exit, which means a diferent training approach with diferent RL algorithms has to be tested. The best one would be to train the agent for much longer, with a dynamic change of the environment (random or via curriculum learning). This means that the placed ifres’ positions have to be changed, along with the exit door, the building’s layout and the agent’s attributes (speed and size), every time the agent finishes an episode. In general, a better fine-tuning of the algorithms could provide more accurate evacuations with fewer losses. Moreover, allowing pedestrians to interact with each other (cooperative learning) will require the agents to be trained in such way that they take into account the number of agents near them or near an exit. In addition to that, future work on the system includes the introduction of other features and functionalities, such as: real time simulation statistics, more explanatory and graphical statistics at the end of a simulation, ability to build multi-level and multiple buildings, more realistic fire propagation, more types of crisis (in addition to fire, such as earthquake, flooding etc.) and allow pedestrians to interact with each other. Moreover, some important considerations towards the future improvement of the system are the incorporation of emotional and psychological features into the agents. This aspect is an important one and has been extensively studied in the literature of CSCM. Lastly, an important feature of a system that aims for longevity and extensibility is to add support for the user (auto guide) to extend the system’s functionalities.
This work is supported by the MPhil program "Advanced Technologies in Informatics and Computers", hosted by the Department of Computer Science, International Hellenic University.