Sanitizing railway stations is a relevant issue especially due to the recent evolution of the Covid-19 pandemic. In this work, we propose a multi-robot approach to sanitize railway stations based on a distributed Deep Q-Learning technique. The framework relies on anonymous information from existing WiFi networks to localize passengers inside the station and to develop a map of possible risky areas to be sanitized. Starting from this map, a swarm of cleaning robots, each one endowed with a robot-specific convolutional neural network, learns how to on-line cooperate inside the station in order to maximize the sanitized area depending on the presence of the passengers.
In recent years, the spreading of diseases such as the Covid-19 has emphasized the problem
of sanitizing large and crowded public environments like railway stations. In the present
work, our aim is to design a solution for the sanitizing by the Deep Q-Learning technique
in a real case of study of interest for Italian railway infrastructure manager RFI s.p.a., in a
real environment ofered by the most important italian railway station of the capital, Roma
Termini. The framework relies on anonymous information from existing WiFi networks to
localize passengers inside the station and to develop a map of possible risky areas to be sanitized.
Starting from this map, we propose a decentralized approach where a swarm of cleaning robots,
each one endowed with a robot-specific convolutional neural network, learns how to on-line
cooperate inside the station in order to maximize the sanitized area depending on the presence
of the passengers. In the multi-robot sanitizing system literature, the prominent approach
is based on coverage path planning (CPP) [
Robot 1 ER-Buffer1
ER-DQN1
Robot 2 ER-Buffer2
ER-DQN2
Robot N ER-BufferN
ER-DQNN a1 a2 aN
Server
diferent applications like exploration [
Our multi-robot sanitizing problem can be described as follows. Starting from a gridmap representing the environment to be sanitized, we define as the set of possible heatmaps (i.e., priority distributions) on the map , and as the set possible free-obstacle positions of . In this setting, we assume agents, tasked to sanitize the environment , each one endowed with a set of single-agent actions . Our aim is to find a set of agent-specific strategies ( 1, … , ) such that each ∶ × → drives an agent towards prioritized areas, in coordination with the other agents, in order to maximize the global cleaning efect. This distributed approach is mainly designed to support the scalability: we adopt a client-server approach, where each agent (client) learns a decoupled agent-specific strategy by communicating with a central system (server).
A representation of the overall architecture is depicted in Figure 1. The framework is
composed of a set of intelligent agents, representing mobile cleaning robots, each one communicating
with the central server. The role of the server (server-side) is to merge the outcomes of the
agents activities with (anonymized) data about people positions in order to produce a heatmap
for the risky areas to be sterilized. The role of each agent (agent-side) is to elaborate the
heatmap by means of an agent-specific Deep Q-Network (DQN) and to update the local strategy
considering the environmental settings and the diferent priorities in the map. In this
framework, the cleaning priority can be defined as a heatmap, whose hot/cold points are high/low
priority areas to be sanitized. Following this perspective, a state-position couple (, ) ∈ ×
is defined as a 2 channel matrix × × 2 where and are the width and the height of the
environment, respectively. The first channel of the matrix represents the cleaning-priority
on the environment, whose elements are real numbers in the interval [
In our framework, the update of priorities is performed by the server, which collects the
outputs of the single agents, and integrates them considering the position of people and obstacles.
More specifically, the cleaning priority is computed from the position of clusters of people
by modeling possible spreading of viruses or bacteria. In our setting, we exploit the periodic
convolution of a Gaussian filter (, 2) every steps, where , 2 and are suitable parameters
that can be regulated depending on the meters/pixels ratio, the timestep, and the considered
typology of spreading (in this work we assume a setting inspired to the aerial difusion of
the Covid-19 [
(,) = { if > 0; otherwise.
(1) (2) as the sum of the element-wise priorities from matrix in the area sterilized by the agent (i.e. where (, ) = 1 ). The value in Equation 1 is then exploited to define the reward for the agent : Specifically, when an agent sanitizes a priority area, the reward is equal to the cumulative value ; otherwise, if no priority is associated to the cleaned area (i.e., = 0) a negative reward < 0 is earned (we empirically set = −2 for our case studies). This way, agents receive a reward that is proportional to the importance of the sanitized area, while routes toward zero-priority areas, such as obstacles or clean regions, are discouraged. Notice that in this framework, when the action of an agent leads to an obstacle (collision), no motion is performed. This behavior penalizes the agent (no further cleaning are performed), thus producing an indirect drive towards collision-free paths. Moreover, as long as an agent moves through the environment it leaves a wake of cleaned space behind. This way, since the priority of already visited areas is 0, agents can indirectly observe their mutual behavior from the priority update, in so avoiding explicit communication, hence robots in our experiments are not directly aware of the position of the other agents which is indirectly estimated from their paths.
A graphical representation of the environment is shown in Figure 2. We selected a region of space of 100 × 172 meters in front of the rails, where people usually stands waiting for the incoming trains. From that region we also isolated shops, stairs and walls as obstacles to be avoided by the robot during the sanitizing process (black areas in the Figure, 2, right). Agents can move by one pixel in any direction, hence the set includes 8 actions (4 linear and 4 diagonal) while, in case one action leads to an inconsistent location (obstacle or out of bound) the agent stays in the current location. In this setting, we propose two case studies: in the ifrst one we assess the system performance during the learning phase considering diferent numbers of robots (2 to 6 robots) while, in the second case, a more realistic scenario is considered, where the cleaning performance of robots are assessed considering an increasing number of moving clusters. In this first case study, we show how the learning performance of the proposed approach scales over the number of cleaning agents. The starting point of every robot in the heatmap is set at random, because in our study we want to find a solution that is independent by this initial condition. We designed a training process where, at the beginning of each episode, a random number of clusters is selected and each cluster is randomly positioned inside the station. Specifically, each obstacle-free location of the map has a 0.02 probability of generating a cluster. Each episode ends when agents successfully clean up to the 98% of the map or until a timeout is reached (400 steps are performed). During the training process we collect the overall reward as the sum of the single agents rewards and the number of steps needed to accomplish the task. This setting is intentionally designed to train agents to address a generic distribution of priorities, which can be generated during daily cleaning processes. As for the execution time, the number of steps needed to accomplish the task, namely to clean the 98% of the map, decreases with the increasing number of agents. Specifically, the 2 agents configuration needs 174 steps on average to accomplish the task, while the 4, 6 and 8 agents ones need 127, 112, and 94 steps, with a time reduction of 27%, 12%, and 16%, respectively. Also in this case, the time reduction indicates that the proposed approach successfully scales to diferent number of robots. In order to assess the performance of the system into more realistic scenarios, we propose a diferent setting by considering diferent number of clusters and a simulated WiFi server that periodically updates the position of clusters at a specific rate (once every 15 steps). The numbers of clusters have been selected according to the average number of visitors-per-hour of the considered portion of the station (see Figure 2); moreover, during the runs, the values are designed to be randomly reduced up to the 30% in order to simulate the departure/arrival of passengers in the station.
In this work we proposed a scalable multi-robot sanitizing framework based on a distributed Deep Q-Learning technique, suitable for the eficient cleaning of large and crowded indoor environment such as railways stations. The proposed simulated experiments indicate that, as expected, the cleaning performance of the framework is proportional to the number of robots and inversely proportional to the number of people in the station. To asses the performance of our framework we proposed a worst-case test, where a large number of moving people is scattered (uniformly distributed) all around the station and robots should cover a wide area to perform the task. This setting is challenging compared to a real railway station, where people are often grouped near specific areas like shops, info points or ticket ofices (see example in Figure 2, left) and robots can easily converge to those areas to maximize the sanitization efect. As future research activities, we plan to extend our pilot study by testing the proposed framework in a more realistic scenario, considering more complex robotic models and daily recorded data about the real people distribution in the station.