During the COVID-19 pandemic, the use of a people tracking system could have been crucial, particularly in sensitive environments, such as hospitals. DPPL Hallway Tracker is a framework that uses security camera footage to determine which rooms in a corridor a person has entered. It generates a database containing all the people identified and allows quick identification of potential cases of infection based on the time spent in a room and its maximum capacity. DPPL Hallway Tracker is structured in two phases: detection and re-identification. In the first phase, it exploits Mask RCNN to identify people and room doors. In the second one, it uses the deep association metric model from DeepSORT to re-identify a person as he leaves a room.
COVID-19 Pandemic
Managing a pandemic has proved to be a dificult
challenge despite the technological developments of the past
decades. Containment measures based on restrictions on
personal mobility (such as lockdowns) have proved to be
very efective for infection containment [
individual in a closed environment is a central problem
Face masks, in combination with good room ventilation,
help to reduce the risk of transmission. However, it is not
suficient to eliminate all the risks. Tracking operations
are required to ensure the identification of the chain of
tagion. Tracking turns out to be even more essential in
public settings, such as public ofices and hospitals [
people already known to verify the person’s identity. Another contribution, in addition to the general approach networks, built from scratch or starting from existing ones.
and the risk of contagion increases with exposure time. the level of saturation of the room given its maximum contacts and the estimation of the relative risk of con- people’s movements, exploiting the characterization of
Some countries, such as Italy and Germany, used spe- tion features to determine a distribution of the positions OS, poor estimation of distances and related false posi- array) that “describes” the way they appear in the scene. CEUR
ceur-ws.org in tracking people entering and leaving rooms facing a corridor. The use of appearance features turns out to be suficiently robust to allow correct identification, even if it is less efective in recognizing people who reappear in the corridor without leaving a room.
This report describes the project’s workflow, from the description of the datasets to the results’ analysis. 2. Related works multiple people entering the same room collapse at the same value, thus providing no valuable information for the ID attribution when a person leaves the room. On the contrary, the use of a re-identification network based on appearance features in DeepSORT is functional for the current application and is therefore also implemented in this project.
In today’s literature, at best of our knowledge, there
are no studies aimed at analyzing the specific context of
tracking and re-identifying people who enter and leave
rooms. Pedestrians on streets or people moving around
indoors are usually the focus of most approaches. Other
works specialize in counting people in some particular
environments. For example, Rabaud and Belongie [
The absence of a similar application makes the comparison between the implementation proposed in this project with a baseline more complex. Therefore, in the following Sections, the individual modules that constitute it are compared with corresponding existing solutions, in the attempt to ofer an objective yardstick on the choices made. 3.1.1. Door detection
tuned with a dedicated dataset, assembled for the purpose.
It includes a selection of 2773 out of 3000 RGB images of the DeepDoors2 dataset [24], which is freely available online. These images represent one or multiple doors in diferent outdoors and indoors scenarios, which do not necessarily correspond to a corridor: in fact, the large majority of them represent doors from the front.
They also include obstacles that partially occlude part
of the doors. The annotations in the DeepDoors2 data
set are provided as additional images where each one
has a black background and diferent coloured masks for
the doors. Being interested in this project more in the
Figure 1: General scheme of the R-CNN Mask framework. portion of space occupied by the door than in the profile
The layers indicated with the letters C and P are convolutional of the door itself, all the images are re-masked to segment
layers that represent the backbone network. The classic pyra- exclusively the door casing. Hence, almost all images
mid architecture improves the detection of objects of various have quadrilateral-shaped masks (thus with four vertices
sizes. only). Moreover, the generated annotation files are no
more encoded as images like in the original DeepDoors2
dataset, but they are fully compatible with the COCO
task. More specifically, it employs the Mask RNN frame- dataset specifications [ 25]. In fact, the annotation files
work [
Initially, the image is passed as input to a convolution- "height":1920, "file_name":"frame.jpg"}, based Feature Pyramid Network [23], which has the task ... of extracting meaningful information from diferently- ] sized feature maps. An object can appear in the fore- } ground (and therefore very large in the image) or further away from the camera; hence, this pyramidal structure {"annotations": [ facilitates its detection. The features thus extracted are {"id": 519, "iscrowd": 0, passed to the Region Proposal Network (RPN), which "image_id": 514, "category_id": 1, produces several Regions Of Interest (ROI), each with its "segmentation": [[587.52,...,1097.77]], bounding box. At this point, the first-mentioned ROI- ""abrbeoax"":: 1[2416076.82.08,75}8,1.407,295.90,809.02], Align is applied and its result is passed to the second ... stage of the network, from which a series of fully con- ] nected layers allow to refine the position of the bounding } box, the class of the object it contains and its mask.
Moreover, assuming the camera to be static and, there- Table 1 fore, the position of the doors to be fixed over time, this An example of the formatting of JSON files containing image project exploits two distinct models: one for the door annotations according to COCO specifications is represented detection only and the other for people detection. Door in this table. The first row shows the data structure used detection is applied just in the starting phase of the frame- to list all the images in the dataset, the second row instead work while, from then on, people detection is performed. shows the one used to specify the annotations associated with The process of generating the two models and the related each image, thus including the mask (“segmentation”) and the results are analyzed below. bounding box (“bbox”). The “category_id” field is always set to 1, as there is only one category (door or person, depending on the dataset).
sets. These subsets are disjoint; the training set contains 70% (1941) of the images, while the remaining 30% is equally divided between the validation and test sets (416 each).
With the new dataset available, called Dppl, we
finetuned the model pre-trained with the COCO dataset, mated mask is considered to be True if its IoU is greater
which is available on the framework’s GitHub reposi- or equal than k, false otherwise.
tory. Consequently, ResNet101 was used as the backbone, The primary challenge metric for the COCO dataset is
and training was done in the same manner as the frame- AP@[.50:.05:.95] (usually referred to simply as AP), which
work’s authors. In particular, we trained the head only is the average AP for IoU (Intersection over Union) from
for the first ten epochs; for the following thirty epochs, 0.5 to 0.95 with a step size of 0.05. This metric is also used
we fine-tuned stages four and above of the backbone too; to evaluate the results of our test set. In particular, with
ifnally, in the last ten epochs, we extended the training the Dppl dataset and the training procedure described
to the entire network. Unlike [
On the test set, the AP metric was used to assess the cluded in the dataset: the door on the right that is parquality of the results produced by the training. AP, the ticularly “thinned” from the perspective is indeed not acronym for Average Precision, computes the average detected. Precisely for this reason, the framework proprecision value for recall values over 0 to 1. In practice, vides a specific graphical interface that allows adding AP is computed as the mean of precision values at a set of new door positions, as shown in Section 4.3. equally spaced recall levels, as defined by the following formula 3.1.2. People detection
people detection is also generated. Mask R-CNN with where, given (⋅) the precision, ( ) = max ∶̃ ≥̃ ( )̃ the weights of COCO is already alone able to detect and and = 101 in COCO. AP@k stands for the average pre- segment people with acceptable accuracy. However, finecision for IoU (Intersection over Union, i.e. how much tuning was done using a dedicated dataset built specifithe predicted mask overlaps with the ground truth) of k. cally for the occasion from videos captured along a hallMore specifically, in the computation of AP@k, an esti- way. More in detail, the dataset contains 793 frames captured in a corridor by a 1080x1920 px resolution camera that was positioned a few centimeters from the ceiling (approximately 2.9 meters from the floor) with a vertical image layout. In the scene, six people appear walking down the hallway and entering/exiting the adjoining rooms. They wear various types of clothing (including a white coat to simulate the presence of a doctor); they are of diferent ages and all wear face masks. One of the people has a foot cast and crutches. All frames are handannotated to generate high-quality masks, accurately respecting the person’s shape. The related annotation ifles follow the COCO specifications, as described before.
The split of files between training (555 images), validation (119) and test (119) sets follows the same proportion as the Dppl dataset.
With this second dataset available, called dPPL, we once again fine-tuned the model pre-trained with the COCO dataset. All the Mask R-CNN’s parameters are kept the same, but Gamma Contrast is used as a data augmentation technique in conjunction with horizontal lfipping in this case.
Figure 4 shows the graph of the training and validation losses. As for the performance on the test set, Table 2 shows the comparative Average Precision values between the use of a model trained only with COCO and that obtained by doing fine-tuning with the dPPL dataset.
This second option provides better results for both AP and AP@.75. The same applies for the Average Accuracy. These good results should be evaluated considering
Method
COCO only COCO+fine-tuning on dPPL
AP 70.5 76.3
AP@.75 92.9 95.5
Acc. 99.08% 99.74% the not very high number of images that compose the dataset. Indeed, environments with completely diferent illumination and compositions will certainly attenuate the good performances provided by this model.
sufice to ensure accurate tracking. As mentioned above,
one can use additional information extracted from the
images within more or less complex systems, which may
exploit appearance, movement and shape features. An
example is DeepSORT [
In this project, Deep Cosine Metric Learning [26], the same used in DeepSORT for appearance re-identification, is used. It applies a variation of Softmax classifier called Cosine Softmax Classifier, which allows obtaining a different representation space in which compact clusters are formed based on the appearance features. This is achieved by first applying the 2 normalization, which uses the 2 -norm to normalize the input values so that, if squared and summed, they would result in the value 1, and, secondly, by normalizing the weights. Finally, the cosine softmax classifier is applied, which is defined as follows: ( = | ) = exp( ⋅ ̃ ) ∑=1 exp( ⋅ ̃ ) where is a free scaling parameter. 1 × 10−8; moreover, the input images are scaled to 128x64 px.
beneficial for the network training since it provides improved results according to the CMC Rank@K and mAP metrics1, as shown in Table 4. The table also shows the results of two state-of-the-art solutions on the original MARS dataset. Both largely outperform the solution proposed in this project, however, they also use much more
distractors to make it more realistic. The goal of the Re- sophisticated methods or networks with many more padataset after applying object instance segmentation.
Method
Rank1
Rank5
DCML on MARS
DCML on masked MARS B-BOT + Attention & CL loss
MGH
that employs Mask R-CNN and the above-mentioned re-identification network. It also provides additional features to improve the user experience and optimize the search for people. More precisely, the workflow is the following: the first frame is first passed as input to Mask R-CNN for doors detection. Once doors are located, that frame and the following ones are passed to the same network (with diferent weights) for people detection. The portion of the image containing each person is then multiplied by the corresponding mask (to have a black background) and, after being resized to 128 x 64 px, is passed to the re-identification network. The latter has its head cut of so that it outputs an array of size 128 (generated by the last Dense layer). This array is a descriptor of the person’s appearance and is used by the framework’s main algorithm to associate a unique identity ID with each person.
Data: _ ,
Result: People identified 1 _ ← [] ; 2 for in _ do 3 , ← ; 4 ← [[0] ∶
[
get_person_identifier ( _) 7 , ← ifnd_nearest( ); if pID == -1 then
// New person appeared , ; After selecting the video, the first frame is analyzed // Person in the corridor or exited from a through mask-RCNN to locate the doors in the scene. room If one or more doors are not detected, the user can man- 12 end ually add additional ones, as shown in Section 4.3. Only 13 _ ← at that point, the analysis of the following frames begins. 14 end Pseudocode 1 shows the main steps. As previously de- 15 for in _ _ _() do scribed, Mask R-CNN is again used to identify people, 16 if not in _ then while the re-ID network provides the people appearance 17 if close to a room then descriptors. At that point, for each person, the find_near- 18 // Person entered in a room est function allows identifying the already-known closest 19 else identifier to the detected descriptor, if any. In this way, 20 // Person disappeared from the scene it is possible to determine whether that person already (may due to an occlusion) appeared in the past and, depending on their position 21 end and on the knowledge derived from past frames, a log is 22 end added to the database if they are leaving a room. If there is no similar person, the algorithm adds a new one to the 23 end scene. The final for loop finds all people who were in the environment up to the previous frame but are now missing. In this case, there are two alternatives: the per- people who last left the corridor, then moving on to all son may either have entered a room (if in the preceding the known people. The similarity between two identifiers frame they were suficiently close to the relative door) ID and ID is computed with the cosine similarity, as or may have disappeared, for example, because they left follows the hallway or are temporarily occluded. To improve = ID ⋅ ID the eficacy of the algorithm, the framework starts track- ‖ID‖‖ ID‖ ing a person when he appears entirely in the scene and Two identifiers are more similar as the cosine similarhis bounding box is at a minimum distance from the ity goes to one. Hence the need to define, for each of image edges. Furthermore, it uses the area of the bbox the listed searches, a threshold that defines when two to interrupt (temporarily or not) the tracking when an descriptors must be considered suficiently similar (and object/person occludes the subject or when the tracked therefore belonging to the same person) or not. The person has nearly entirely entered a room. choice of the threshold heavily influences the tracking
A fundamental step is the one implemented by the efectiveness. In the various phases a diferent threshold ifnd_nearest function, shown in Pseudocode 2. It uses is used, more specifically: (1) if a person is walking along diferentiated searches to find the already-known person the corridor without other people in the close vicinity with the most similar identity to the one passed as input. and, if compared to the previous frame, that person has First, it searches among the people visible in the scene in not moved too far from their previous position in the the previous frame. In case of failure, if the detection is scene, then a greater dissimilarity between the descripclose enough to a door - according to a given threshold tors is tolerated; (2) in other cases, the threshold is set to - it searches among the people who are known to be in a value between 0.85 and 0.9. Section 5 discusses some that room. As a last chance, it starts searching among the critical issues regarding the choice of the threshold.
a room (Figure 7). In the latter case, the interface highlights the riskiest situations (for example, if the room capacity has been exceeded) in addition to providing all records linked to the entered ID.
frameID personID roomID "in/out/new" where frameID is an incremental value representing the currently processed frame, personID is a unique integer associated to a person (diferent from the identifier representing the way that person looks in the scene), roomID is the ID of the room the person is entering/leaving - if any - and it is equal to −1 otherwise. The last label has the value “in” or “out” when “roomID” is diferent from −1, while it assumes the value “new” when a new person appears in the scene.
For simplicity, the database is implemented via a simple CSV file containing all the logs, but more complex and scalable solutions (such as NoSQL) are also possible. Knowing the video framerate, the framework derives an estimate of the time spent in the room, to highlight possible dangerous situations. The same is done by counting the number of people in the same room and alerting when the maximum capacity is exceeded.
4.3. GUI ent setups of incremental dificulty. In the first setup, peoA simple user interface, implemented with the PySim- ple walk down a corridor one after the other, in a perfect pleGUI library, is also available to provide the user with lfow that limits the occasions when two or more people more flexible interaction with the framework. The user are simultaneously in the same room. This modality alcan select a file or directory containing the needed frame lows focusing mainly on an inter-frame re-identification images, as well as add new doors that Mask R-CNN did and on the correct detection of people entering and leavnot detect. In this second case (shown in Figure 6), by ing the rooms. In the second setup, multiple people can using a simple library such as Matplotlib, it is possible to enter the same room. The challenge, in this case, is to be ofer a response in real-time on the location of the new able to identify the identity of a person when he leaves doors and their heights (used by the algorithm). Finally, the room. The results show that the algorithm can handle at the end of the processing of all frames, the user can a wide range of situations with ease, producing results search all the times a particular ID has entered and left that are similar - if not identical - to the ground truth.
First of all, it is beneficial to analyze how accurately on to analyze the accuracy of people tracking. In particthe framework can detect the presence of one or more ular, the inter-frame re-identification of a person in the people in the scene. To calculate the overall accuracy of scene scores 100% accuracy, even in the case of several the detections we used two methods. The first consists people in the corridor; the same happens when the perof considering only those frames in which a person is son leaves a room, even when more than one is inside shown entirely (i.e., he is not hidden - even partially - by it. The criticalities are mainly two: (1) the dificulty in objects or other people). The second way is to consider defining an eficient threshold for cosine similarity, since all frames, including all borderline cases in which only the method adopted is susceptible to sudden changes in a portion of a person’s arm or leg appears in the frame. the person’s position (such as front and rear vision of Figure 8 shows an example of the frames considered with the person); (2) the influence of the quality of masks proboth methods. The results - obviously better in numerical duced by Mask R-CNN on the re-identification network. terms in the first case - are shown in Table 5. A sudden change in the portion of the image taken into consideration (even without sudden movements of the
Overall (Detection) Accuracy subject) can reduce the cosine similarity.
Method 1 100% Cosine similarity can be a powerful tool for guiding the Method 2 91.76% re-identification task: limiting the search to the people Table 5 inside the room and using the cosine similarity always The accuracy of people detection computed with two methods leads to correct identifications. Nevertheless, the weakis shown here. With the first one, we only considered those nesses listed above heavily reduce its efectiveness when frames in which people bodies are shown wholly in the image. it is necessary to recognize a person who had previously The second method also includes those frames in which a left the corridor (without entering any room) and who person is only partially visible. reappears later on. Indeed, the choice of a high threshold (i.e., ≥ 0.9) makes it dificult to assign the same ID in the situation under analysis, because usually, the person will reappear in a completely diferent pose (for example, from behind and not the front) which will reduce the value of the cosine similarity. In this case, there will be no ID switches between diferent people, but each time one reappears in the scene it will be assigned a new ID.
On the contrary, lowering the threshold facilitates the ID switches, creating some cascading problems in the framework (an ID already assigned - even if incorrectly - to a person will not be re-assigned as long as the person is in the scene, not even if the one it was originally assigned to reappears). However, these problems do not afect the recognition of people leaving the rooms: the identifier produced by the Re-ID network and the similarity computed with the cosine similarity is suficient for the correct attribution of the ID. Compared to the baseline (Re-ID network trained on the original MARS dataset), it can be observed that the cosine similarity of the same person in two diferent situations (frames) is greater (by 1-2%) when assessed with our method.
Figure 8: The frame on the left is an example of those con- As a final benchmark, the accuracy of the logs (seen sidered with Method 1 for calculating the Overall Detection as the ratio of the logs equal to ones of the ground truth Accuracy. The person’s body is entirely included in the scene. over the total number of them) produced in the tests is The frame on the right is instead an example of those consid- equal to 50%. The accuracy goes up to 84% if we also ered with Method 2, that takes also into account all borderline include those logs with labels “in” and “out” that difer cases in which only a portion of a person’s arm or leg appears only in the person ID from the ground truth (but only in the frame. In this case, the two people in the scene are only if that ID is a new one, and therefore if there is no ID dpeatreticatlelyd vbiysitbhlee amnoddtehl.e arm of the uppermost person is not switch with a previously known identity). When a person enters a room, the relative log at the exit is always correct, as already mentioned above. As for performance, an
Having ascertained that the framework can detect the Nvidia Tesla K80 is capable of processing 1.4-1.5 frames presence of people with good reliability, we then move per second.
We also ran a test in a setup with slightly diferent specifications. In fact, the recording device was placed at eye level, tilted almost parallel to the floor and with an image ratio of 16:9. The results obtained are comparable to those indicated above, although tracking people in areas very distant from the camera (and therefore at lower resolution) turns out to be more critical. Under these conditions, it is quite easy for two diferent subjects to appear very similar even to the human eye. An example is shown in Figure 9. Ultimately, the framework is most efective when the distance to the doors is not excessively large.
point for developing a framework capable of tracking people entering and leaving multiple rooms. The use of a re-ID network that exploits the masks produced in the detection and segmentation phase leads, even in the tests performed, to improvements in identification.
A project extension might be able to address some of the remaining issues: (1) the enrichment of the datasets of people and doors could lead to better detection in several more challenging contexts: for example, as discussed above, the detection and segmentation of doors “thinned” from perspective remains dificult; (2) using a dynamic threshold and investigating complementary solutions to 51–61 the re-identification network could alleviate the dificulty of assigning the same ID to a person who reappears in the corridor without leaving a room. The study of solutions for tracing people entering and leaving the rooms is of great importance for the application developments that it can have. It not only allows contact tracing in the event of pandemics but it can be also used for other contexts, as for the analysis of the movements of patients and medical operators and the optimization of hospital wards.