In this paper, we propose a new approach to estimating waiting time in queue based on object tracking and person re-identification by upper body. The task we are considering is practically important in video analysis, because this data can be used for predictive analytics and improvement of customer services. The main idea of the proposed method is to use upper body detections instead of full body detections. This decision is due to the following fact: in queues, the upper bodies are more visible. Using re-identification allows us to perform video analytics on sparse frames and thereby increase the computational eficiency of the estimation algorithm. Also in this work, we introduce a novel upper body regression by head, upper body random size augmentation to improve re-identification performance in real-world scenarios and a method for calculating metrics for queue waiting time estimation algorithms. Our experimental evaluation showed that the proposed algorithm has a high accuracy of queue waiting time estimation.
The task of estimating waiting time in queue is practically an important task, since algorithms for solving this problem are used in commercial products to improve customer services. There are many solutions to this problem, one of them is object tracking. The task of object tracking is to create tracks for each person and every track unambiguously corresponds to a person. It marks this particular person locations in all frames in which person is visible.
We use appearance embeddings obtained using the re-identification algorithm (next, we will call it as Re-ID appearance embeddings) to bind detections on new frames with existing tracks or to create new tracks. Using re-identification allows us to perform video analytics on sparse frames and thereby increase the computational eficiency of the estimation algorithm. This fact is very important for real-world tasks, because in practice we have strict limits on computing resources. Also, in this paper, we use upper body detections for re-identification algorithm n o i t c e t e D d a e H n o i s s e r g e R y d o B r e p p U
Input Video Stream
instead of full body detections, because in queues, the upper bodies are more visible.
In this work, we proposed a fully automatic algorithm for estimating waiting time in queue. The input of the algorithm is a video stream {}=1 of frames captured by single stationary camera, the coordinates of the region of interest (ROI) and — fragment length in seconds. The output of the algorithm is a set {}=1 of maximum waiting time every seconds.
Summarize, in this article we ofer the following ideas: • novel upper body regression by head; • new upper body random size augmentation; • neural network algorithm for person re-identification by upper body detections; • object tracking algorithm based on Re-ID appearance embeddings for creating tracks; • new method for calculating metrics for queue waiting time estimation algorithms.
Most of the existing methods for queue monitoring use idea of counting people who have
crossed a special signal line, which located at the entrance and exit of the queue. For this
purpose, special equipment is often used, which is installed at the entrance and exit of the
queue, for example, infrared sensors in [
There are a lot of types of object tracking algorithms, most of them are based on
tracking-bydetection. There are several approaches to detecting objects on the frame for further tracking.
For example, detection of body [
As mentioned earlier, we use person Re-ID appearance embeddings to bind detections to
the tracks. Re-identification algorithms have been particularly developed with the growing
popularity of neural network methods. Occlusions — is the one of the main problems in
reidentification task. To solve the problem of occlusions, binary object masks [
The fig. 1 shows the general scheme of the proposed method. Our method consist of five main steps: • Head Detection — at this step, the head detector is used to search for the heads of all people on sparse frames; • Upper Body Regression — at this step, our proposed upper body regression by heads is used to regress the upper bodies for each head found on sparse frames; • Re-ID Appearance Embeddings — at this step, all the regressed upper bodies are fed to the input of the neural network for re-identification, which generates embeddings; • Tracking — at this step, Re-ID appearance embeddings are used to bind detections to the tracks or create new tracks; • Waiting Time Estimation — after tracking we get set of tracks { }=1, = { , }=1, where , — frame and coordinates of the bounding box, respectively. Using the set of tracks, we can calculate waiting time in the queue as the maximum lifetime (by the lifetime () of the track , we mean the duration of its existence in a video sequence) of one of the tracks from this set.
Below are detailed descriptions of the first four steps of the proposed algorithm.
In this paper, we use the head detector based on SSD [
Next, head detections are used to regress the upper parts of people’s bodies.
The main idea of the proposed method is to use upper body detections instead of full body detections for re-identification algorithm. This decision is due to the following fact: in queues, the upper bodies are more visible. Also, the upper body contains suficient visual information (a) Heuristic (b) Neural Network Regression for re-identification compared to the heads. Our upper body regression approach consist of two steps: 1. using heuristics, we find the approximate position of the upper body; 2. using neural network regression, we clarify the position of the upper body. The First Step In our heuristics we are using the following fact from anatomy: the width of a person’s body is on average three times the width of a person’s head, and the height of a person’s body is on average eight times the height of a person’s head.
Under the upper part of the human body in this work, we understand the part of the body that is equal in height to two heights of the human head, then: upperBodyleft = headleft − headwidth,
upperBodytop = headtop, upperBodywidth = 3 · headwidth, upperBodyheight = 2 · headheight, (1) (2) (3) (4) where (headleft, headtop, headwidth, headheight) — the coordinates of the head bounding box and (upperBodyleft, upperBodytop, upperBodywidth, upperBodyheight) — the coordinates of the upper body bounding box.
The Second Step At the second step we specify the exact position of the upper body bounding box, obtained in the previous step. To do this, we developed neural network to regress two coordinates: the left and right extreme points of the upper part of the human body (upperBodyregLeft and upperBodyregRight, respectively). This coordinate refinement is especially important if the person on the frame is standing in profile (see fig. 2). The fig. 3 shows the architecture of the proposed neural network for upper body regression. Our regression neural network has a simple architecture, because we need our algorithm for estimating waiting time in queue to work in real time. Despite the simple architecture, the proposed neural network has a good regression quality.
As a result, we have the final coordinates of the upper body bounding box, which is further used in re-identification algorithm: (upperBodyregLeft, upperBodytop, upperBodyregWidth, upperBodyheight), where upperBodyregWidth = upperBodyregRight − upperBodyregLeft.
MobileNetV2 t u o p o r D
Re-identification Neural Network We use the solution from [
Upper Body Random Size Augmentation Consider, for example, the case of full body reidentification. Usually, when training a neural network, images are used for re-identification, in which people are represented in full growth. However, in real-world scenarios, due to occlusions (or detector mistakes), there may be cases where the detector finds only a part of the body, for example, only the upper part of the body. The use of such detections for re-identification can seriously reduce the quality of person re-identification, since the neural network has previously “seen” people only in full growth.
The same is possible for person re-identification by upper body. To solve this problem, we introduce a novel upper body random size augmentation to improve re-identification performance in real-world scenarios. The main idea of the proposed augmentation is to randomly change the boundaries of the upper body bounding box (see fig. 4).
Let (left1, top1, width1, height1) — the coordinates of the upper body bbox obtained by the regressor, (left2, top2, width2, height2) — the coordinates of the upper body bbox after using the proposed augmentation, width and height — width and height of the full body bbox, then: left2 = max(0, left1 + 1 · (− width1, width1)), top2 = max(0, top1 + 2 · (− height1, height1)), width2 = min(width, left1 + width1 + 3 · (− width1, width1)) − left2, height2 = min(height, top1 + height1 + 4 · (− height1, height1)) − top2, where 1, 2, 3, 4 — pre-selected constants, in this paper we use 1 = 3 = 4 = 0.25 and 2 = 0.05, and (, ) is a function that returns a random integer value ∈ [, ]. (5) (6) (7) (8) (9) (10)
Upper Body Regression To train our regression neural network (see section 3.2) we used a
modified CrowdHuman public dataset [
We use the solution from [
Let {}=1 is a set of re-identification descriptors for detections on new frame and { }=1 is a set of lists of re-identification descriptors for all existing tracks. Let’s set the cost matrix ∈ R× as follows: , =
max (, ), where ∈ { } =1, ∈ [1, ], ∈ [1, ], ∈ [, ] = {︃ , , if , ⩾ ℎ 0, else , ∈ [1, ], ∈ [1, ], where ℎ is the threshold for the cosine similarity, in this article we use ℎ = 0.4. Next, for cost matrix the assignment problem is solved using the Hungarian algorithm to maximize cost.
1. using the head detector (see section 3.1), head detections were obtained for each image;
2. using the Detectron2 [
Re-identification To train our re-identification neural network we used a modified MSMT17 [21] public dataset that combined the training and test parts (we named the resulting dataset MSMT17 Merged). Our modification is as follows: we used our upper body regression neural network to find the upper body detection for each image. The obtained upper body detections were used to train person re-identification by upper body.
Waiting Time Estimation Algorithm To test the entire algorithm proposed in this paper we need datasets captured by static camera with head tracks markup. Also, the video sequences should be long enough to evaluate the quality of the estimating waiting time in queue. We used 6 videos from the collection of the Video Analysis Technologies company. These videos are obtained from real security cameras in stores and other public places where queues are possible, and allow us to bring testing closer to real-world scenarios. In addition, we have developed an efective procedure for marking up long real videos. The table 1 provides detailed information about each test video.
In this paper, we propose a new method for calculating metrics for queue waiting time estimation algorithms. Our method is as follows: 1. we divide videos to segments of equal length seconds (with the possible exception of the last segment). In this paper, = 300 seconds; 2. for each segment we calculate = max∈ () and = max∈ (), where and are sets of the ground truth and predicted tracks on the segment, respectively. () is the lifetime of the track on the segment. That is, for each segment in the ground truth and the results of the algorithm, a track with the maximum lifetime on the segment is sought. Important: track lifetimes are end-to-end, meaning they are shared across the entire video, not separate for each segment; 3. for each segment we calculate the absolute and relative error: = | − |, = ; = = ∑︀=1
, ∑︀=1 4. for the entire video we calculate the mean absolute error and mean relative error in percentage: where is the number of segments.
In this section, we present experimental results for the entire proposed algorithm. We consider
4 types of experiments:
• Detectron2 Det. — in this experiments, we used person re-identification by full body
and detections from Detectron2 [
The table 2 provides detailed information on the results of each of the above experiments.
The experimental evaluation shows that our hypothesis that the upper parts of the body are seen better than the full bodies in the queues is correct. In addition, huge error in experiments with re-identification by full body and detections from Detectron2 related to the fact described in the section 3.3.
A comparison of the results of experiments with heuristic upper body detections and experiments with neural network upper body detections shows that the proposed regression strategy has a positive efect on the quality of estimating waiting time. This is because heuristic detections often have wide bboxes that either get a lot of background or other people into them (see fig. 2). And this entails a re-identification error.
The experimental evaluation shows that the proposed Upper Body Random Size Augmentation for person re-identification also has a positive efect on the quality of estimating waiting time, since this augmentation solves the problem described in the section 3.3. Solving this problem allows us to reduce the number of broken tracks and, as a result, improve the quality of object tracking.
We have proposed the algorithm to estimating waiting time in queue based on object tracking and person re-identification by upper body. Using re-identification allows us to perform video analytics on sparse frames and thereby increase the computational eficiency of the estimation algorithm. In this work, we have introduced a method for calculating metrics for queue waiting time estimation algorithms. In addition, we have proposed a novel upper body regression by head, upper body random size augmentation to improve re-identification performance in real-world scenarios, which improved quality of the algorithm. [21] L. Wei, S. Zhang, W. Gao, Q. Tian, Person transfer gan to bridge domain gap for person re-identification, 2018. arXiv:1711.08565.