We consider the problem of people counting in video surveillance. This is one of the most popular tasks in video analysis, because this data can be used for predictive analytics and improvement of customer services, traffic control, etc. Our method is based on the object tracking in video with low framerate. We use the algorithm from [1] as a baseline and propose several modifications that improve the quality of people counting. One of the main modifications is to use a head detector instead of a body detector in the tracking pipeline. Head tracking is proved to be more robust and accurate as the heads are less susceptible to occlusions. To find the intersection of a person with a signal line, we either raise the signal lines to the level of the heads or perform a regression of bodies based on the available head detections. Our experimental evaluation has demonstrated that the modified algorithm surpasses original in both accuracy and computational efficiency, showing a lower counting error on a lower detection frequency.
Counting people passing through certain zones of a public infrastructure, such as pedestrian crossings, sidewalks, squares, etc., is a practically important task. There are many solutions to this problem, one of them is object tracking. The task of object tracking is to create tracks for each person. A track unambiguously corresponds to a person. It marks this particular person locations in all frames in which he or she is visible. In order to count people a signal line is usually specified in the frame (see fig. 1). If the track crosses the signal line, we can say with confidence that the person also crossed it. ? Publication is supported by RFBR grant 18-08-01484
We propose a fully automatic people counting algorithm. The algorithm takes as input a video stream fFigi=1 of frames captured by stationary camera and signal line that specified by an ordered pair of points (La; Lb) on the frame. The output of the algorithm is a set of events fEigi=1 that represented by triples of values Ei = (ki; ri; di). The first value indicates the number of the frame where the signal line was crossed, the second value specifies the coordinates of the bounding box and the last value indicates the direction of the signal line intersection.
Our solution is an extension of the algorithm described in [
The modern methods of tracking are based on tracking-by-detection. There are a lot
of ways to detect the desired object on the frame. Three most popular methods are:
detection of body [
The head-tracking approach is well suited to track people in a crowd: usually video surveillance cameras are installed above the height of the person, where heads in a crowd can be seen better than full bodies. Heads are more resistant to overlapping than
A Distributed Tracking Alg. for Counting People in Video by Head Det... 3
bodies. The number of ready-made solutions and data for training is less than for bodies.
There are methods that use body parts detectors for tracking [
After detection we need to bind all the detections to tracks. As in the detection task,
there are a lot of methods. The first group of algorithms for creating tracks is greedy
algorithms. In most online algorithms tracks are constructed frame by frame, each frame
creates a matrix of the cost of matching new detections and existing tracks, then the
problem of matching is solved by a greedy algorithm (searching for the maximum in
each row/column) [
Recently, neural networks have been used more often in tracking. For example,
in paper [
We use the solution from [
Baseline works in online mode and use the Hungarian algorithm [
The baseline algorithm consists of the following steps: (1) detection; (2) evaluation of the speed of detections using visual tracking; (3) prediction of the position of tracks by the Kalman filter; (4) matching; (5) extrapolation of the tracks; (6) detection of signal line crossing events.
Proposed improvements to some of the steps above are described below. 3.2
Since heads are seen better in the video and are less prone to occlusions we decided
to use the head detector based on SSD [
Datacenter
. . . During the experiments we realized that IOU metric used in the basic algorithm is not suitable for head matching. As head bounding boxes are small enough due to the errors in position and speed estimation bounding box pairs required to match are closely positioned, but have no intersection. IOU equal to zero acquired in the case which leads to track partitioning. Therefore we suggest to increase by s times the size of the bounding boxes (while saving the center of the bounding box) before matching them using IOU metric. This approach solves the problem described above. We got the problem of missing signal line intersections as head tracks are located on a human height and the signal line is located on the ground. Problem reveals itself most on the scenes where the signal line is placed orthogonal to the camera. Therefore we offer two solutions: rising of signal lines and body bounding box regression by head. Rising of Signal Lines We propose to raise the signal line to the level of the human head (see fig. 3). This solution reduces the computational cost and time of the algorithm as it doesn’t require any extra steps to project head track to the ground plane.
The signal line can be raised either manually or automatically. Automatic approach we propose is using the following anatomical fact: the height of the head is 18 of the
A Distributed Tracking Alg. for Counting People in Video by Head Det... 5 height of the entire human body. We can use a body detector to calculate average height of human body for the scene. And raise the signal line to a height equal to 78 of the average height of a person on the scene.
This approach has a drawback: people’s growth is different, which means that people’s heads are on different planes, while people’s legs are always on the same plane. Therefore, the raised signal line doesn’t look so clearly since it’s not clear where the plane is located. Bounding Box Regression by Head To solve the drawbacks that arose in the previous paragraph we propose to use regression of a body by a head, which is trained for the specific scene. Our idea consists of two stages: combining the detections of heads and bodies for the scene and training linear regression on the received data. Combining head and body detections First of all we launch head and body detectors. To combine heads and bodies a matrix of head and body correspondence is built on each frame.
CM k; i; j = costmatch(Bk; i; Hk; j ); i 2 [1; nbk]; j 2 [1; nhk] (1)
In eq. (1) nbk; nhk – the number of body and head detections on the k-th frame, Bk; i; Hk; j – the bounding box of the i-th body and j-th head on the k-th frame. We calculate matching cost as: costmatch(Bk; i; Hk; j ) = (iohk; i; j ; if iohk; i; j 0; else
I1 ^ 1 dist(Bk; i; Hk; j ) iohk; i; j = area (intersection(Bk; i; Hk; j ))
area(Hk; j ) dist(Bk; i; Hk; j ) = centery(Hk; j ) height(Hk; j ) top(Bk; i)
In eq. (2) I1 is threshold for iohk; i; j (we are using I1 = 0:5), 1; 2 – minimum and maximum normalized distances between the center of the head and the upper point of the body (we are using 1 = 1 and 2 = 1). So at least half of head bounding box area should be inside body bounding box area and vertically head center shouldn’t be far from body top by at least one head height.
Next the assignment problem is solved using the Hungarian algorithm to maximize cost. Combined head and body detections with non-zero match cost form training dataset for a regression (see fig. 4).
Linear regression After the previous step we have the data to train a linear regression. The training dataset consists of head and body bounding box pairs: (Bi; Hi). The regressor predicts the following values: (Bh; Bw; shif tx), where Bh; Bw – height and width of the body bounding box, shif tx – a shift from the center of the head to the center of the predicted body normalized by head width: 2 (2) (3) (4) (5) (7) (8) Bcx = centerx(H)
Bcw + shif txHw 2
Bcy = Hy
This approach helps us keep the signal line on the ground plane, which solves drawbacks of the previous approach.
shif tx = centerx(B)
centerx(H)
Hw Prediction is done by linear regression with quadratic members: Bch = k0 + k1Hx + k2Hy + k3Hw + k4Hh + k5Hx2 + k6HxHy+
The same way Bcw; s\hif tx are predicted, but with separate sets of coefficients. After learning coefficients of linear regression on training dataset, we use predicted value to restore body bounding box: : : : + k13HhHw + k14Hh2 (6)
A Distributed Tracking Alg. for Counting People in Video by Head Det... 7
Now we have a choice to do speed estimation by visual tracking with head or regressed body detections. As heads are less prone to occlusions visual tracking of them may be more reliable. But body bounding boxes are several times larger and have more visual information to track. We check both choices on experimental evaluation. The entire frame is not required to find events of crossing the signal line. We can limit the detection area to the area around the signal line – region of interest (ROI). This solution speeds up the SSD detector as it is slower for images with higher resolution.
If the signal line is horizontal then the ROI is located on the top and bottom of the line. Otherwise the ROI is located on the left and right side of the line, as well as on the top of the line.
Let be 2 – the minimum angle between signal line and horizon, wmean; hmean – the average width and height of body detections and (xa; ya), (xb; yb) – the coordinates of the beginning and the end of the signal line.
x1 = min(xa; xb) swwmean
1 + x2 = max(xa; xb) + swwmean
1 + y1 = min(ya; yb)
shhmeancos y2 = max(ya; yb) + shhmeancos sin
2 sin 2 (9) (10) (11) (12)
Then A = (x1; y1; x2; y2) is the region of interest. sw; sh in equations 9-12 are parameters. After visual testing we selected the following values for these variables: sw = 2; sh = 1. 4 4.1
For an experimental evaluation of our algorithm we need datasets filmed by static
camera with body tracks markup. Video sequences should be long enough to evaluate people
counting quality. If dataset provides head tracks markup it allows us to check effect of
body regression by comparing with head tracking and raised signal lines (section 3.4).
Most of the public datasets including popular MOTChallenge dataset [
As a quality metric we use the average error of counting the number of intersections
(events) [
A Distributed Tracking Alg. for Counting People in Video by Head Det... 9
have no correspondences in the reference labeling. We say that an event Ei in the input
set of data matches the event Eci in the reference labeling if they correspond to the same
person crossing the signal line at the same time. We match all events as described in the
paper [
After the events have been matched we divide videos to segments with 10 reference events and calculate the following characteristics on them: – GT seg is the number of reference events on the segment; – F P seg is the number of unmatched events from the algorithm on the segment; – F N seg is the number of unmatched events from the reference events on the segment; – Eseg = F P seg F Nseg is an error on the segment.
GT seg
i=1 ENseg , where N is the number of the segThen final error is calculated as E = PN ments. 4.3
Rising of Signal Lines At first we have tested baseline with modifications proposed in sections 3.2, 3.3 and manually raised signal lines as described in section 3.4. The algorithm is marked as heads-no-regression in the results table (see table 2) and parameter s shows how many times head bounding boxes have been increased.
Experiments with rising the signal line clearly shows the advantage of the head detector over the body detector. The head detector gives a significant increase in an accuracy reducing the error by 2 times. The algorithm without matching modification (s = 1) has poor results at low FPS due to the previously voiced problems which shows importance of the modification proposed in the section 3.3.
Body Bounding Box Regression by Head Next we have tested body bounding box regression by head. As mentioned in section 3.4 there are two alternative ways to apply it. There are two algorithms we have tested (see table 3): – heads-regression-vistrk — baseline with modifications proposed in sections 3.2, 3.3, visual tracking of regressed bodies to estimate speed and signal lines on the ground plane (section 3.4);
– heads-vistrk-regression — baseline with modifications proposed in sections 3.2, 3.3, visual tracking of heads to estimate speed and signal lines on the ground plane (section 3.4).
As you can see the second configuration gives better results. Visualization showed us that visual tracking of heads is more reliable as they are less prone to occlusions. It worth noting that heads-vistrk-regression performed better than heads-no-regression (s = 2) on low detection frequency.
Limiting the Detection Area Next we tested limiting of the detection area
(section 3.5). It allowed us to increase the speed of the algorithm almost without affecting
the counting error (see table 4). Detection area reduced by almost 60% on some of the
videos.
We have proposed the algorithm of counting people in a video, which is an extension
of the algorithm described in [
For the future work we plan to use automatic camera calibration algorithms (see
fig. 5) [
A Distributed Tracking Alg. for Counting People in Video by Head Det... 11