In this work, deep convolutional neural networks are used to automate the process of feature extraction from CCTV images. The extracted features serve as a strong basis for a variety of object recognition tasks and are used to address a tracking problem. The approach is to match the extracted features of individual detections in subsequent frames, hence creating a correspondence of detections across multiple frames. The developed framework is able to address challenges like cluttered scenes, change in illumination, shadows and re ection, change in appearances and partial occlusions. However, total occlusion and similar persons in the same frame remain a challenge to be addressed. The framework is able to generate the detection and the tracking results at the rate of four frames per second.
Copyright c by the paper's authors. Copying permitted only for private and academic purposes. across the frames and can be used to address a tracking-by-detection problem accurately. So, in a novel way features are used to address a tracking problem. Tracking is formulated as the correspondence of the detections across multiple frames and is achieved by matching the extracted features of individual detections in subsequent frames. The main contributions are:
A framework for real-time pedestrian detection and tracking using CNNs is developed
A new algorithm is developed to establish correspondence between the detections across the frames The framework addresses challenges such as partial occlusion, variations in illumination, changes in pose, shape and scale of pedestrians, cluttered backgrounds and total occlusions for short periods. The framework is not able to handle total occlusions of long periods and fails to address the problem of having similar appearing persons in the same frame. 2
Tracking is de ned as the creation of trajectory of an object in an image plane and a tracker assigns correct
labels to the tracked objects in di erent frames of a video. There are three fundamental aspects of pedestrian
tracking that are analogous to object tracking: 1) detection of the pedestrian in the video frame, 2) tracking
of the detection, and 3) analysis of the tracks for the speci ed purpose
Recently, there has been a signi cant performance improvement in the eld of image category classi cation
and recognition by training a deep CNN with millions of images of di erent classes
Motivated by the success of image classi cation and recognition, attempts have been made to exploit the
usefulness of deep CNN for tracking tasks.
The developed framework uses CNNs both to detect pedestrians within the frames and track across the frames.
A state-of-the-art object detection framework, Faster R-CNN
A simpli ed layout of the framework is provided in Fig. 2. The CCTV image frames are input to the detector that detects and localises individual pedestrians. Features from the cropped images of the pedestrians are extracted by a pre-trained CNN. The developed algorithm is used to make correspondences of the detections across the frames and ids are allocated to individual detections. Tracking results are shown by overlaying the ids of the detections on the respective frames. 3.1
A(1000;1) : : :
A(1000;i(k)) The last layer before the classi cation layer (FC-8) of the CNN generates a vector of 1000 features for each input image. Individual detections from each frame are in form of a bounding box around pedestrians. Subsequently, the detections are cropped and fed to the CNN that generates a matrix of feature vectors. Mathematically this can be represented as Eq. 1, where F V~i(k) denotes the matrix of feature vectors of the i detections for a single Frame k, the set fA(1;i(k)); ::::; A(1000;i(k))g denotes the activations of ith detection in Frame k, and i(k) denotes the number of detections in Frame k.
2 A(1;1) : : : A(1;i(k)) 3 ... . . . ... (1) (2) 2x(1;k) : : : P C~i(k) = 46 ... . . .
y(1;k) : : : x(i;k)3 . . 7 . 5 y(i;k)
The centroids of the detections can be expressed by Eq. 2. Where P C~i(k) denotes the matrix of x and y coordinates of the centroids of i detections in Frame k. Correspondence is established by calculating a feature distance and a pixel distance between every pair of detections in two consecutive frames. Let F Vi(k) and F Vj(k+1) denote respectively the feature vectors for i and j detections in Frame k and Frame k + 1. The normalised feature 1For MATLAB implementation visit https:==github:com=debaditya unimelb=CNN pedestriantracking/.
F Vj~(k+1)j
Pd(i(k);j(k+1)) = jP C~i(k) jP C~i(k)jjP Cj~(k+1)j
P Cj~(k+1)j distance between the two detections Fd(i(k);j(k+1)) is expressed as Eq. 3, where jF V j denotes l2-norm of a real vector FV. Let P Ci(k) and P Cj(k+1) denote the centroids for i and j detections in Frame k and Frame k + 1 respectively. Similarly the normalised pixel distance between the two detections Pd(i(k);j(k+1)) is expressed as Eq. 4.
A distance matrix Fd(k+1) for the feature vectors is generated from the normalised pairwise feature distances and is represented by Eq. 5. A distance matrix for the pixel distances Pd(k+1) is generated from the normalised pairwise pixel distances and is represented by Eq. 6. The matrices Fd(k+1) and Pd(k+1) are combined using a weight w (0 w 1). The combination result is called a tracking matrix Td(k+1) and is de ned by Eq. 7. Where ti(k);j(k+1) represents the weighted additions of Fd(i(k);j(k+1)) and Pd(i(k);j(k+1)).
Fd(k+1) = 6 4
: : : . . .
Fd(i(k);1) : : :
Fd(1;j(k+1)) 3 . . . In the rst frame, the ids are generated randomly and tracked in the subsequent frames. The number of generated ids in the rst frame is equal to the number of detections. For the detections in the subsequent frames either an id is assigned from the previous frame (which involves the matching based on the minimum distance criteria) or a new id is generated (which is for the case a new person enters the frame).
Let the set ft(1;i(k)); ::::; t(i(k);j(k+1))g denote the weighted distances from the detection j in Frame k + 1 to all detections in Frame k. The minimum value of the set ft(1;i(k)); ::::; t(i(k);j(k+1)g is used to make correspondence of j(k + 1)th detection in Frame k + 1 to the 1st; ::::; i(k)th detections in Frame k, only if this minimum value is below a threshold. Fig. 3(a) illustrates the process of establishing correspondences for this case, where detection 1 of frame k + 1 is compared with i(k) detections of Frame k for a correspondence. If the minimum value of the set ft(1;i(k)); ::::; t(i(k);j(k+1))g for a detection j(k + 1) in Frame k + 1 is above the threshold, no correspondence is made to the Frame k, but the detection is compared to the detections of previous z frames for a match. This is explained in Fig. 3(b), where detection 1 of frame k + 1 is compared with all the detections from Frame k to Frame k z and each frame can contain di erent number of detections (a,b,g,h,i and j).
If a match is found, a correspondence of j(k + 1)th detection is made to the corresponding id of the detection in (k z)th frame . If there is no match after comparing the previous z frames, the detection is assumed as a new pedestrian entering the frame. The new pedestrian is allocated a new id and it is tracked in the subsequent frames. If a pedestrian leaves the scene or is totally occluded in Frame k + 1, the the corresponding detection in Frame k will not have any match in Frame k + 1, but, that id will be stored in the database for future correspondences. However, if the algorithm is able to re-identify the pedestrian after total occlusion in the z previous frames, it is allocated the corresponding id of the detection in the (k z)th frame.
Multiple correspondences from j(k + 1) detections to i(k)th detection might happen, if j(k + 1) > i(k). Such situations may be resolved by creating a correspondence of j(k + 1)th detection to the n(k)th detection having the least value of the set ft(1;i(k)); ::::; t(i(k);j(k+1))g. The correspondence of unallocated detections is done by using the second least value of the set ft(1;i(k)); ::::; t(i(k);j(k+1))g if it is below the threshold. If not, then the unallocated detections are compared to the detections of previous z frames for a match. 4
Town centre dataset2 was used for evaluation in the study. First 30 seconds of the video was used at a reduced frame rate of 8 frames per second. The detection and tracking are evaluated separately using tracking matrices. The results of the detections and tracking are shown in Fig. 4.
Multi-object tracking precision (MOTP) and multi-object tracking accuracy (MOTA)
M OT P = Pi;t dit (8) M OT A = 1 Pt(mt + f pt + mmet) (9)
Pt Ct Pt gt
Where dit is the distance between the detection and the ith pedestrian (from the ground truth) and Ct is the number of matches found in time t. mt, f pt and mmet are the number of misses in detection (false negatives), number of false positives and the number of mismatches in the correspondence respectively, and gt represents the number of pedestrians present at time t. MOTP is the total error in estimated position of detections over all frames, averaged by the number of correspondences made. Higher value of MOTP signi es low accuracy of the bounding boxes around the object. Higher values of MOTA signi es high accuracy in tracking. Experimental results of MOTP and MOTA for the dataset are 27.92 pixels and 71.13 % respectively. 5
Tracking is achieved by creating a correspondence of detections of two consecutive frames only (provided that there are no multiple correspondences or correspondence to z previous frames). Hence, the appearance of pedestrians are updated over time and the framework is robust to change in appearance (pose, shape and scale). The detector misses some of the detections due to total occlusions and hence explains the low value of recall. Another contributor to lower recall values is that the detector misses pedestrians that appear smaller due to their distance to the camera. This can be alleviated in a multi-camera setting, where pedestrians that are missed in one camera are likely to be detected in another camera. On a closer observation, the low value of precision 2available at: http:==www:robots:ox:ac:uk=ActiveV ision=Research=P rojects=2009bbenf old headpose=project:html is due to the false detections created by the re ection of the pedestrians in a glass panel that is present in the dataset. High value of MOTP is due to the inaccuracy of the bounding boxes of the detected pedestrians. This is insigni cant considering the high resolution of the dataset. Low value of MOTA is mainly due to the large number of misses in the detection and partially due to the false detections and mismatches in the correspondence.
A framework is developed for real-time detection and tracking of pedestrians in CCTV image frames using CNNs. A new algorithm is developed for making correspondence of the detections across multiple frames. The detector is able to overcome the challenges of variations in the illumination, cluttered backgrounds, partial occlusions and changes in the scale. The tracking algorithm is able to track pedestrians with 71.13 % accuracy and addresses the problem of changes in appearance (pose and shape) and total occlusions for short periods. However, total occlusions for longer periods remains a challenge to be addressed for future work. To improve the accuracy, it is proposed to perform the evaluation and estimation of pedestrians' future trajectories from past observations (e.g. Kalman ltering) for overcoming the problem of unpredictable pedestrian movements. To address the problem of total occlusions and similar persons, an average representation of individual pedestrians (for all the tracked frames) can be used.
This research was supported by a Research Engagement Grant from the Melbourne School of Engineering and the Melbourne Research Scholarship. The authors thank Active Vision Laboratory, Department of Engineering Science, University of Oxford for the publicly available dataset and ground-truth data.