Data organization in video surveillance systems using deep learning

Abstract

In this paper we propose to organize information in video surveillance systems by grouping the video tracks, which contain identical faces. Aggregation of the features of individual frames extracted using deep convolutional neural networks are used in order to obtain a descriptor of video track. The tracks with identical faces are grouped using the known face verification algorithms and clustering methods. We experimentally compare frame aggregation methods using the YouTubeFaces dataset and contemporary neural networks (VGGFace, VGGFace2, LightenedCNN). It is shown that the most accurate video-based face verification is achieved with the L2-normalization of average unnormalized features of individual frames of each video track. Finally, we demonstrate that the best video grouping is obtained by sequential and rank-order clustering methods.

1. Introduction

Nowadays, the automatic organization of visual information is attracting increasing attention due to the growth of the multimedia data volume. The multimedia data organization systems are required not only for a particular user who has an archive of photographs, but also for the field of public security, where video surveillance technologies are used [ 1, 2 ]. The limited functionality and the fact that the operator is physically unable to monitor the situation in real time with growth of the number of video cameras indicate the need to improve procedures and techniques, to increase the requirements for the training of operators. One approach is the use of intelligent video analytics systems to automatically process video streams.

Dramatic increase of size of collected media data leads to necessity of their grouping [ 3 ]. Existing solutions, e.g. Google Photos or Apple iPhoto, are designed to search, organize and display images of the person. However, they were not developed to process in real-time such large amount of data from video surveillance systems [ 4 ]. For example, they collect thousands of images (frames) every second [ 5, 6, 7 ]. Consequently, there is a challenge of ordering the visitors, whose faces were observed by a surveillance system [ 8 ].

Nowadays, the state-of-the-art results in image processing, object detection or feature extraction are obtained with deep convolutional neural networks (CNNs) [ 9, 10 ]. In this paper we use the clustering techniques in order to achieve automated organizing of video data where only one person is shown. As the quality of clustering mainly depends on the correctness of measuring the closeness of examined objects, in this paper we primarily focus on choosing the most appropriate representation of videos by aggregation of features obtained from each frame of video track.

The paper is organized as follows: in Section 2, we discuss the frame aggregation techniques. In Section 3, we present the proposed approach of video data organizing and our software prototype . In Section 4, the experimental results for the YouTubeFaces (YTF) dataset [ 9 ] are presented. In Section 5, the concluding comments and future plans are given.

2. Video frame aggregation techniques

The task of paper is to divide the input video sequence of T>1 frames into M < T subsequent tracks {X(m)}, m = 1,2,...,M contained face images of one person and cluster similar tracks. Each m-th track is characterized by the indices of its start t1(m) and end frame t2(m). We denote the number of frames in the m-th track as ∆t(m)= t2(m)- t1(m)+1.

In order to group tracks contained images of one person clustering methods were used [ 11, 12 ]. To utilize them it is necessary to extract face features in each frame [ 13, 14 ], aggregate features of separate frame in descriptor for whole track [ 15 ] and then compare these descriptors. The output of the CNN’s last (bottleneck) layer of the facial image in the t-th frame is stored in the D-dimensional feature vector x(t). These features are usually matched with the Euclidean (L2) metric ρ(x(t1), x(t2)) [ 16 ]. However, when the video sequences are grouped, it is required to compute the distance ρ(X(m1), X(m1)) between tracks (subsequences of frames) X(m1) and X(m2). The most obvious way to define this distance is the computation of the mean pairwise distances between all frames:  X (m1), X (m2 )  1 t2 (m1) t2 (m2) x(t), x(t')

 t(m1)t(m2 ) tt1(m1) t't1(m2 )

However, the run-time complexity is rather high due to the pair-wise matching of all frames in these tracks causing the computation of ∆t(m1)∆t(m2) distances between high-dimensional features. Therefore, we used the following methods to match single representations of the whole tracks. 1. The distance between their medoids: 2. Average features vectors of each track are matched:

 X (m1), X (m2 )   x(m1),x(m2 ),  X (m1), X (m2 )   x*(m1),x*(m2 ), x*(mi ) 

t2 (mi ) argmin  x(t),x(t') x(t),t[t1(mi ),t2 (mi )]t't1(mi ) x(mi )  1 t2 (mi )

 x(t) t(mi ) t't1(mi ) (1) (2) (3) 3. Comparison of the median features x’(mi) of each track:

 X (m1), X (m2 )   x(m1),x(m2 ). (4) 4. Learnable pooling of video features of the m-th track using the neural aggregation network [ 17 ], which includes special attention blocks to process features of all frames. The goal of this block is to learn the D-dimensional vector q, which is used to assign each t-th frame with its weight a(t) using the softmax function:

exp(qT x(t)) a(t)  t2 (m) (5)

 exp(qT x(t')) t't1(m)

The final representation of the m-th track is computed as the weighted average of all frames with weights (5):

t2 (m) r(m)   a(t)x(t) (6)

tt1(m)

In order to improve the quality of such learnable pooling it is significant to use two sequential attention blocks [ 17 ]. Let q(0) be the first block weights and r(0) be the first aggregating features using q(0) weights (5), (6). The weights in the sequential block are computed as follows: q(1)  tanh(Wr(0)  b) , (7) where and are the learnable weight matrix and bias vector of the neurons respectively. The feature vector r(1) generated by q(1) using (6) will be the final descriptor of the m-th frame.

In order to make the video features more resistant to the conditions of observation (camera resolution, illumination, etc.) usually normalization in the Euclidean metric is applied [ 10 ]. Usually, the preliminary normalization of the features of each frame is performed. However, in this paper we also analyze the normalization of aggregated video features [ 18 ].

3. Proposed video data organizing system

We implemented a special software prototype using PyCharm from JetBrains (Python 3.6 language), OpenCV [ 19 ], Caffe and TensorFlow libraries. The data flow in our system is presented in Figure 1.

Video data

Face detection and recognition

Saving information to database

Tracking

Cluster update

Feature extraction

Track verification

Feature normalization

Track creation

At first, the faces are detected in each frame with the TensorFlow Models [ 20 ]. This repository contains different preliminary trained models of neural networks and provides interface for object detection (TensorFlow Object Detection API). In particular, faces are detected with the MobileNet SSD trained on the WiderFace dataset [ 21 ]. Further tracking of highlighted faces is processed but face detection is repeated periodically in order to: 1) precise tracking results; 2) obtain new faces and 3) mark disappeared faces. After that we extract features using particular CNN and normalize the features vector. Then consecutive frames of one person are united into one track (homogeneous segment). On the final step (Fig.1) subsequent clustering is processed: features vector of the last track is matched with the features of previously detected clusters. If the distance to the nearest cluster does not exceed a certain threshold, this track is added to the cluster and the information about the last is updated. The resulted set of selected clusters of homogeneous tracks is saved into NoSQL database Cassandra.

The architecture of the proposed software is shown in Fig. 2.

Visualization module

Software module

Camera module

Face detection module

Feature extraction module Database module

Track module

Results processing

module Clustering module Database module

Using the cross-platform Qt framework for software development a graphical user interface was created (Fig. 3). (c) (d) Figure 3. Results of working system (a – the main window, b – loading video processing and results, c – export data to file, d – real video processing and results).

4. Experimental results

In this section we describe the experimental results of our pipeline (Fig. 1). To extract features we used the Caffe framework [ 9 ] and three publicly available CNNs suitable for face recognition, namely, the VggNet [ 22 ], Lightened CNN (version C) [ 10 ], VggFace2 [ 23 ]. The VggNet extracts D = 4096 features vector in the output of “fc7" layer from 224x224 RGB images. The Lightened CNN extracts D = 256 features vector (“eltwise fc2" layer) is computed from 128x128 grayscale image. The VggFace2 is the ResNet50 model, which extracts D = 2048 features vector from “pool5_7x7_s1”. Their advantages are high velocity of image processing and high accuracy of detection.

In this paper we analysed two types of distance between frames[ 24, 25 ]: traditional L2 (Euclidiean) metric and the Student criterion (t-test):

Here we used the YTF dataset [ 26 ], which contains 3,425 videos of 1,595 different people. The shortest track duration is 48 frames, the longest track contains 6,070 frames, and the average length of a video clip is 181.3. We have calculated following indexes: AUC (Area under curve) and FRR (False Reject Rate) for fixed FAR (False Accept Rate) = 1%. Weights in attention modules (5)-(7) were trained on 1895 videos of 500 subjects from the IJB-A dataset.

The AUC (area under curve) and the false reject rate (FRR) for 1% false accept rate (FAR) for Lightened CNN, VggNet and VggFace2 features are shown in Table 1, Table 2 and Table 3, respectively. Average face detection time for one frame is 60 ms at MacBook Pro laptop.

According to these tables, one can clearly see that the proper normalization plays a significant role. The most efficient algorithm is computation of average vector of normalized features. AUC of average vector search is more by 10-12% than AUC of medoid (2) comparison and by 14-16% than AUC of aggregation module (5)-(7) training.  ( X (m1 ), X (m2 ))

D(m1 )  D(m2 ) t(m1 ) t(m2 ) (8)

Distance (1)

L2-norm->Distance (1)

Medoids (2)

L2-norm (2) medoids

Attributes averaging(3)

L2-norm attributes averaging (3)

L2-norm of average attributes vector (3) L2-norm of median (4)

L2-norm of average attributes vector (3) L2-norm of median (4)

Attributes averaging(3)

L2-norm attributes averaging (3)

L2-norm of average attributes vector (3)

L2-norm of median (4)

Also we implemented sequential clustering where threshold for resulting clusters is set by fixing the FAR value. In addition, we examined the clustering algorithm [ 27 ] from the DominantSet library [ 28 ] and the Rank-Order hierarchical clustering [29]. The results are presented in Table 4.

Total quantity of clusters is bigger than quantity of different people from YTF dataset (1,595) because different videos with one person could be mentioned in different clusters. Moreover, average processing time of YTF tracks for hierarchical algorithm is 8 minutes, while algorithm DominantSet utilized for video tracks grouping consumed more than 20 minutes. The most effective results were demonstrated by Rank-Order clustering algorithm. As usual, the most accurate results are obtained for the VggFace2 facial features.

5. Conclusion

In this paper we solved the problem of video subsequences clustering for video surveillance systems. In particular, we focused on calculating the degree of proximity of video tracks using the aggregation of features vectors extracted by deep CNNs. Experimental study demonstrated that the features vectors averaging of all frames and subsequent normalization lead to the highest accuracy of video face verification. In the future work we plan to analyze other clustering algorithms deeper in order to achieve low calculation complexity and high accuracy of data processing.

Zhu C, Wen F and Sun J 2011 A rank-order distance based clustering algorithm for face tagging IEEE International Conference on Computer Vision and Pattern Recognition 481-488

Acknowledgements

The work was conducted at Laboratory of Algorithms and Technologies for Network Analysis, National Research University Higher School of Economics and supported by Russian Federation President grant MD-306.2017.9.

References

[1] Chellappa R , Du M , Turaga P and Zhou S K 2011 Face Tracking and Recognition in Video Handbook of Face Recognition 323-351

[2] Shan C 2016 Face Recognition and Retrieval in Video Video Search and Mining , Studies in Computational Intelligence 287 235 - 260

[3] Savchenko A V 2016 Search Techniques in Intelligent Classification Systems Springer International Publishing

[4] Savchenko A V and Belova N S 2018 Unconstrained Face Identification Using Maximum Likelihood of Distances Between Deep Off-the-shelf Features Systems with Applications 108 170 - 182

[5] Chen J C , Ranjan R , Kumar A , Chen C H , Patel V M and Chellappa R 2015 An end-to-end system for unconstrained face verification with deep convolutional neural networks IEEE International Conference on Computer Vision Workshops 118-126

[6] Li H , Hua G , Shen X , Lin Z and Brandt J 2014 Eigen-PEP for video face recognition Asian Conference on Computer Vision , LNCS 9005 17 - 33

[7] Savchenko A V 2017 Deep neural networks and maximum likelihood search for approximate nearest neighbor in video-based image recognition Optical Memory and Neural Networks (Information Optics) 26 ( 2 ) 129 - 136

[8] Sokolova A D , Kharchevnikova A S and Savchenko A V 2017 Organizing Multimedia Data in Video Surveillance Systems Based on Face Verification with Convolutional Neural Networks Lecture Notes in Computer Science 10716 213 - 220

[9] Jia Y et al 2015 Caffe: Convolutional architecture for fast feature embedding Proceedings of the 22nd ACM international conference on Multimedia ACM 675-678

[10] Wu X , He R and Sun Z 2015 A Lightened CNN for Deep Face Representation Preprint arXiv: 1511 . 02683

[11] Kaufman L and Rousseeuw P J 2009 Finding groups in data: an introduction to cluster analysis (John Wiley & Sons)

[12] Savchenko A V 2017 Clustering and maximum likelihood search for efficient statistical classification with medium-sized database Optimization Letters 11 ( 2 ) 329 - 341

[13] Savchenko A V 2018 Trigonometric series in orthogonal expansions for density estimates of deep image features Computer Optics 42 ( 1 ) 149 - 158 DOI: 10.18287/ 2412 -6179-2018-42-1- 149 -158

[14] Savchenko A V 2017 Maximum-likelihood dissimilarities in image recognition with deep neural networks Computer Optics 41 ( 3 ) 422 - 430 DOI: 10.18287/ 2412 -6179-2017-41-3- 422 -430

[15] Nikitin M Yu , Konushin V S and Konushin A S 2017 Neural network model for video-based face recognition with frames quality assessment Computer Optics 41 ( 5 ) 732 - 742 DOI: 10.18287/ 2412 -6179-2017-41-5- 732 -742

[16] Goodfellow I , Bengio Y and Courville A I 2016 Deep learning (MIT press)

[17] Yang J 2017 Neural aggregation network for video face recognition arXiv Preprint

[18] Savchenko A V and Belova N S 2015 Statistical testing of segment homogeneity in classification of piecewise- regular objects International Journal of Applied Mathematics and Computer Science 25 ( 4 ) 915 - 925

[19] OpenCV (Access mode: http://opencv.org/)

[20] TensorFlow API (Access mode: http://github.com/tensorflow/models/tree/master/research/ object_detection/)

[21] Wider Face : A face detection benchmark (Access mode: http://mmlab .ie.cuhk.edu.hk/projects /WIDERFace/)

[22] Parkhi O M , Vedaldi A and Zisserman A 2015 Deep face recognition Proceedings of the British Machine Vision 6 -17

[23] Cao Q , Shen L , Xie W , Parkhi O M and Zisserman A 2017 VGGFace2: A dataset for recognising faces across pose and age Preprint arXiv: 1710 . 08092

[24] Nemirovskiy V B , Stoyanov A K and Goremykina D S 2016 Face recognition based on the proximity measure clustering Computer Optics 40 ( 5 ) 740 - 745 DOI: 10.18287/ 2412 -6179-2016- 40-5- 740 -745

[25] Nemirovskiy V B and Stoyanov A K 2017 Clustering face images Computer Optics 41 ( 1 ) 59 - 66 DOI: 10.18287/ 2412 -6179-2017-41-1- 59 -66

[26] Wolf L , Hassner T and Maoz I 2011 Face recognition in unconstrained videos with matched background similarity IEEE International Conference on Computer Vision and Pattern Recognition 529-534

[27] Pelilo M and Pavan M 2007 Dominant sets and pairwise clustering IEEE Transactions on Pattern Analysis and Machine intelligence 29 ( 1 ) 167 - 172

[28] Dominant Set Library (Access mode: https://github .com/xwasco/DominantSetLibrary) [ 29 ]