This paper presents a new solution to the frontal face detection problem based on a compact convolutional neural networks cascade. Test results on an FDDB dataset show that it is able to compete with state-of-the-art algorithms. This proposed detector is implemented using three technologies: SSE/AVX/AVX2 instruction sets for Intel CPUs, Nvidia CUDA, and OpenCL. The detection speed of our approach exceeds considerably all the existing CPUbased and GPU-based algorithms. Thanks to its high computational efficiency, our detector can process 4K Ultra HD video stream in real time (up to 27 fps) on mobile platforms while searching objects with a dimension of 60×60 pixels or higher. At the same time, its processing speed is almost independent of the background and the number of objects in a scene. This is achieved by asynchronous computation of stages in the cascade.
The need to identify people in millions of photos uploaded daily to social services has led to significant progress in the solution of the problem of detecting faces. New methods are distinguished by invariance with respect to the pose and face expression. Moreover, they are also capable of operating in conditions of complex illumination and strong occlusion. However, many algorithms that demonstrate outstanding performance on a face detection benchmark have very high computational complexity. This circumstance prevents their use for video analysis.
The object of our interest is megapixel video analytics systems that require fast and accurate face detection algorithms. Such systems run on equipment whose computing power is often greatly limited due to the increasing demands for a compact form factor and a lower cost. Because of this, the increase of frame rate or frame resolution is often carried out at the expense of the performance of detection (such as large size objects search only, frames skipping, etc.). Moreover, the use of cameras capable of shooting video with a 4K Ultra HD resolution increases the amount of generated data by several times. In conditions when it is impossible to reduce the search area (for example, by motion analysis), even optimized detectors based on Viola-Jones method are unable to operate effectively at such a video stream resolution.
Despite the fact that the development of modern face detection methods is progressing towards an increased invariance with respect to the head position and the occlusion, we are considering here only a particular problem of frontal faces detection. Our goal is to achieve a high performance of detection at a low computational complexity of the detector, which is difficult to achieve when dealing with this task in the framework of the most general definition of the problem. At the same time, the frontal position of a person in relation to the camera is natural for many scenarios of using video analytics systems. That is why these detectors are so popular in practical applications.
In this paper, we present a frontal face detector based on a cascade of a convolutional neural
network (CNN) [
It is not possible to build a simple and rapid detector with a high precision and response to all the
possible face image variations because of the big interclass variance, the variety of ambient light
conditions, and the complex structure of the background. The standard approach to solving this problem is
to use different models for each pose of the head [
Many different descriptors were proposed to describe facial features. The most famous ones are
rectangular Haar-like features [
Multidimensional SURF descriptors [
Usually, boosting cascades are trained using grayscale images for the solution of face detection
problem. M. Mathias et al. [
Recently, H. Li et al. [
It will be shown further that, with a view to solving the frontal faces detection problem, our CNN cascade may surpass boosting cascades not only in performance but also in speed. The CNN densely extracts high-level features directly from raw data, without requiring any preliminary processing apart from building the image pyramid. Besides, the CNN calculation algorithm can be easily vectorized using SIMD instructions of CPU, and it can be adapted perfectly to the massively parallel architecture of GPU.
Similarly to [
1) A small number of cascade stages. Our CNN cascade has only 3 stages. For example, the
shortest boosting cascade consists of 4 stages and uses MCT descriptors for the extraction of facial
features [
3) Asynchronous execution of the stages. A particular design of the detector makes it possible to execute the second and the third stages of the cascade in parallel with the first one on different processing units. Due to the fact that 99.99% of sliding window positions are rejected already at the first operation stage, in this mode the detector is capable of processing video frames in constant time, regardless of the content of the image.
4) Optimization. During the implementation of the detector, three technologies were used: SIMD extension of CPU, CUDA, and OpenCL. The SIMD (CPU) and CUDA implementations of the CNN were optimized for each of the network architectures used. The giving up on the traditional approach of the CNN calculation through the organization of a stack of layers, combined with optimization at the assembler level, made it possible to achieve the code execution performance which was close to the peak performance of hardware.
The CNN architectures composing our cascade are shown in Figure 1. Each CNN solves the
problem of a background/face binary classification and contains 797 (CNN1), 1,819 (CNN2) and 2,923
(CNN3) parameters. Similarly to the Convolutional Face Finder [
3 x , tanh y sgn y 1
1 1 y y2 1.41645 y4 .
(1)
The relative error of the following approximation does not exceed 1.8% on the entire number axis,
and only 11 instructions are required to calculate it. Popular ReLU functions turned out to be less
efficient in our experiments. It should be noted that CNN1 contains the smallest number of filters in
comparison with previously proposed network architectures for face detection [
CNN3 (stage 3) input 27×31 conv 4×4 max 2×2 conv 3×3 max 2×2 conv 5×6 conv 1×1 conv 1×1 output 1×1 input 36×40 conv 4×4 average 2×2 conv 3×3 average 2×2 conv 7×8 conv 1×1 conv 1×1 output 1×1
For the CNN training, aligned face images were taken from YouTube Faces Database [
The experiments were carried out with simple models which have a small number of parameters.
We aimed to find the minimal configuration of a CNN which would be able to classify the test set with
an error below 0.5%. For the CNN training, a Levenberg-Marquardt algorithm was used [
The detector design is shown in Figure 3. The first CNN densely scans, in series, each image of
the pyramid. The responses in the output network layer correspond to the positions of the scanning
window with a size of 27×31 pixels during its uniform motion with a 4-pixel step. The coordinates of
the windows where the CNN response exceeded the predetermined threshold T1 are transmitted to the
selective unit for a further analysis of these regions of the image. Even when T1 = 0, more than 99.99%
of the total number of positions of the window at all pyramid levels are rejected at this stage. For
comparison, the first-stage of a Haar cascade of Viola-Jones [
The pre-processing and classification of an image region are carried out in the selective unit, after which the final decision about their belonging to a faces class is made. At the step of pre-processing, the analyzed region is read from the original grayscale image together with a certain neighborhood and is scaled to 51×55 pixels. Then, the equalization of its histogram and mirror reflection with respect to the vertical axis are carried out. Illumination alignment enhances the response of the CNN on shaded faces and effectively suppresses false detections. The use of mirror reflection also reduces the response to complex non-facial images.
During the second step, the second and third stages of the cascade perform region classification. The output of each CNN is a response map of a 5×5 size. Additional classification in the region neighborhood is necessary to prevent the loss of response due to an incorrect positioning of the face in the scanning window. The decision about the type of the region is made on the basis of the number of responses Knn of each classifier that exceeds the predetermined threshold T2: KnCnNN 2 Tnn KnCnNN 3 0 KnCnNN 2 0 KnCnNN 3 Tnn ,
1, face decision rule 0, no face .
(2)
The discrete parameter Tnn makes it possible to control the number of detector false alarms more robustly as compared to thresholds T1 and T2. If the response of the CNN2 does not agree with the decision rule, further analysis of the region stops.
The last stage of the pipeline detector is the Non-Maximum Suppression (NMS) algorithm which aggregates the detected regions to form the resulting areas of faces localization.
grayscale image image pyramid
CNN1 selective unit
NMS selective unit
resize region equalize CNN2
mirror reflection
CNN3 decision rule
The detector has been implemented using three technologies: SIMD extension of x86 processor family (for each of three instructions sets – SSE, AVX, AVX2 – supported by microarchitectures of Intel Sandy/Ivy Bridge and Haswell/Broadwell processors), Nvidia CUDA, and OpenCL. The calculations are carried out using single precision, and the precision-recall characteristic is identical for all implementations.
Thus, the implementations of all stages of the pipeline detector (except for NMS) are presented in several versions of a manually vectorized code. We used vector intrinsic functions (which are directly translated into the appropriate assembly instructions by the compiler), took into account the limited number of logical registers, and minimized the number of queries to the memory. At the same time, the SSE code can be ported to the ARM platform since all the SSE instructions used have analogs in the NEON set of instructions. The Scalar C++ code and OpenCL allow running the detector on most devices though at a lower execution speed.
Figure 4 shows the comparison of our convolution implementation to its implementation in the
Intel IPP, Nvidia NPP and cuDNN, and ArrayFire libraries. Time measurements were made for the first
CNN1 convolution layer calculation (Fig. 1) on an image with a 4K Ultra HD resolution and were
averaged over 103 launches. When implementing the convolution for GPU, we used the method
proposed by F.N. Iandola et al. [
ms 120 90 60 30 0 78 67 74 47 68
60 15
114 9 11 3.4.1 Asynchronous mode
Typically, face images occupy only a small area of the image. The main advantage of cascade detectors is the ability to rapidly reject the majority of background regions as early as during the first stages. However, complex non-facial images may be rejected only during the later stages. If there is a large number of stages, the speed of detector becomes strongly dependent on the structure of the background and the number of the objects in the scene. At the same time, the non-uniform distribution of the processing load over the image area negatively affects the runtime efficiency of GPU implementations.
In order to solve this problem, we proposed the asynchronous mode of cascade execution. In this mode, the first stage which is run on GPU scans each level of the image pyramid sequentially. Coordinates of detected regions are transmitted to CPU which operates as a selective unit. CNN1 proceeds to the scanning of the next level of the pyramid regardless of whether the analysis of the detected regions on CPU has been finished or not.
Due to the low-level optimization, the selective unit carries out a single region analysis in 0.1 ms on a single core processor (CPU PC2, for AVX code). Since only 0.01% (on average) of all window positions passes through CNN1, by the time the scanning of the last level of the image pyramid is complete, the majority of the detected regions will have already been processed on CPU. A similar situation is possible under the condition of asynchronous execution of the cascade on different CPU cores. Thus, the CNN cascade is able to provide a constant frame processing time dependent only on frame resolution and the productivity of the first stage execution. 3.4.2 Hybrid mode
The effectiveness of GPU in image processing is significantly higher than that of CPU. However,
if small images (less than 0.01 megapixels) are considered, the calculation time is limited by delays in
the running of kernels. In order to improve the search speed of large size faces, we made it possible to
run the first stage of the cascade on CPU and GPU simultaneously. In the hybrid mode, CPU begins
scanning the image pyramid from the upper level, while GPU processes a high-resolution image on the
lower level. Also, it is possible to use the asynchronous mode. A similar technique was used in [
In order to reduce the kernel launch delays to a minimum, it is possible to use a patchwork
technique [
In this section, the results of the proposed detector testing on two public face detection
benchmarks – FDDB [
In addition, we tested several face detectors whose source code or demo versions are in open access. In order to compare the performance and speed of algorithms for video processing tasks, tests were carried out on the annotated video.
This section also provides execution speed of all detector implementations for different video stream resolutions and asynchronous mode demonstration. The test results show that the CNN cascade provides a very high data processing speed on both GPU and CPU, outperforming all previously proposed algorithms.
Specification of the equipment used: PC1 (laptop): Intel Core i7-3610QM CPU (2.3 GHz, Turbo Boost disabled), Intel HD Graphics 4000 GPU1 (GT2, 16 core, 1,100 MHz) and Nvidia GeForce GT 640M GPU2 (GK107, 384 core, 709 MHz);
PC2 (desktop): Intel Core i5-3470 CPU (3.2 GHz, 3.6 GHz with Turbo Boost) and Nvidia GeForce GTX 960 GPU (GM206, 1,024 core, 1,228 MHz).
The FDDB [
The result of the offered detector evaluation is shown in Figure 5. The following search settings were used: the minimum object size (minSize) was 15×15 pixels, the scale factor for the image pyramid construction (scaleFactor) was 1.05, T1 = 0, T2 = 0, and Tnn = 1. Since the detector localizes rectangular areas, in some cases this leads to errors when they are matched with elliptical shape annotations. For a correct evaluation, we manually modified 105 received bounding boxes (Fig. 6) so that their IoU ratio would exceed a predetermined threshold.
Based on the adjusted evaluation, the performance of our detector exceeds the performance of
SURF [
Discrete score
Continuous score Ours Ours (corrected) Faceness-Net CascadeCNN HeadHunter SURF−multiview SURF−frontal PEP−Adapt Pico Jain et al.
Viola Jones 1200 1
rejected windows, % 99.995 57.019 24.036
The Annotated Faces in the Wild (AFW) [
This benchmark was developed relatively recently and is mainly used for multi-view detectors
evaluation. Because of this, we additionally tested 7 frontal face detectors, including: two Haar
cascades (OpenCV-default, OpenCV-alt) and an LBP cascade (OpenCV-lbp) from the OpenCV library; a
Haar cascade [
For each detector, precision and recall scores were calculated for different values of minNeighbors = {1, 2, 3} (a parameter specifying how many neighbors each candidate rectangle should have to retain it), and the mean value of F1 measure. MinNeighbors parameter is used in the OpenCV and SURF detectors and is equivalent to Tnn (2) for our detector. In Pico implementation, the level of false alarms is regulated by the threshold of the decision rule which in this test was assumed to be equal to minNeighbors + 2.
Comparison of the detectors was carried out with the following search settings: minSize = 80×80 pixels and scaleFactor = 1.1. The SURF and OpenCV-Köstinger detectors localize a smaller face area in comparison with other detectors, which is why their minSize value was reduced by 25%. The configuration of the detectors was as follows: OpenCV: version 3.0.0, useOptimized = 1; SURF: modelType = 0, fast = 1, step = 1; Pico: strideFactor = 0.1.
A standard IoU ratio with a threshold of 0.5 was used to evaluate the detections. Moreover, 44 bounding boxes were additionally generated for each annotation by scaling it with a factor from 0.9 to 1.2 (Fig. 7). Such a multiple check of detections made it possible to take into account differences in the size of areas localized by detectors and to eliminate errors appearing during matching.
The test results are presented in Figure 8. The proposed detector (F1 = 0.75) holds the second
position in terms of F1 measure value. It is second to the OpenCV-Köstinger detector (F1 = 0.78) due to
producing more false alarms. However, on this benchmark, modern multi-view detectors show
significantly better results, but they cannot work in real-time mode. For example, the Faceness-Net [
Ours (0.75) Pico (0.67) SURF−frontal (0.64) OpenCV−Pham (0.71) OpenCV−Koestinger (0.78) OpenCV−default (0.46) OpenCV−alt (0.72) OpenCV−lbp (0.53) Faceness−Net 0.2 0.4
Recall 0.6 0.8 1
During the last test, the performance of detectors was evaluated on a high-resolution video sequence. We annotated the first 12,000 frames of the 12th episode from the 5th season of «How I Met Your Mother» TV series in HD resolution. This video segment comprises 10 different scenes, and the number of faces in a frame varies from 1 to 88 (39,976 faces in total).
For testing, we used the same detector parameters as for the AFW benchmark, except the minSize parameter which was equal to 40×40 pixels. Also, in this test the parameter T2 for our detector was equal to 1.7. All detectors were running on CPU PC1 in a single-threaded mode.
The proposed detector also ranks second in terms of F1 measure value (F1 = 0.61, 10.6 fps), yielding to OpenCV-Köstinger (F1 = 0.65, 1.7 fps), but it is superior to all the detectors in terms of speed (Fig. 10). Thus, the CNN cascade provides the best performance/speed ratio in comparison with boosting cascades.
A greater level of recall, with a significantly reduced precision, can be achieved by using a weaker decision rule in the selective unit: KnCnNN 2 Tnn KnCnNN3 Tnn (3)
A higher level of precision can be achieved by performing an additional validation of detections with a Haar cascade OpenCV-alt. A Haar cascade was used only after the NMS algorithm, and the detections were pre-scaled to the size of 80×80 pixels. This made it possible to keep the high speed of video processing.
The results of the evaluations of detectors with the use of the AFW and video data allow drawing the following conclusions. Frontal detectors whose processing speed (on CPU) is comparable to that of a compact CNN cascade achieve a significantly lower recall and precision of detection. Detectors whose recall is comparable to that of a CNN cascade have a processing speed which is several times lower. Thus, we have shown that CNNs (even classic ones) allow building a frontal faces detector with better characteristics than it can be achieved using the modern modifications of Viola-Jones algorithm. ion0.6 s i c re0.5 P 0.4 0.3 0.20
Ours (0.61), 10.6 fps Ours−weak (0.59), 9.3 fps Ours−weak+OpenCV−alt (0.63), 9.1 fps Pico (0.59), 9.2 fps SURF−frontal (0.6), 8 fps OpenCV−Pham (0.59), 4 fps OpenCV−Koestinger (0.65), 1.7 fps OpenCV−default (0.43), 1.2 fps OpenCV−alt (0.61), 1.1 fps OpenCV−lbp (0.47), 4.1 fps 0.1 0.2 0.3 0.4
Recall 0.6
The runtime efficiency was the key priority at all stages of the proposed detector development. Our CNN cascade turned out to be from 2.6 to 10 times faster than Haar and LBP cascades from the speed-optimized OpenCV 3.0 library when processing HD video on CPU PC1 (Fig. 10). Meanwhile, the computing complexity of our CNN cascade is comparable to that of Haar-like cascades. In case with the search parameters from Section 4.3 C++ code (without auto-vectorization), it reaches 1.2 fps, whereas the Haar-like classifier from the OpenCV-alt reaches only 1.1 fps.
Among other open source projects (for example, http://libccv.org and http://dlib.net), we do not
know a CPU-based algorithm faster than Pico [
Figure 11 shows the dependence of the speed of various detectors on the content of the scene for the first 4,000 frames of the annotated video. When using the asynchronous mode, the CNN cascade provides nearly constant processing time on both CPU and GPU even when there is a significant increase in the number of faces in the scene (88 faces in frames from 1,771 to 1,834). This property is important for video analytics systems as it makes it possible to predict more precisely the speed of the system in different use cases.
Processing speed
OpenCV−lbp (CPU) Pico (CPU) SURF−frontal (CPU) Ours (CPU) Ours (CPU, async mode ) Ours (GPU2) Ours (GPU2, async mode )
Ours (GPU2+CPU) 2000 frames 500 1000 1500 2500 3000 3500 4000
For algorithms that have been tested on the FDDB benchmark, the authors reported the following
data on the detection speed of frontal faces for images with VGA resolution: NPD1 [
The CNN cascade proposed by H. Li et al. [
Many investigations are focused on the optimization of Viola-Jones algorithm for GPU with the
purpose of increasing the processing speed of a Full HD video stream: D. Oro et al. [
Figure 12 shows the diagram of an average video frame rate in 4 standard resolutions which can be reached with different detector implementations optimized for CPU and GPU. Testing was conducted on first 4,000 frames of the annotated video that was scaled to the size of the appropriate resolution. The results indicate that the CNN cascade copes even with the extreme task of real-time processing of the video stream with a 4K Ultra HD resolution. For example, the speed of an LBP cascade face detector from the OpenCV library reaches 3 fps only on GPU2 PC1 with the same search settings. 400 350 300 250 PS200 F 150 100 50 0 9 5 3 5 2 HD
7 92 20 1 6 3 1 1 1 1
8 31 14 1
0 5 9 7 17 26 3
1 Full HD
In this paper, we proposed a cascade of compact CNNs for a rapid detection of frontal faces in an HD video stream. The first stage of the cascade is capable of rejecting 99.99% of windows containing background. In combination with the asynchronous execution mode, this factor substantially reduces the dependence of the detector speed on image content.
The CNN cascade performance is comparable with that of the best frontal face detectors on the FDDB benchmark, but it surpasses them in speed on both CPU and GPU. Thus, the proposed algorithm establishes a new level of performance/speed ratio for the frontal face detection problem and makes it possible to reach acceptable processing speed even on low-power computing devices.