A method for detecting a face and recognizing a person or a group of persons at pictures or videos that can be corrupted by non-gaussian noise using different architectures of convolutional neural networks is proposed. The technical details of building a deep learning-based face recognition system are discussed. The test results confirm the prospects of using the developed method for solving tasks of face detection and recognition.
Visual pattern recognition is one of the most important components of information management and processing systems, automated systems and decision-making systems. Tasks related to the classification and identification of objects, phenomena and signals, characterized by a finite set of certain properties and characteristics arise in areas such as robotics, information retrieval, monitoring and analysis of visual data, and are the subject of artificial intelligence. The main task of artificial intelligence is to build intelligent information systems that would have a level of effectiveness in solving informal problems that can be compared with human capabilities or those that exceed them.
When interacting with other people, a person's face is an important source of information. Facial expressions, gestures when talking, head movements are a convenient and natural way to convey information. The inability of the computer on the one hand to perceive, and on the other hand to reproduce the natural human ways of communication complicates the transmission and perception of information when working with a computer. To achieve the goal of computer recognition of head movements, facial expressions, it is necessary to implement stable algorithms for analysis and classification of digital images of human faces keeping in mind that they can be corrupted by noisy pixels.
Such algorithms can perform a wide range of commercial and non-commercial tasks. To date, there is no algorithm that could identify a human face with 100% accuracy. However, it is already possible to implement programs for detecting faces, which can be successfully used in education, medicine, security and safety, computer games, advertising and other areas.
The purpose of this work is the development and testing of a method for detecting faces and recognizing a person or a group of persons at pictures and videos that can be corrupted by nongaussian noise based on the use of convolutional neural networks.
It is generally assumed that full face recognition consists of four main steps: 1. Face detection 2. Preliminary processing 3. Isolation of signs 4. Classification
The first step is to detect faces in the image, regardless of scale and location. To do this, an
advanced filtering procedure is often used to distinguish locations representing faces and filter them
using some classifiers. It is noteworthy that all changes in displacement, scaling and rotation must be
processed during the face detection phase. Note that facial expressions and changing hairstyles or
smiling and frowning faces still present difficulties at the stage of pattern recognition [
While the accuracy and speed of face detection systems has improved, the two biggest challenges
remain to some extent. Face detectors are essential to deal with large and complex variations of facial
changes and effectively distinguish between faces and non-faces in unrestricted environments. In
addition, large differences in face position and size in a large search space create problems that reduce
detection efficiency [
In the next step, a system based on an anthropometric dataset predicts the approximate location of key features such as eyes, nose and mouth. The entire procedure is repeated to predict small features in relation to major features and is checked with collocation statistics to reject any misplaced features.
Highlighted anchor points are generated as a result of geometric combinations on the face image, and then the actual recognition process begins. It does this by finding a local representation of the facial appearance at each of the anchor points. The presentation scheme depends on the used approach.
Feature extraction usually occurs immediately after face detection and can be considered one of
the most important steps in face recognition systems, since their effectiveness depends on the quality
of the extracted features. This is because the facial landmarks are identified by a given network that
determines how accurately features are represented. Traditional landmark locators are model-based,
while many recent methods are based on cascade regression [
After feature extraction, face recognition is performed. The next stage of classification is simply performed using the general nearest neighbor technique, while more specific algorithms are developed for the preprocessing stage. This step usually consists of cropping, zooming, and aligning the detected faces. Unless sophisticated techniques are required for cropping and zooming, alignment is not an easy problem to solve. In practice, alignment consists in detecting a series of landmarks on the face (nose, eyes, etc.) and then transforming the face pictures so that the position of these landmarks is constant.
Significant progress has been made in face detection and recognition through the use of convolutional neural networks (CNN).
The first CNN framework, known as LeNet-5, was developed in 1990 to classify handwritten
digits by recognizing visual patterns in image pixels without the need for preprocessing [
Because face recognition differs from object recognition in that it requires alignment before extraction, this is reflected in the differences between CNNs designed for face recognition and those used for object recognition.
Because of the proven ability of deep neural network (DCNN)-based systems to outperform human performance in face-checking tasks, research in this area has skyrocketed.
Face detection and recognition research is currently focusing on DCNNs, which have
demonstrated impressive accuracy on highly complex databases such as the WIDER FACE dataset
[
The growth in deep neural network research has been accompanied by the emergence of many deep learning frameworks and the development of the Caffe, TensorFlow, Torch, MXNET, and Theano frameworks that use platforms such as CUDA and libraries such as cuDNN. Finally, these frameworks can be used with a number of programming languages such as C ++, Python, or Matlab.
Rapid progress in this direction has been driven by the increase in the availability of powerful
GPUs and improvements in CNN's architecture for real-world applications. In addition, the
development of large annotated datasets and a better understanding of nonlinear mapping between
input images and class labels have contributed to increased research interest in these networks.
DCNNs are very efficient due to their ability to approximate non-linear functions. Their significant
disadvantage is high computational costs due to the presence of intensive convolution and nonlinear
operations [
Using CNN in face recognition tasks consists of two main stages: training and inference. Training
is a global optimization process that involves identifying network parameters by observing huge
datasets. Inference essentially involves deploying a trained CNN to classify observed data [
CNN face recognition systems are characterized by the training data used to build the model, the
network architecture and settings, and the type of the loss function [
Modern face recognition systems that use DCNN implement deep feature extraction and similarity comparison, which involves conversion of test images to deep representations and computation of Euclidean or cosine distance.
Initially, the structure of the convolutional neural network was created taking into account the structural features of some parts of the human brain that are responsible for vision. The development of such networks is based on three mechanisms: - local perception; - formation of layers in the form of a set of feature maps (shared weights); - subsampling.
According to these mechanisms to build a convolutional neural network three main layers are applied: convolution, pooling (or subsampling), fully connected layer. 4.1.
The convolution formula for the l -th (l 1,...,L) layer of the network, which looks like [
As can be seen from (1), convolution operations are performed for each element i, j of image matrices xl .
Convolution preserves spatial relationships between pixels.
Each convolutional layer is followed by a downsampling (subsampling) layer, or a computational layer, which serves to reduce the image dimension by local averaging the neuron output values. 4.2.
The subsampling layer reduces the scale of the planes by locally averaging the neuron outputs. Thus, a hierarchical organization is achieved. Subsequent layers extract more general characteristics less dependent on image distortion.
The pooling layer is described by the expression
xilj l downwalb y(lis1a)( jsb) bl , (2)
a b where down() – the pooling function. This function adds blocks of the input image, thus reducing the dimension of the output image of the layer. In this case, any output map is set by two offset parameters: multiplicative l and additive bl .
The difference between the subsampling layer and the convolution layer is that the regions of neighboring neurons overlap, which does not happen in the subsampling layer.
The pooling layer operates independently of the slice depth of the input data and scales the volume spatially using a maximum function.
In the architecture of the convolutional network, it is generally accepted that the presence of a feature is more important than information about its location. Therefore, from several neighboring neurons in the feature map, the maximum is selected and its value is considered one neuron in the feature map of a lower dimension.
In addition to the maximum subsampling, pooling layers can perform other functions, such as an averaging subsampling or even an L2-normalized subsampling. 4.3.
On these layers within the network, a non-linear activation function is applied to all input values f () and the result is sent to the output. Thus, the activation layer does not change the size of the entrance.
Usually, due to significant positive properties, ReLu is used for hidden layers. (ReLU(x) = max(0, x)) and its various modifications (Leaky ReLU, Parametric ReLU, Randomized ReLU), and for fully L N L 1 connected layer – SoftMax function (when solving classification problems) f jL ex j exiL or i1 linear (for regression problems). 4.4.
Various regularization methods are used to avoid overfitting the network.
Dropout is a simple and effective method of regularization and consists in the fact that in the process of training a network from its aggregate topology, a subnet is repeatedly randomly allocated, i.e. some neurons are turned off from the process and the next update of the weights occurs only within the dedicated subnet. Thus, weights change only in the remaining neurons. Each neuron is dropped from the aggregate network with a certain probability, which is called the dropout ratio.
This layer reduces the time of one training epoch due to the smaller number of parameters to be optimized, and also makes it possible to better deal with network overfitting in comparison with standard regularization methods. 4.5.
On this layer, the standard normalization of the inputs occurs (the sample mean of their values is subtracted, and the result is divided by the root of the sample variance). Sample values are calculated taking into account the values at the inputs of this layer at previous training iterations. This approach allows you to increase the speed of network learning and improve the final result. 4.6.
This layer is a conventional multilayer perceptron, the purpose of which is classification. It simulates a complex nonlinear function, optimization of which improves the recognition quality.
The neurons of each map of the previous subsample layer are associated with one neuron of the hidden layer. Thus, the number of neurons in the hidden layer is equal to the number of maps in the subsample layer.
As in conventional neural networks, in a fully connected layer, neurons are connected to all activations in the previous layer. Their activations can be calculated by multiplying matrices and applying an offset.
The difference between fully connected and convolutional layers is that the neurons of the convolutional layer 1) are connected only to the local input area; 2) can share parameters. 4.7.
CNN training is an iterative process. Each iteration calculates the network outputs for one (or more) samples in the training set, and adjusts the network weights to reduce the error between the actual network output. ( yiL, p )(i 1,..., N L ) and the target output for a given sample di, p . Therefore, training is reduced to minimizing some functionality.
In practice, a quadratic function, cross entropy, or some combined functional is used as a criterion for the error function.
A backpropagation neural network (BPNN) is a multi-layer feedforward neural network that uses a supervised training algorithm known as an error backpropagation algorithm. Errors accumulated on the output layer are propagated back to the network to correct the weights. A conventional MP, which consists of three types of layers: input, hidden and output has no backward computation other than the operations used in training. All operations are performed in the forward direction during simulation.
CNN training is similar to training of any direct propagation ANN and consists in correcting its weight parameters based on minimizing some selected loss function, which are usually used as a quadratic function or cross entropy.
To train convolutional neural networks, both the standard backpropagation method and its various
modifications can be used. The derivation of the backpropagation algorithm for CNN training is
discussed in sufficient detail in [
In practice, the most common neural network learning algorithm is usually used, based on the
gradient descent method (the backpropagation error method) and its modifications: SGD, Adam,
AdaGrad, AdaDelta, etc. [
To train the neurons of this layer, a gradient backpropagation algorithm is used
(k 1) (k) J ( (k)), where – network parameters (elements of weight matrices, displacements, angles of inclination of activation functions, etc.) ; J ( (k )) – objective function (loss function E or C); – the parameter of the learning rate. 6.1.
If the quadratic function E is chosen as the network loss function, and the sigmoidal function is chosen as the activation function of neurons, then to adjust the weights matrix of the L -th layer following equations are used (3) (4) (5) (6) where wkLi J (wkLi ) E wkLi yEiL yxiiLL wxikLLi iL wxikLLi iL ykL1, i (0,..., N L ),k (0,..., N L1), iL E yiL E yiL ; yiL xiL yiL di ; yiL yiL (1 yiL ); xiL di – required value of the i -th output. yiL yiL (1 yiL ) , and besides, yiL yiL y Lj , because E xiL x Lj xiL j1 yELj xyiLLj , i (1,..., N L ). If, however, as the activation function of neurons SoftMax is selected, then in (4) one should take N L Similarly, one can get the procedure for adjusting the offsets for neurons of a fully connected layer E biL
yiL xiL biL E yiL xiL iL xiL iL. biL When training neurons of other (hidden, l -x, l 1,..., L 1) layers it is needed to calculate gradients E N L1 N L1 ykl i1 il yxklil1 i1 il1 wkli1, (7) i (1,..., N L )k (1,..., N L1).
Here wilj1 – the weight of the connection between the neuron i of the current (hidden) layer and neuron j of the next layer. When choosing the cross entropy as the loss function, the training procedure for the output (fully connected) layer of the CNN will take the form
(k 1) (k) C( (k)), (8) where (w, b) .
Calculating the partial derivatives with respect to the tunable parameters, we have (here it is taken into account that for the output layer yiL fiL ) wC C wiLk
yCiL wyiiLLk fkL1( fkL1 di ); bC C biL C yiL fiL di ;
yiL biL f C fCjL fjL in1 di ln fiL dfjLj fiL di. When calculating the gradient, the following derivatives are used: 2 ex Lj ex Lj fiL kn1exkL kn1exkL i j; fiL (1L fLiL ),i j; x Lj enxiL ex Lj 2 i j; fi f j exkL
k1 C xiL n d j ln fiL n C f jL yiL diL.
xiL j1 j1 f jL xiL 6.3.
Training neurons in the subsampling layer (9) (10) (11) (12) (13) (14) The backpropagation algorithm is also used to train the neurons of this layer.
Core is a filter that slides over the entire image and finds its features anywhere, i.e. provides invariance to displacements.
Formula for updating the convolution kernel walb has the form [
i j yilj xilj Only one offset can be used for one feature map bl , which is “connected” with all the elements of The peculiarity of this layer is that it sits in front of both fully connected and convolutional layers. In the first case, it has the same neurons and connections as a fully connected one. Therefore is calculated in the same way as in a hidden layer. If the subsampling layer is in front of convolutional layer, is computed by the inverse convolution method using the rotation by 1800 rot180o ilj [16– 6.4.
Training of neurons of the convolutional layer (15) E yilj1 this map. Accordingly, when correcting the value of this offset, all values from the map obtained during the back propagation of the error should be taken into account. In this case (when using one displacement for one feature map) we have E bl i j yilj xilj bl E yilj xilj i j yilj xilj E yilj .
As an alternative, one can take as many offsets bilj for a separate feature map, as many elements parameters of the convolution kernels themselves). For this case
To train neurons in the convolution layer, the gradient procedure uses the derivative calculated as follows: E yilj1 i j yilj xilj yilj1 E yilj xilj E yilj w(isi)( js j).
l i j yilj xilj (16)
Histogram of Oriented Gradients (HOG) – is a method of information evaluation of special points
of the image, based on the calculation of the number of directional gradients in the space of these
points. The basic idea of the algorithm is the assumption that the appearance and shape of the object
in the image area can be described by the distribution of intensity gradients or the direction of the
edges [
HOG algorithm contains following stages: 1. Gradient calculation. 2. Calculation of the histograms of the image cells. 3. Forming and normalizing descriptor blocks. 4. The final step is to classify the HOG descriptors using a training system with a teacher.
The result of the classifier's work is two object images with positive and negative weights of reference vectors. Positive weights mean that the features belong to the target class, and negative weights mean that the features belong to the background.
The SSD method is based on such CNN architectures as Faster R-CNN and YOLO, however, the authors of [20] took into account their shortcomings, thanks to which SSD could achieve new peaks of accuracy and speed. The method is based on the Feed-forward convolutional network, which creates a finite set of limiting rectangular frames and quantitative estimates of the presence of objects of various classes in these frames, after which it produces the suppression of maximums ("Nonmaximum suppression predictors"). The structure of the network is shown in Fig. 1 and can be divided into four functional parts: 1. Network input. It accepts a three-channel (color) image with a size of 300x300.
2. Backbone network. Some standard architecture for image classification is usually used as a backbone network. In this case it is VGG-16. However, the fully connected layer of VGG-16 is not used in SSD. This architecture is described in detail in [20].
3. Layer of additional features. These are convolutional layers and subsampling layers that generate feature maps of different resolution for detecting objects of different scales.
4. At this stage operations are performed to merge the outputs of various layers and suppress maximums to obtain the output of the network. The SSD method has the following key features: 1. Detection of objects of various scales. Several convolutional layers of different sizes are sequentially added to the base network, which make it possible to obtain maps of signs of different resolutions and, accordingly, to predict objects at different scales (unlike, for example, YOLO, which receives only one map).
2. Convolutional predictors for detection. Each feature layer generates a finite set of assumptions about the class and position of the object using a set of convolutional filters. Each such convolutional filter has a kernel that determines the probability of belonging to the default frame offset class itself.
3. Default bounding frames. The default set of frames is linked to the cards of signs of different resolutions described in point 1: each cell of the card of signs corresponds to a frame from the set by default. For each cell of the feature map, estimates are determined according to the class of objects, as well as four offsets of the default frame relative to its initial position.
Combining these properties allowed efficiently sampling many different forms of the resulting bounding frames, which had a positive impact on the speed of the network.
Of the neural network approaches in face detection, Multi-task Cascaded CNN (MTCNN) is
especially significant. This network is described in details in [
RetinaFace Network
The architecture of the RetinaFace network [
- RetinaFace has a self-checking grid decoder branch to predict per-pixel information about 3D face shapes in parallel with existing controlled branches;
- By utilizing lightweight backbones, RetinaFace delivers performance that allows it to be run on the CPU, while most similar systems achieve such performance solely using the GPU. Table 1 System performance results
Method HOG fr128d 86.67% arc512d 87%
Acknowledgements
The European Commission's support for the production of this publication does not constitute an endorsement of the contents, which reflect the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.