The paper considers the task of obtaining a quality assessment of facial images for usage in various video surveillance systems, video analytics and biometric identification. Accuracy of person recognition and classification depends on the quality of the input images. We consider an approach to obtaining single face image quality assessment using neural network model, which is trained on pairs of images that are split into two possible classes: the quality of the first image is better or worse than the quality of the second one. Two modifications of the selected baseline algorithm are proposed. A face recognition system is applied to change the loss function and image and face quality attributes are used when training the model. Experimental studies of the proposed modifications show their effectiveness. The accuracy of selecting the best and worst frame is increased by 1.3% and 1.9%, respectively.
Computer vision algorithms such as face recognition, algorithms for determining
emotions, demographic characteristics and key points of a human face, are widely used in
video surveillance systems, video analytics and biometric identification. The received
data in these systems is a video stream, which contains a set of several frames for each
person. But most algorithms are built so that they process frames independently of each
other and, as has been shown in many studies, their accuracy depends on the quality of
the input images [
The task of face quality assessment is to obtain one scalar value for the input image
that reflects the overall quality and takes into account both image quality attributes
(illumination, blur, noise, etc.) and face quality attributes (head pose, face occlusion, etc.).
Usually, this scalar value is enclosed in the range from 0 to 100, where the values 0 and
100 correspond to the image with the worst and best quality, respectively. Algorithms
for obtaining this value are trained either on pairs of images split into two possible
classes (the quality of the first image is better or worse than the quality of the second
one), as in [
Many of the recent works consider the problem of face quality assessment from a
different point of view: as an indicator that reflects the usefulness of the image for the
specific algorithm being used. Algorithms for obtaining this value use one of the
existing face recognition systems, based on which either the training and test dataset is
marked up [
In this paper, we use the approach from article [
Improving the Neural Network Algorithm for Assessing the Quality of Facial Images 3 2
In the article [
HingeLoss = max(0, Q2 − Q1 + 1). (1) The minimum of this function is achieved when the quality score of the second image in the pair is less than the quality score of the first image in more than 1. Proposed approach allows to obtain a ranking model by training on pairs of images without manual markup. At the second stage, only one branch of the siamese network is used to evaluate the quality of a single face image. This branch is also fine-tuned on a separate dataset using regression.
Our algorithm trains in the same way on pairs of images, but we use pairs of different
images containing people's faces. For each pair, the markup into two possible classes
was obtained using experts: the quality of the first image is better or worse than the
quality of the second one. The best image in a pair was considered to be the one that
best matches the combination of the following properties: front projection, normal
illumination, absence of occlusion, noise and blur, etc. This approach for obtaining pairs
was chosen because, from our point of view, it allows us to take into account more
complex cases that occur between pairs of different images, which cannot be obtained
by applying distortion to the original image. Different strategies for selecting pairs for
markup were used to cover more cases, and pairs for which it is not possible to uniquely
define a class were not used later. We don't apply fine-tuning on a separate dataset.
Resnet-10 [
As the first modification of the baseline algorithm, a new loss function is proposed, which uses the output of the face recognition system. In the baseline algorithm loss function for quality assessment has the following form (2):
LFQA = max(0, Q2 − Q1 + margin), (2) where margin is a constant value equal to 1. For a part of the training dataset that consists of pairs of images of the same person, we calculate a set of probabilities using the face recognition system. A probability is in the range from 0 to 1, where the value 1 means that a pair of images belongs to the same person and the value 0 means that they belong to different people. We use this probability as an indicator of the similarity of two images in terms of quality. This set is then normalized so that the expected value would be 0 and the standard deviation would be 1. In the new loss function, margin has the following form (3):
margin= { max(α, 1 − β ∗ FR), a pair of images of a single person ,
(3)
where α ∈ (0, 1) – new minimum value of the margin; β ∈ ℝ+ – custom parameter;
FR ∈ ℝ – output of the face recognition system after normalization. In our experiments,
we use the following parameter values: α = 0.4 , β = 2. Face recognition system is
developed by Video Analysis Technologies [
As a second modification of the baseline algorithm, we consider multi-task learning of the neural network to evaluate the face quality and image attributes such as illumination and blur, as well as face attributes such as head pose and face occlusion. The general loss function has the following form (4): Loss = w1 ∗ LFQA + w2 ∗ LIllumination + w3 ∗ LBlur + w4 ∗ LPose + w5 ∗ LOcclusion, (4) where {wi}i5=1 − weight coefficients. In our experiments, we use the following parameter values: w1 = 64, w2 = 48, w3 = 48, w4 = 2, w5 = 4. 4.1
Evaluation of illumination and blur occurs on pairs of images similar to the face quality assessment in the baseline algorithm, with the use of Hinge Loss as LIllumination and
Improving the Neural Network Algorithm for Assessing the Quality of Facial Images 5 LBlur. During training, the value of the loss function for pairs without markup is assumed to be zero. 4.2
The head pose estimation consists of determining three angles for each image in a pair: pitch, roll and yaw. Each angle is enclosed in the range from −90° to +90°. Training is performed using regression and Weighted Mean Absolute Error is used as the loss function (5):
LPose =
α∗∣P−P̅∣+ βα∗+∣Rβ−+R̅γ∣+ γ∗∣Y−Y̅∣ ,
(5)
where ̅P, R̅, ̅Y ∈ ℕ — ground truth labels; P, R, Y ∈ ℕ — neural network output; α,
β, γ ∈ ℝ — weight coefficients for pitch, roll and yaw respectively. In this work, we
use the following parameter values: α = 1, β = 1, γ = 0.05 . The low value of γ is
explained by the fact that the marking for yaw is less accurate than for pitch and roll.
Fig. 2 shows an example of head rotation angles using three guide vectors.
Face occlusion task is to determine one of two types of visibility for 23 areas of the
face: the visible region and the invisible region due to the rotation of the head,
overlapping by an external object, or going beyond the boundaries of the image. The scheme
of dividing the face into regions is based on the approach proposed in [
LOcclusion = ∑i2=31 CE Lossi (6) where CE Lossi is the Cross Entropy Loss for the i-th region. 5
Since the necessary dataset are not publicly available, we created our own training set.
Images for constructing pairs were provided by Video Analysis Technologies [
It should be noted that in the case of illumination, training is carried out on pairs of images, the markup for which is obtained automatically based on their illumination levels, because this approach turned out to be more stable. General characteristics of the obtained dataset, taking into account the transitive closure and the number of pairs used together with the face recognition system, are given in Table 2.
Improving the Neural Network Algorithm for Assessing the Quality of Facial Images 7 Since the most common datasets are designed to evaluate the quality of an arbitrary image that does not necessarily contain a human face, we have created a new dataset for this purpose. The test dataset consists of tracks – sets of 5 to 12 frames containing images of faces belonging to one person. The total number of tracks is 7070, and for each track the markup of the best and worst frames was made in terms of quality assessment. The best frames are those that have the best matches the combination of the following properties: front projection, normal illumination, absence of occlusion, noise and blur, etc. Similarly, the worst frames are those that have the worst correspondence to these properties. Each track was marked by three experts, and the frame was considered the best or worst only if the opinions of at least two experts were the same. It was also required to select as few frames as possible. Fig. 4 shows an example of such a track. For this dataset, we define three metrics: Best Shot Accuracy, Worst Shot Accuracy and Pair Accuracy.
Frame 1
• S = {St}tN=1 – set of N tracks;
• Mt ∈ ℕ, Mt ∈ [
Notation for the algorithm output: • Q = {Qt}tN=1 – collection of N sets with quality scores; • Qt = {QFkt}kM=t1 – set of quality scores for a track with the number t. We define two indicator functions:
ItB = { ItW = { 1, Fmt ∈ Bt , m = argmaxQFkt, 0, Fmt ∉ Bt ∀k∈[1, Mt] 1, Fmt ∈ Wt , m = argminQFkt.
0, Fmt ∉ Wt ∀k∈[1, Mt] The indicator function (7) corresponds to the choice of the best frame – it is equal to 1 if and only if the frame with the highest quality score is contained in the set of the best frames. Similarly, (8) corresponds to the selection of the worst frame. Based on the indicator functions, we define two metrics:
N∑ ItB BestShotAccuracy= t=1 ,
N Improving the Neural Network Algorithm for Assessing the Quality of Facial Images 9 WorstShotAccuracy= t=N∑1ItW .
N (10) Since each track consists of frames of three types (best, worst and normal), we can also create image pairs for each track that consist of two different types of frames. The resulting pairs are marked up into two classes (the quality of the first image is better or worse than the quality of the second one), which is uniquely determined based on the types of images in the pair. Pair Accuracy is defined as the percentage of correctly classified specified pairs, the total number of which is 173 940. 7
On the test datasets four algorithms are compared: Baseline, Baseline with Modified Loss, Baseline with Attributes, Baseline with Modified Loss and Attributes. On Fig. 5 the scheme of the baseline algorithm along with two modifications is given.
During training a polynomial change of the learning rate with the degree of
polynomial 2 is used, and the initial value of the learning rate is 0.001. The total number of
epochs is 25. We also use the Ranger optimizer, which is a combination of two methods
proposed in [
Improving the Neural Network Algorithm for Assessing the Quality of Facial Images 11 8
In this paper the modified neural network algorithm for face quality assessment is
proposed. An approach from article [