=Paper=
{{Paper
|id=Vol-2845/Paper_31.pdf
|storemode=property
|title=Mathematical Methods for Information Technology of Biometric Identification in Conditions of Incomplete Data
|pdfUrl=https://ceur-ws.org/Vol-2845/Paper_31.pdf
|volume=Vol-2845
|authors=Oleksii Bychkov,Oleksii Ivanchenko,Kateryna Merkulova,Yelyzaveta Zhabska
|dblpUrl=https://dblp.org/rec/conf/iti2/BychkovIMZ20
}}
==Mathematical Methods for Information Technology of Biometric Identification in Conditions of Incomplete Data==
https://ceur-ws.org/Vol-2845/Paper_31.pdf
Mathematical Methods for Information Technology of
Biometric Identification in Conditions of Incomplete Data
Oleksii Bychkov, Oleksii Ivanchenko, Kateryna Merkulova and Yelyzaveta Zhabska
Taras Shevchenko National University of Kyiv, Volodymyrs’ka str. 64/13, Kyiv, 01601, Ukraine
Abstract
The purpose of this research is to develop mathematical methods for information technology
of biometric identification which will allow to recognize person’s face in conditions of
incomplete data such as wearing a medical face masks during the pandemic. During the
ongoing pandemic researchers focus on quick and effective solutions to develop technologies
that handle this problem. This research concentrates on the analysis of the already existing
solutions and proposes a mathematical method of face identification for information
technology based on wavelet transform under the condition of wearing masks by people.
During this research, the experiments with face detection and recognition have been
performed with the constraint information of covered face. There is no database of face
images with masks, therefore a new database was created. This database contains 820 images
of 40 people, whose faces was limited only by top part of the face (forehead, eyes). First
experimental part was performed with the use of standard Python library face_recognition,
which allows to perform face recognition from Python or from the command line with one of
the simplest face recognition libraries. It built based on dlib face recognition toolkit with
deep learning – it is a ResNet network that contains 29 conv layers. In the second set of
experiments FaceNet system was used. It is a system that after high-quality features
extracting from the face containing image creates a face embedding and predicts these
features representation in a form of 128-element vector. Third part of experiments was
performed to analyze the efficiency of three well-known face recognition methods:
Eigenfaces, Fisherfaces and LBPH. Eigenfaces algorithm considers that different face parts
are not identically significant in a face recognition process. It processes all the training
images of all the people as a whole and extracts the components which are relevant and
useful. Fisherfaces algorithm extracts principal components that differentiate one person
from the others, so an individual's components become more useful over the others. The of
LBPH algorithm is to find face local structure by comparing each pixel to its neighboring
pixel, forming a list of local binary patterns, that can be converted into a decimal number.
Mathematical methods proposed in this research based on wavelet transform, that is widely
used in the image processing tasks. Wavelet transform provides processing of patterns
hidden in the data performing data analysis in general as well as in the detail. To compare the
results of the commonly used algorithms with wavelet transform there was developed
algorithm with the use of Daubechies wavelets and reverse biorthogonal wavelets. The
results of experimentation analysis indicate that the popular and commonly used methods of
face identification do not demonstrate high efficiency results. Proposed mathematical
methods for information technology based on wavelet transform improves the face
recognition and identification process under the condition of faces covered with mask.
Specifically, the most accurate identification rate of 77,5% was obtained with the use of
Daubechies wavelets.
Keywords
Biometric identification, face recognition, wavelet transform 1
IT&I-2020 Information Technology and Interactions, December 02–03, 2020, KNU Taras Shevchenko, Kyiv, Ukraine
EMAIL: : bos.knu@gmail.com (O. Bychkov); ivanchenko.oleksii@gmail.com (O. Ivanchenko); kate.don11@gmail.com (K. Merkulova),
y.zhabska@gmail.com (Y. Zhabska)
ORCID: 0000-0002-9378-9535 (O. Bychkov); 0000-0002-8526-8211 (O. Ivanchenko); 0000-0001-6347-5191 (K. Merkulova),
0000-0002-9917-3723 (Y. Zhabska)
©️ 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
336
1. Introduction
COVID-19 is an unprecedented crisis that has caused a huge number of casualties and security
concerns. One of the methods which are using to reduce the spread of the coronavirus, is to wear
masks in all public places to protect yourself and the others from being infected. So protective masks
in these conditions have become an attribute of everyday life. It makes face recognition very
challenging because some parts of the face are hidden. But face is the most common pattern that
people use to identify each other in everyday life. Also, governmental identity documents, such as
passport or driver’s license, contains face image. Even in ideal conditions, face recognition
technologies often struggle with accuracy.
Nowadays wearing medical masks caused increasing of the probability of false identification
results of recognition systems, as the US National Institute of Standards and Technology (NIST)
concluded. The National Institute of Standards and Technology conducted the research in which 89
face recognition algorithms with error rate of 0.3% were tested. And researchers found, after applying
of those algorithms on images of persons with face masks, that the error rate increased from 5% to
50% [1].
Mask-wearing is now recommended as a measure to provide the spread of COVID-19. Therefore,
the government, that uses face recognition algorithms to track and identify people across the US,
conducted the research performed by the NIST in collaboration with the US Customs and Border
Protection and Department of Homeland Security, both of which apply face recognition methods in
their work. The research suggests that face mask wearing would decrease a person identification rate
by face recognition. The result of a face recognition depends on mathematical models of the relative
positions of face features. Anything that reduces the visibility of key characteristics of a face (such as
the nose, mouth and chin) obstructs the positive outcome of face recognition. Findings also suggest
that dark masks make it more difficult to recognize a face than blue surgical masks, and that wide
masks that cover a person's entire face prevent the recognition more than round N95-style masks [2].
Notably, the report only lists a type of face recognition known as one-to-one matching. This is a
procedure used at border crossings and at passport control, where an algorithm checks if a person
matches their ID. However, facial recognition systems of mass surveillance scan the crowd to find
matches with faces in the database, which is called a one-to-many system. And while the NIST report
does not cover one-to-many systems, they are generally considered the most error-prone. The process
of identifying a face in a crowd is more difficult, because of the different image scaling, angle of face
position and lightning. Thus, wearing masks is likely to seriously interfere with one-to-many
algorithms as well. Many companies that work on face recognition technologies have claimed they
can identify people with high accuracy rate even in conditions of face masks wearing, but the latest
research results show that the face coverings significantly increase face recognition error rates [3, 4].
During the ongoing pandemic researchers focus on quick and effective solutions to develop
technologies that handle problem of face identification accuracy decreasing. This paper concentrates
on the analysis of the already existing solutions and proposes a method of face identification based on
wavelet transform under the condition of wearing masks by people, considering different image
scaling, angle of face position and lightning. Experiments with face detection and recognition have
been performed during the research with the incomplete data such as covered with mask face. Since
there is no database which contains face images with masks, a new database was created. This
database contains 9 images of 80 people, whose faces was limited only by top part of the face
(forehead, eyes). These images prepared with different image scaling, angle and lightning.
2. Task solution methods
2.1. Face_recognition method
Python face_recognition library allows to perform face recognition and manipulation on images
from Python or from the command line with one of the simplest face recognition libraries, built based
337
on dlib face recognition toolkit with the use of deep learning. Applying of this model on the Labeled
Faces in the Wild database indicated an accuracy rate of 99.38% [5].
Dlib's face recognition library performs conversion of a face image to a 128-dimensional vector
space where images of the same person have a small Euclidean distance between them and images of
different persons are, on the contrary, long-distance by Euclidean distance metric calculation.
Dlib model is based on the ResNet network that contains 29 conv layers. It is a version of the
ResNet-34 network from the research study [6] with an exclusion of a few layers and filters per layer
reduction by half. The network training was performed on a 3 million face images dataset, that
includes several datasets (the VGG dataset [7], face scrub dataset [8]) and a large amount number of
images derived from the internet. The total number of face images in the dataset is 7485 images.
Also, the face_recognition library method is using the k-nearest-neighbors (KNN) algorithm for
face recognition [9]. It is useful when it is needed to recognize a large set of people, whose face
images are stored in the database, and make a prediction for an unknown person in a feasible
computation time. KNN algorithm provided by Python data science library - scikit-learn.
Neighbors-based classification based on prediction of a class labels using majority rule voting for
the nearest neighbors of each point: the data class that has the most representatives within the nearest
neighbors of the point defines as the query point.
Scikit-learn provides two different nearest neighbors classifiers: KNeighborsClassifier and
RadiusNeighborsClassifier. The most used method in KNeighborsClassifier is k-neighbors
classification. Each query point can be obtained with the use of k nearest neighbors learning, where k
is an integer value specified by the user. The determination of k value depends on the data meaning
that a larger k reduces the noise effects but, at the same time, makes the classification boundaries less
distinct.
For the classification based on nearest neighbors there are commonly used uniform weights. But
some conditions require to use the neighbors weighting so that nearer neighbors contribute more to
the fit. For the purpose of this research it is better to use distance weights which are proportional to
the inverse of the distance from the query point.
The KNN classifier is initially trained on a set of face images stored in the database that were
labeled. Then it can identify the person in an input image by finding the k most similar faces by
calculation of the closet face-features under Euclidean distance in its training set, and performing a
majority vote (possibly weighted) on their label. For example, if k=3, and the three closest face
images to the inputed image in the training set are one image of class1 and two images of class2, the
result would be class2.
2.2. FaceNet method
FaceNet is a face recognition system that was presented in the paper [10]. It creates a mapping
from face images to Euclidean space where, on the basis of a deep convolutional network
computation, calculated distances correspond to a measure of face similarity.
System extracts high-quality features from the face image, given as an input, and transform they to
a 128-element vector representation, performing a face embedding. This compact 128-D embedding
uses a LMNN based triplet loss function [11]. Triplet loss function uses to determine similarity of the
vectors, that can be defined by the calculation of the distance between vectors. Vectors for image of
the same person is more similar (distance between those vectors is smaller) and vectors for the images
of different persons is less similar (distance between those vectors is larger). It means that an
embedding f(x), from an image x into a feature space Rd, such that, independently of image quality,
the squared distance between all face images of the same person is small, and the squared distance
between face images of different persons is large.
Triplet loss function can be described as following. Image embedding representation expression is:
𝑓 (𝑥 ) ∈ 𝑅 𝑑 . (1)
It means that an image x is being converted into a d-dimensional Euclidean space. Then it is
necessary to determine that an input image xi a of a person is closer to other images of the same person
xi p than all images of different persons xi n. It can be expressed with the next:
‖𝑓(𝑥𝑖 𝑎 ) − 𝑓(𝑥𝑖 𝜌 )‖22 + 𝛼 < ‖𝑓(𝑥𝑖 𝑎 ) − 𝑓(𝑥𝑖 𝑛 )‖22 , (2)
338
∀(𝑓 (𝑥𝑖 𝑎 ), (𝑓𝑥𝑖 𝜌 ), 𝑓(𝑥𝑖 𝑛 )) ∈ 𝑇, (3)
where α – is a limited space between images of the same person and all images of any other
person; T represents the training set of all possible triplets with N cardinality.
The loss is needed to minimize, that can be described with the following:
𝑁
(4)
∑[‖𝑓(𝑥𝑖 𝑎 ) − 𝑓(𝑥𝑖 𝜌 )‖22 − ‖𝑓(𝑥𝑖 𝑎 ) − 𝑓(𝑥𝑖 𝑛 )‖22 + 𝛼 ].
𝑖
As the basis for classifier training system uses obtained face embeddings.
FaceNet model is provided by the Python deep learning API – Keras, that contains the pretrained
Inception ResNet v1 model. This model has certain restrictions concerning input data. Expected input
images must be colored, with pixel values standardized, and to have a square shape with the size of
160×160 pixels.
Face detection process is based on the use of Multi-Task Cascaded Convolutional Neural Network
(MTCNN) [12]. It performs face finding and extracting from images with solution of 3 tasks: face
classification, bounding box regression and facial landmark localization. For each sample of
classification xi the cross-entropy loss can be calculated:
𝐿𝑑𝑒𝑡
𝑖 = −(𝑦𝑖𝑑𝑒𝑡 log(𝜌𝑖 ) + (1 − 𝑦𝑖𝑑𝑒𝑡 )(1 − log(𝜌𝑖 ))), (5)
where pi is the network probability of a sample being a face; yidet∈ {0, 1} is the correct label.
For each window, that possibly contains face, the offset between it and the nearest correct label
can be predicted. Therefore, the research objective can be expressed as a regression problem, so the
Euclidean loss for each sample xi can be applied:
2
𝐿𝑏𝑜𝑥 = ‖𝑦̂𝑖𝑏𝑜𝑥 − 𝑦𝑖𝑏𝑜𝑥 ‖2 , (6)
𝑖
where 𝑦̂𝑖𝑏𝑜𝑥 regression target obtained from the network and 𝑦𝑖𝑏𝑜𝑥 is the correct coordinate. There are
4 coordinates, including left top, height and width, and thus 𝑦𝑖𝑏𝑜𝑥 ∈ R4.
After face detection and feature vector extraction, face classification takes part. For this purpose,
Linear Support Vector Machine [13] can be used. The SVM algorithm is implemented using a kernel.
Linear SVM can be presented with the use of the inner product between any two observations, rather
than the observations themselves. The inner product between two vectors is the sum of the
multiplication of each pair of input values.
A prediction for an input can be expressed with the following equation that uses the dot product
between the input (x) and each support vector xi:
𝑓 (𝑥 ) = 𝐵0 + 𝑠𝑢𝑚(𝑎𝑖 ∗ (𝑥, 𝑥𝑖 )), (7)
where the coefficients 𝐵0 and 𝑎𝑖 (for each input) must be obtained after applying of the learning
algorithm to the training data.
Thereby, this face classification system uses 3 methods: an MTCNN model for face detection,
FaceNet model to create a feature vectors for each detected face and Linear Support Vector Machine
(SVM) classifier model to predict the identity of a given face.
2.3. OpenCV methods: Eigenfaces, Fisherfaces, and LBPH
OpenCV (Open Source Computer Vision Library) is a computer vision and machine learning
library, that provides commonly used tools for computer vision software. It contains well-known face
recognition methods, such as Eigenfaces method, Fisherfaces method, and Local Binary Patterns
method.
The Eigenfaces method [14] can be described with the following. Let 𝑋 = {𝑥1 , 𝑥2 , … , 𝑥𝑛 } be a
random vector with observations 𝑥𝑖 ∈ 𝑅𝑑 . First step is to compute the mean value μ of these
observations:
𝑛
1
𝜇 = ∑ 𝑥𝑖 . (8)
𝑛
𝑖=1
Then the covariance matrix S can be obtained:
339
𝑛
1
𝑆 = ∑(𝑥𝑖 − 𝜇)(𝑥𝑖 − 𝜇)𝑇 . (9)
𝑛
𝑖=1
With the values of covariance matrix S the eigenvalues λi and eigenvectors vi can be expressed
with the next formula:
𝑆𝑣𝑖 = 𝜆𝑖 𝑣𝑖 , 𝑖 = 1,2, … , 𝑛. (10)
After that it is needed to order the eigenvectors descending by their eigenvalue. The core of the
Eigenfaces method is Principal Component Analysis (PCA) [15]. It computes a features linear
combination that maximizes the total variance in data. The k principal components are the
eigenvectors corresponding to the k largest eigenvalues. The k principal components of the observed
vector x are:
𝑦 = 𝑊 𝑇 (𝑥 − 𝜇), (11)
where 𝑊 = (𝑣1 , 𝑣2 , … , 𝑣𝑘 ).
The reconstruction with the use of the PCA basis is provided by:
𝑥 = 𝑊𝑦+𝜇 , (12)
where 𝑊 = (𝑣1 , 𝑣2 , … , 𝑣𝑘 ).
Mathematical description of the Fisherfaces [16] method is provided with the following. Let X be a
random vector with samples drawn from c classes:
𝑋 = {𝑋1 , 𝑋2 , … , 𝑋𝑐 }, (13)
𝑋𝑖 = {𝑥1 , 𝑥2 , … , 𝑥𝑛 }. (14)
From this representation the scatter matrices SB and Sw can be calculated as:
𝑐
𝑆𝐵 = ∑ 𝑁𝑖 (𝜇𝑖 − 𝜇)(𝜇𝑖 − 𝜇)𝑇 , (15)
𝑖=1
𝑐
𝑆𝑊 = ∑ ∑ (𝑥𝑗 − 𝜇𝑖 )(𝑥𝑗 − 𝜇𝑖 )𝑇 , (16)
𝑖=1 𝑥𝑗∈𝑋𝑖
where μ is the total mean, that expresses as:
𝑁
1
𝜇 = ∑ 𝑥𝑖 . (17)
𝑁
𝑖=1
Mean value μi of class i ∈ {1, …, c} is:
1
𝜇𝑖 = ∑ (𝑥𝑗 ). (18)
|𝑋𝑖 |
𝑥𝑗∈𝑋𝑖
Fisherfaces algorithm performs search of a projection W, that maximizes the class separability
criterion:
|𝑊 𝑇 𝑆𝐵 𝑊 |
𝑊𝑜𝑝𝑡 = 𝑎𝑟𝑔 𝑚𝑎𝑥𝑊 . (19)
|𝑊 𝑇 𝑆𝑊 𝑊 |
Following [17], this optimization problem can be solved by finding a solution of the general
eigenvalue problem:
𝑆𝐵 𝑣𝑖 = 𝜆𝑖 𝑆𝑤 𝑣𝑖 , (20)
𝑆𝑤 −1 𝑆𝐵 𝑣𝑖 = 𝜆𝑖 𝑣𝑖 . (21)
Eigenfaces and Fisherfaces methods based on a holistic approach to face recognition that involves
more efficient concentrating on extracting image local features. The basic idea of Local Binary
Patterns (LBP) method [18] is to summarize the local structure in an image by comparing each pixel
with its neighbors. Description of the LBP operator expressed with the following:
𝑃−1
𝐿𝐵𝑃(𝑥𝑐 , 𝑦𝑐 ) = ∑ 2𝑝 𝑠(𝑖𝑝 − 𝑖𝑐 ), (22)
𝑝=0
340
where (xc, yc) is a central pixel with intensity ic; in is intensity of the neighbor pixel; s is the sign
function defined as:
1 𝑖𝑓 𝑥 ≥ 0
𝑠 (𝑥 ) = { , (23)
0 𝑒𝑙𝑠𝑒
Position of the neighbor (xp, yp), p ∈ P for a pixel point (xc, yc) can be calculated by:
2𝜋𝑝
𝑥𝑝 = 𝑥𝑐 + 𝑅𝑐𝑜𝑠( ), (24)
𝑃
2𝜋𝑝
𝑦𝑝 = 𝑦𝑐 − 𝑅𝑠𝑖𝑛( ), (25)
𝑃
where R is the radius of the circle and P is the number of sample points.
2.4. Daubechies wavelets transform
The technology proposed in this research based on wavelet transform, that is widely used in the
image processing tasks. Wavelet transform provides processing of patterns hidden in the data
performing data analysis in general as well as in the detail. Wavelets apply in image processing when
it is necessary for the result of analysis to contain information about location of characteristic
frequencies and scales.
Daubechies wavelets [19] are the type of basic wavelets, that orthonormal basis defined as:
𝑗
𝜙𝑟,𝑗,𝑘 (𝑥 ) = 22 𝜙𝑟 (2𝑗 𝑥 − 𝑘), 𝑗, 𝑘 ∈ 𝑍, (26)
where function {𝜙𝑟 (𝑥 − 𝑘)|𝑘 ∈ 𝑍}, j is the scaling index, k is the displacement index, and r is the
filter index.
To analyze these equations in more detail at a certain scale, it is necessary to define an
orthonormal basis 𝜓𝑟 (𝑥) with similar properties of 𝜙𝑟 (𝑥)
𝜙 (𝑥 ) = √2 ∑ ℎ𝑘 𝜙(2𝑥 − 𝑘), (27)
𝑘
𝜓(𝑥 ) = √2 ∑ 𝑔𝑘 𝜙(2𝑥 − 𝑘), (28)
𝑘
where ∑𝑘|ℎ𝑘 |2 < ∞. Property of the scaled functions orthogonality allows to determine the
coefficients:
∑ ℎ𝑘 𝑔𝑘+2𝑚 = 𝛿0𝑚 . (29)
𝑘
Wavelets are orthogonal to scalable functions, thus wavelet coefficients gk are depend on the
scaling function coefficients hk:
𝑔𝑘 = (−1)𝑘 ℎ2𝑀−1−𝑘 . (30)
Due to these properties, Daubechies wavelets provide good results of the image processing.
2.5. Reverse biorthogonal wavelets transform
Linear phase of the biorthogonal wavelets [20] allows to use them in image processing as well. It
is maintained by the filter coefficients symmetry. Comparing to the orthogonal wavelets, biorthogonal
wavelets involve higher degree of freedom and contain a symmetrical compact support.
Let’s denote through f ( x) and y ( x) bases double to f ( x) and y ( x). , expressing it with the
formula:
[〈𝜙(𝑥 )|𝜙̅(𝑥)〉] = 𝐼, (31)
[〈𝜓(𝑥 )|𝜓̅(𝑥)〉] = 𝐼, (32)
Basic scaling functions in biorthogonal wavelet basis are orthogonal to double wavelets and
scaling functions. This can be expressed with the following:
[〈𝜙(𝑥 )|𝜓̅(𝑥)〉] = 0. (33)
341
Reverse biorthogonal system uses wavelet and scaling functions separately for signal analysis and
design both time and frequency domains. Problem of time and frequency resolution in reverse
biorthogonal wavelet function solves with the property that it is always equal to the viewport range.
3. Experimental research and analysis
First set of experiments was performed with the use of standard Python library face_recognition.
The training of KNN classifier was performed on the set of the images of 80 people, who are wearing
masks, 8 images for each person. Images are not stable in scaling, position of a face and contain
different levels of lightning. The total number of training set – 640 images. The dataset for the
identification experiment was prepared with other images of same people. Obtained results indicate
accuracy rate of identification of 55%.
In the second set of experiments FaceNet system was used. The dataset for the identification is the
same. In this case system output provides an information of the predicted class of the input image and
probability with which input image belongs to the class. FaceNet method performance on the set of
masked face images indicated the identification accuracy rate of 72,5%.
Third part of experiments was performed to analyze the efficiency of three popular facial
recognition methods: eigenface, fisherfaces, and LBPH. The accuracy of identification by eigenface
method on the set of masked face images is 12,5%. Accuracy rate of the fisherfaces method is 25%.
LBPH method provided accuracy rate of 5% of correctly identified images.
Results of performed experiments are presented in the Table 1 and Table 2.
Table 1
Results of experiments performed with face_recognition and FaceNet methods on the dataset of
masked images
Face_recognition library FaceNet
Correctly Incorrectly Correctly Incorrectly
identified images identified images identified images identified images
Total number
80 80
of images
Number 44 36 58 22
Percentage 55% 45% 72,5% 27,5%
Table 2
Results of experiments performed with Eigenfaces, Fisherfaces and LBPH methods on the dataset of
masked images
Eigenfaces Fisherfaces LBPH
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number
80 80 80
of images
Number 10 70 20 60 4 76
Percentage 12,5% 87,5% 25% 75% 5% 95%
As can be seen in a Figure 1, Eigenface, Fisherface and LBPH algorithms performed identification
with a small part of a dataset, therefore percentage of incorrectly identified images is much greater
than identification accuracy rate.
On the other hand, face_recognition and FaceNet methods do not tend to have such an issue, but
still the difference between correctly and incorrectly identified images is decreasing.
342
Figure 1: Results of identification with the use of face_recognition, FaceNet, Eigenfaces, Fisherfaces
and LBPH methods on the dataset of masked images
Results of the experimentation with the same methods on the dataset of unmasked face images are
presented in Table 3 and Table 4.
Table 3
Results of experiments performed with face_recognition and FaceNet methods on the dataset of
unmasked images
Face_recognition library FaceNet
Correctly Incorrectly Correctly Incorrectly
identified images identified images identified images identified images
Total number
80 80
of images
Number 78 2 79 1
Percentage 97,5% 2,5% 98,75% 1,25%
Table 4
Results of experiments performed with Eigenfaces, Fisherfaces and LBPH methods on the dataset of
unmasked images
Eigenfaces Fisherfaces LBPH
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number
80 80 80
of images
Number 60 20 74 6 48 32
Percentage 75% 25% 92,5% 7,5% 60% 40%
Figure 2 depicts percentage of correctly and incorrectly identified images.
Comparative diagram of the results between methods applied to the dataset of unmasked face
images and to the dataset of masked face images is depicted on Figure 3. As can be seen, presence of
the mask on the images considerably decrease identification accuracy rate from 26% to 67%.
343
Figure 2: Results of identification with the use of face_recognition, FaceNet, Eigenfaces, Fisherfaces
and LBPH methods on the dataset of unmasked images
Figure 3: Comparative diagram of the results between face_recognition, FaceNet, Eigenfaces,
Fisherfaces and LBPH methods applied to the dataset of unmasked face images and to the dataset of
masked face images
During the analysis of experimentation results of the research [21] it was found that the system
provided the highest accuracy of identification (92,5%) on unmasked face images with the use of
Daubechies wavelet transform for image processing, standard deviation and variance methods for
feature extraction stage and image classification by this vector by calculating distance with Euclidean,
quadratic Euclidean and Canberra metrics [22]. Therefore, in this research it was decided to use the
same methods for the dataset of face images with protective masks.
Results of experiments performed with Daubechies wavelet transform, standard deviation and
variance calculation, and Euclidean, quadratic Euclidean, Canberra metrics presented in Tables 5 and 6.
Figure 4 depicts results of the Daubechies wavelet transform and standard deviation calculation on
the dataset of masked images.
Figure 5 depicts results of the Daubechies wavelet transform and variance calculation on the
dataset of masked images.
Analyzing the results of Daubechies wavelet transform, the highest identification accuracy rate
result was obtained with the use of standard deviation calculation and Euclidean distance metric
(77,5%). Accuracy rates for Daubechies wavelets and other methods are following: standard deviation
calculation and Canberra metric (65%), standard deviation calculation and quadratic Euclidean metric
(70%), standard deviation calculation and quadratic Euclidean metric (57,5%), variance calculation
and Canberra metric (65%), variance calculation and quadratic Euclidean metric (57,5%).
344
Table 5
Results of experiments performed with Daubechies wavelet transform and standard deviation
calculation methods on the dataset of masked images
Standard deviation
Euclidean distance Canberra distance Squared Euclidean distance
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number of
80 80 80
images
Number 62 18 52 28 56 24
Percentage 77,5% 22,5% 65% 35% 70% 30%
Table 6
Results of experiments performed with Daubechies wavelet transform and variance calculation
methods on the dataset of masked images
Variance
Euclidean distance Canberra distance Squared Euclidean distance
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number of
80 80 80
images
Number 46 34 52 46 34 52
Percentage 57,5% 42,5% 65% 57,5% 42,5% 65%
Figure 4: Results of the Daubechies wavelet transform and standard deviation calculation on the
dataset of masked images
In the research [23] it was experimentally obtained the identification accuracy rate of 97,5% on
face images without wearing mask.
This result was provided by the method of image processing based on reverse biorthogonal
wavelets, standard deviation calculation and variance methods of feature vector extraction and Bray-
Curtis, Canberra, and Manhattan distance metrics of image feature vector classification.
Experimentation results of the same methods on the face images with mask dataset are provided next.
345
Figure 5: Results of the Daubechies wavelet transform and variance calculation on the dataset of
masked images
Results of experiments performed with reverse biorthogonal wavelets, standard deviation and
variance calculation and Bray-Curtis, Canberra, Manhattan distance metrics are presented in Table 7
and Table 8, accordingly.
Table 7
Results of experiments performed with reverse biorthogonal wavelet transform and standard
calculation methods on the dataset of masked images
Standard deviation
Bray-Curtis distance Canberra distance Manhattan distance
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number of
80 80 80
images
Number 52 28 52 52 28 52
Percentage 60% 40% 60% 60% 40% 60%
Reverse biorthogonal wavelets transform based experiments provided the highest identification
accuracy rate of 65% with combination of the following methods: standard deviation calculation and
Bray-Curtis distance metric, standard deviation calculation and Bray-Curtis distance metric, variance
calculation and Manhattan distance metric.
The results of other methods are next: variance calculation and Bray-Curtis metric – 52.5%,
standard deviation calculation and Manhattan metric – 55%, variance calculation and Canberra
distance metric – 55%.
Figure 6 depicts results of the reverse biorthogonal wavelet transform and standard deviation.
Analysis between standard deviation and variance calculation methods for feature vector
extraction indicated that the use of the first method is more effective during the experiments on both
wavelet transforms.
346
Table 8
Results of experiments performed with reverse biorthogonal wavelet transform and standard
calculation methods on the dataset of masked images
Variance
Bray-Curtis distance Canberra distance Manhattan distance
Correctly Incorrectly Correctly Incorrectly Correctly Incorrectly
identified identified identified identified identified identified
images images images images images images
Total number of
80 80 80
images
Number 42 38 44 36 44 36
Percentage 52,5% 47,5% 55% 45% 55% 45%
Figure 6: Results of the reverse biorthogonal wavelet transform and standard deviation calculation
on the dataset of masked images
Figure 7 depicts results of the reverse biorthogonal wavelet transform and variance calculation.
Figure 7: Results of the reverse biorthogonal wavelet transform and variance calculation on the
dataset of masked images
347
4. Conclusion
During the research experiments were performed on the dataset of faces covered with mask in a
conditions of different image scaling, face position and level of lightning with the use of
face_recognition, FaceNet, Eigenfaces, Fisherfaces and LBPH methods. Results of the
experimentation research are the following: face_recognition method – 55% of correctly identified
images, FaceNet method – 72,5% of correctly identified images, Eigenfaces method – 12,5%,
Fisherfaces method – 25% of correctly identified images, LBPH method – 5% of correctly identified
images.
The obtained results indicate that commonly used face recognition methods identification rates are
decreasing in a range from 26% to 67% in conditions of incomplete data, such as face covered with
protective masks, and different image scaling, face position and level of lightning.
Appliance of Daubechies wavelet transform method indicated the highest identification accuracy
rate result with the use of standard deviation calculation and Euclidean distance metric - 77,5% of
correctly identified images, that was obtained on the same dataset of faces covered with mask in a
conditions of different image scaling, face position and level of lightning. Accuracy rates for
Daubechies wavelets and other methods are following: standard deviation calculation and Canberra
metric - 65% of correctly identified images, standard deviation calculation and quadratic Euclidean
metric - 70% of correctly identified images, standard deviation calculation and quadratic Euclidean
metric - 57,5% of correctly identified images, variance calculation and Canberra metric - 65% of
correctly identified images, variance calculation and quadratic Euclidean metric - 57,5% of correctly
identified images.
Appliance of reverse biorthogonal wavelet transform method indicated the highest identification
accuracy rate of 65% on the dataset of faces covered with mask in a conditions of different image
scaling, face position and level of lightning. This result was obtained with combination of the
following methods: standard deviation calculation and Bray-Curtis distance metric, standard deviation
calculation and Bray-Curtis distance metric, variance calculation and Manhattan distance metric.
Results of the other methods are the following: variance calculation and Bray-Curtis metric – 52.5%
of correctly identified images, standard deviation calculation and Manhattan metric – 55% of
correctly identified images, variance calculation and Canberra distance metric – 55% of correctly
identified images.
Summarizing the foregoing conclusions, mathematical methods for information technology of
biometric identification, proposed in this research, can be applied for face recognition in conditions of
masked face images. The highest accuracy rate (77,5% of correctly identified images) of the
identification performed on the dataset of masked face images was obtained during the experiments
based on Daubechies wavelet transform as image processing method, standard deviation calculation
as feature extraction method and Euclidean distance metric as image vector classification method.
5. References
[1] M. Ngan, P. Grother, K. Hanaoka. National Institute of Standards and Technology Interagency or
Internal Report 8311 Natl. Inst. Stand. Technol. Interag. Intern. Rep. 8311, 58 pages (July 2020).
doi: 10.6028/NIST.IR.8311.
[2] W. Yan. Face-mask recognition has arrived—for better or worse. National Geographic,
September 11, 2020. URL: https://www.nationalgeographic.com/science/2020/09/face-mask-
recognition-has-arrived-for-coronavirus-better-or-worse-cvd/
[3] A. Ng. Facial recognition designed to detect around face masks is failing, study finds. CNET,
August 26, 2020. URL: https://www.cnet.com/health/facial-recognition-designed-to-detect-
around-face-masks-is-failing-study-finds/
[4] A. Hern. Face masks give facial recognition software an identity crisis. The Guardian. August
21, 2020. URL: https://www.theguardian.com/technology/2020/aug/21/face-masks-give-facial-
recognition-software-identity-crisis
348
[5] D. King. High Quality Face Recognition with Deep Metric Learning. Dlib C++ Library,
February 12, 2017. URL: http://blog.dlib.net/2017/02/high-quality-face-recognition-with-
deep.html
[6] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016,
pp. 770-778, doi: 10.1109/CVPR.2016.90.
[7] O. M. Parkhi, A. Vedaldi and A. Zisserman. Deep Face Recognition. In Xianghua Xie, Mark W.
Jones, and Gary K. L. Tam, editors, Proceedings of the British Machine Vision Conference
(BMVC), pages 41.1-41.12. BMVA Press, September 2015. doi: 10.5244/C.29.41.
[8] H. Ng and S. Winkler, "A data-driven approach to cleaning large face datasets," 2014 IEEE
International Conference on Image Processing (ICIP), Paris, 2014, pp. 343-347, doi:
10.1109/ICIP.2014.7025068.
[9] scikit-learn. Machine Learning in Python. URL: https://scikit-
learn.org/stable/modules/neighbors.html#classification
[10] F. Schroff, D. Kalenichenko and J. Philbin, "FaceNet: A unified embedding for face recognition
and clustering," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Boston, MA, 2015, pp. 815-823, doi: 10.1109/CVPR.2015.7298682.
[11] K. Q. Weinberger, L. K. Saul. 2009. Distance Metric Learning for Large Margin Nearest
Neighbor Classification. J. Mach. Learn. Res. 10 (12/1/2009), 207–244.
[12] K. Zhang, Z. Zhang, Z. Li and Y. Qiao, "Joint Face Detection and Alignment Using Multitask
Cascaded Convolutional Networks," in IEEE Signal Processing Letters, vol. 23, no. 10, pp.
1499-1503, Oct. 2016, doi: 10.1109/LSP.2016.2603342.
[13] scikit-learn user guide. Release 0.23.2, August 4, 2020. URL: https://scikit-
learn.org/stable/_downloads/scikit-learn-docs.pdf
[14] M. Turk and A. Pentland, "Eigenfaces for Recognition," in Journal of Cognitive Neuroscience,
vol. 3, no. 1, pp. 71-86, Jan. 1991, doi: 10.1162/jocn.1991.3.1.71.
[15] R. Mejía-Campos, D. Néjer-Haro, S. Recalde-Avincho, P. Rosero-Montalvo and D. Peluffo-
Ordóñez, "Face Detection and Classification Using Eigenfaces and Principal Component
Analysis: Preliminary Results," 2017 International Conference on Information Systems and
Computer Science (INCISCOS), Quito, 2017, pp. 309-315, doi: 10.1109/INCISCOS.2017.59.
[16] P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, "Eigenfaces vs. Fisherfaces: recognition
using class specific linear projection," in IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 19, no. 7, pp. 711-720, July 1997, doi: 10.1109/34.598228.
[17] P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, "Eigenfaces vs. Fisherfaces: recognition
using class specific linear projection," in IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 19, no. 7, pp. 711-720, July 1997, doi: 10.1109/34.598228.
[18] T. Ahonen, A. Hadid, M. Pietikäine. Face Recognition with Local Binary Patterns. In: Pajdla T.,
Matas J. (eds) Computer Vision - ECCV 2004. ECCV, 2004. Lecture Notes in Computer
Science, vol 3021. Springer, Berlin, Heidelberg. Doi: 10.1007/978-3-540-24670-1_36.
[19] I. Daubechies. Ten lectures on wavelets. Society for Industrial and Applied Mathematics, USA,
1992.
[20] C. Stolojescu, I. Railean, S. Moga and A. Isar, "Comparison of wavelet families with application
to WiMAX traffic forecasting," 2010 12th International Conference on Optimization of Electrical
and Electronic Equipment, Basov, 2010, pp. 932-937, doi: 10.1109/OPTIM.2010.5510403.
[21] G. Dimitrov, O. Bychkov, P. Petrova, K. Merkulova, Y. Zhabska, E. Zaitseva, L. Adaryukova, P.
Petrov, "Creation of Biometric System of Identification by Facial Image", Third International
Colloquium on Intelligent Grid Metrology, SMAGRIMET 2020, Zagreb, Croatia, 20-23 October.
[22] O. Bychkov, K. Merkulova and Y. Zhabska, "Information Technology of Person’s Identification
by Photo Portrait," 2020 IEEE 15th International Conference on Advanced Trends in
Radioelectronics, Telecommunications and Computer Engineering (TCSET), Lviv-Slavske,
Ukraine, 2020, pp. 786-790, doi: 10.1109/TCSET49122.2020.235542.
[23] O. Bychkov, K. Merkulova and Y. Zhabska, "Software Application for Biometrical Person’s
Identification by Portrait Photograph Based on Wavelet Transform," 2019 IEEE International
Conference on Advanced Trends in Information Theory (ATIT), Kyiv, Ukraine, 2019, pp. 253-
256, doi: 10.1109/ATIT49449.2019.9030462.
349