This paper presents the research of methods for development of information technology for person identification in video stream. For the research conduction following methods were selected: anisotropic diffusion as an image preprocessing method, Gabor wavelet transform as an image processing method, histogram of oriented gradients (HOG) and local binary patterns in 1-dimensional space (1DLBP) as the methods of feature vector extraction from the images, square Euclidean distance metric for vector classification. The purpose of the work is to analyze and test these methods in order to develop an algorithm that will form the basis of the information technology of person identification in video stream. Experimental research of the methods was performed using well-known databases, such as The Database of Faces, The FERET Database and SCface Database. The obtained results of experiments indicate that proposed information technology provides the highest identification accuracy rate of 97.5% on the images with low quality and resolution. That result means that the developed information technology can be applied for person identification in real-world conditions when it is necessary to identify person from video stream.
Face recognition technology has long been in use at law enforcement processes, state borders and
on smartphones. Nowadays it becomes a part of public and private areas of living. Hundreds of
municipalities all over the world have installed cameras equipped with face recognition technology,
sometimes promising to send data to central command centers as part of the programs that improves
crime investigation cases. The COVID-19 pandemic catalyzed the fast spread of such solutions. In
China more than 100 cities were equipped with surveillance systems based on face recognition
technology last year. Now municipal network of cameras, installed on the streets, monitors the
movement of people in residential areas of the city, when entering the offices and shops, as well as
when using vehicles. Presumably, information from cameras can be used by the city police and the data
will also be transferred to a special operational center, where computer algorithms will verify the faces
of citizens and check their status in the database of the Ministry of Health. During the pandemic in
January 2020 it was started to use citywide video surveillance system in Moscow, using software
developed and supplied by NtechLab. During the first few weeks of COVID-19 lockdown company
reported that the system found 200 quarantine violators [
Face recognition is non-invasive biometric technique, so it is the area of interest for small surveillance systems, as well as for the national security purposes. Face recognition is one of the most important law enforcement techniques when video or crime scene images are available. Automatic face recognition technologies have improved the efficiency of the judiciary and simplified the comparison process.
Modern face recognition techniques have achieved impressive results with face images of medium and high quality, but the performance is not satisfactory with low quality images. The main difficulty in recognizing low-resolution face images is the lack of facial details that distinguish it from the background. Another problem is that modern methods of face recognition are based on convolutional neural networks and use convolutional maps of features with a low sampling rate and a large step to represent a face. It leads to the information lose and inaccurate description of low-quality images.
In March 2021 analytical the company Mordor Intelligence presented the report [
Another forecasting company Grand View Research published the report on face recognition market
in May 2021. According to this report surveillance and security segments are expected to significantly
increase a compound annual growth rate (CAGR) through all analyzed period. The rate will grow till
face recognition technology had become steadily used high-security areas. For example, security and
surveillance systems are used by the law enforcement authorities with the purpose of criminals’
identification or search for missing persons [
The purpose of this work is to develop an information technology of person identification in video stream based on the algorithm that provides high identification results being applied on images with low quality and resolution.
It is well-known that information technology of face recognition and identification can be claimed efficient and reliable only when it thoroughly tested and validated, preferably on real data sets. Thus, it is necessary to research the methods, which will be used to develop the algorithm underlying in information technology under development, in order to define their efficiency on the datasets of lowquality images. In the context of this paper low-quality images define as small-sized, blurry, indistinct, pixelated, noised images, and high-quality images vice versa define as more distinct, low-compressed and noise-reduced.
During the research it was decided to use anisotropic diffusion as an image preprocessing method, Gabor wavelet transform for image processing, histogram of oriented gradients (HOG) and local binary patterns in 1-dimensional space (1DLBP). 2.1.
The idea of using anisotropic diffusion as a preprocessing technique before wavelet transform have
been proposed in [
In image processing the concept of anisotropic diffusion is a process of successive diffusion based
on more and more blurring of images in a scale space. Firstly, it was presented by Perona and Malik in
[
This method is based on the approximation of a parameterized version of the above defined process
obtained by applying anisotropic diffusion equation, results in the space variant filters being anisotropic,
close to lines and edges. The attributes of anisotropic diffusion are the anisotropic smoothing and
iterative diffusion for the processing of each image pixel [
Anisotropic diffusion equation can be expressed as following:
=
( ( , , ) ) = ( , , )
+ ∙ ,
(1)
where div is the divergence operator, ∇ is the gradient operator, Δ is the Laplacian operator, I0(x, y) is
the input image, t is the Gaussian kernel variance G(x, y, t0) and I(x, y, t) is a family of derived images
obtained by convolving the original image I0(x, y) with a Gaussian kernel G(x, y, t0). Anisotropic
diffusion helps successfully remove noise and preserve image edges and small structures if the diffusion
coefficient or edge stopping function c(∇I) is estimated correctly. If c(x, y, t) is a constant the equation
can be reduced to the equation of isotropic heat diffusion = [
The usage of the combination of anisotropic diffusion and Gabor wavelet transform was described
in [
Gabor wavelet transform is widely used in pattern recognition field because of its biological significance and technical properties. The complex Gabor function in the spatial domain is defined by the following: s( x, y) – is a complex sinewave (carrier), defined as:
( , ) = ( , ) ( , ).
( , ) = exp(j(2π( 0 + 0 ) + )), where (u0 , v0 ) and P determine the spatial frequency and sinewave phase respectively; wr (x, y) – Gaussian 2D-function (envelope function). The Gaussian envelope is represented as the following: (4) ( , ) = K ∙ exp(− ( 2( − 0) 2 + 2( − 0) 2, where (x0 , y0 ) is the peak of the function, a and b are the Gaussian scaling parameters, and the index r denotes the rotation operation, which can be described this way: (2) (3) (5) (6)
Therefore, the complex Gabor function in the spatial domain can be written as follows [
In [
The Histogram of Oriented Gradients (HOG) method can be applied to images that were processed by wavelet transform to extract important features of image shape. Histogram of oriented gradients calculates with the execution of the following steps. Firstly, before building of the orientation histogram for each cell, there must be obtained the value of the gradient. After that, obtained histograms that grouped by the individual cell are normalized. This process can be described mathematically: = [ 0 ]. derivative mask.
The gradient histogram retrieves with the equations: Gradient size is calculated as: Orientation is calculated with the following equation: ( ) = ∙ ,
( ) = ∙ . | | = √ 2( ) + 2( ).
= tan−1 .
(8) (9) (10) (11) (12) (13)
Then orientation distributes equally in a range of 0 to 180 deg (for an unsigned gradient) or 0 to 360 deg (for a signed gradient). Each pixel computes a channel weighted vote based on the gradient values.
After that cells groups into blocks that are connected in space. It allows to obtain the vector of the normalized histogram elements. 2.4.
For the palmprint recognition task authors of [
Local Binary Patterns in 1-Dimensional Space (1DLBP) firstly was presented in [
The purpose of the 1DLBP method is to describe the local agitation of 1-dimensional signal segment in binary code. That description can be obtained by comparing of the neighbor pixel values with the central pixel value. All neighbor elements get the value of 1 if they are greater or equal to the current element and value of 0 if they are less than the current element. Then, each element of the obtained vector is multiplied by a weight according to its position. Finally, the current element is replaced by the sum of the values of obtained vector. Described process can be expressed as follows:
Dx is a 1-dimensional horizontal discrete derivative mask, Dy is a 1-dimensional vertical discrete where S(x) function defines as S(x)={1 if x⩾0;0otherwise}; g’ and gn are the mean value of the linear neighborhood and the values of its 1-dimensional neighbors, respectively.
The 1DLBP algorithm applies on all blocks of the split image with different resolution. The extracted
histograms are concatenated in one global 1-dimensional vector that represents one face image [
Summarizing the foregoing description of 1DLBP method, the following stages of image processing can be distinguished to obtain 1DLBP feature vector:
Input image preprocesses.
1 = ∑ ( − 0) ∙ 2 . −1 =0 −1 =0 descriptor. summed neighbors:
S(x) function defines as S(x)={1 if x⩾0;0otherwise}; g0 and gn are the values of the central element and its 1-dimensional neighbors, respectively. The index n changes its value increasingly from the left to the right in the 1-dimensional string. The histogram of the 1-dimensional pattern defines the 1DLBP
The idea of the I1DLBP is similar to the idea of the 1DLBP - the local patterns are extracted by thresholding the linear neighbors of each pixel, from the projected image, with the mean value of the 1 =
∑ ( − ́ ) ∙ 2 ,
Image splits into multiple blocks.
Each decomposed block projects in 1-dimensional space. 1DLBP algorithm processes each projected block.
The vectors that obtained after each block processing concatenate in one global feature
vector [
Experimental research of the selected methods was performed using software created with Python. The research was conducted on three different face databases: The Database of Faces, Facial Recognition Technology (FERET) database and Surveillance Cameras Face Database (SCface). The purpose of experimental research is to establish the most efficient combination of methods by comparing results of their work with the use of one feature extraction method (HOG or 1DLBP) and the use of combination of feature extraction methods (HOG and 1DLBP). For the classification of the feature vector, obtained as a result, square Euclidean distance metric was used in all sets of experiments.
The examples of images obtained as a result of applying described methods are depicted on Figure 1.
First set of experiments was performed using The Database of Faces that was created by AT&T
Laboratories Cambridge during the work on face recognition project. Database presented with a set of
face images of 40 different individuals. It is organized in 40 directories with 10 PGM format images in
each directory. The size of each image is 92x112 pixels, with 256 grey levels per pixel. Images were
taken with the person in frontal position with lighting, facial expression and facial details varying. The
database was retrieved openly from official site of AT&T Laboratories Cambridge [
The FERET Database is a part of the Face Recognition Technology program [
Experimental results for The FERET Database are presented in Table 2.
As can be seen from Fig. 3, on high resolution images the usage of 1DLBP method is not necessary, because the identification accuracy rate is the same during the experiment performance with HOG feature vector extraction method only, as well as with combination of HOG and 1DLBP methods, which is 72.5%. For the FERET database the usage of two feature extraction methods is not justified. As a conclusion, false recognition rate obtained during the experiments with The FERET Database varies from 27.5 to 35.5%.
SCface database was created by researchers from University of Zagreb [
From Fig. 4 it can be concluded that using 1DLBP method for feature extraction process by itself provides only 77.5% of correctly identified face images. Usage of HOG method improves algorithm work results within the order of 15% and provides the identification accuracy rate of 92.5%. But fusing of HOG and 1DLBP feature vectors escalate the correctness of algorithm to 97.5%. Results of this set of experiments confirms that the usage of combination of two feature vector extraction methods is justified on low quality and resolution image, and the false recognition rate in this case is the lowest among all the experiments – 2.5%.
Comparative diagram on Fig. 5 presents the results of identification with the use of selected methods on different databases. Taking into consideration the fact that facial images from SCface database, that were used for experimentation performance, are low-quality video surveillance images, it can be concluded that combination of all researched methods provides high identification accuracy rate on low-resolution facial images.
After the analysis of all experimentational results it can be concluded that the combination of such methods as anisotropic diffusion, Gabor wavelet transform, histogram of oriented gradients and local binary patterns in 1-dimensional space demonstrates the highest identification results being applied to all researched databases.
From the perspective of analysis of obtained research results, combination of methods with the highest identification accuracy rate can be used to develop the algorithm underlying in information technology for person identification in video stream.
In general, the person identification process consists of two stages. The first is the determination of
face location in the image taken from video stream. The original image is scanned with a smaller
window, and every time there is a determination of the similarity degree between the image in the
window with a human face. Formally, the face image can be structurally defined, statistically or by
sample face images list [
The algorithm uses a portrait image of a person as input data. Step-by-step algorithm execution can be described as follows:
1. Face detection and localization on the input image. Only face image of a person is used for further processing because it is the area of interest for person identification process.
2. Image pre-processing using anisotropic diffusion that allows to preserve and enhance edge information and remove noise.
3. The image of the face, after applying anisotropic diffusion to it, is processed by Gabor wavelets with various changes in the parameters of the wavelet function so that 16 wavelet-transformed variations of the face image can be obtained. After that, the wavelet-transformed images are summed to form a global image, which is processed by the following methods.
4. The global image formed as a result of the Gabor wavelet transform is simultaneously fed to the input of two independent methods - HOG and 1DLBP. As a result of the operation of each of these methods, two separate 512-value feature vectors are formed.
5. HOG feature vector and 1DLBP feature vector normalization.
Because of the deviation in vector distributions and ranges, feature vectors extracted separately from
1DLBP and the HOG features are incompatible. There are some methods of vector normalization, that
can improve compatibility. For example, min–max normalization that transforms the feature vectors in
the range [
6. The HOG feature vector and 1DLBP feature vector concatenates to form a 1024-value global feature vector of a face image. The global feature vector of the image can be obtained by concatenating into a single feature vector the normalized feature vectors of 1DLBP and HOG features.
Let the normalized feature vectors be D = [d1; d2; d3 ; ...; dn] for 1DLBP and h = [h1; h2; h3; ...; hn]
for HOG extraction. The global vector can be represented as [
The obtained global feature vector is using for further classification.
The algorithm for information technology of person identification in video stream that includes such methods as anisotropic diffusion, Gabor wavelet transform, histogram of oriented gradients and local binary patterns in 1-dimensional space proposed in this paper firstly.
This paper describes the research of methods to develop the algorithm underlying in information technology of person identification in video stream. During the research, experiments were performed on the testing of methods being applicable in face recognition in order to establish the most efficient combination of methods by comparing results of their work with the use of one feature extraction method (HOG or 1DLBP) and the use of combination of feature extraction methods (HOG and 1DLBP).
The obtained results indicated that the highest identification accuracy rate of 97.5% was obtained with both feature extraction methods (HOG and 1DLBP) on the images from SCface database, which are low-quality video surveillance images. Experimental results on the Database of Faces and FERET database provides from 70 to 72.5% of correctly identified images.
Proposed algorithm is based on anisotropic diffusion as an image preprocessing method, Gabor wavelet transform as an image processing method, histogram of oriented gradients (HOG) and local binary patterns in 1-dimensional space (1DLBP) as the methods of feature vector extraction from the images, square Euclidean distance metric for vector classification.
Summarizing the foregoing conclusions, proposed in this work algorithm for information technology of person identification based on anisotropic diffusion, Gabor wavelet transform, histogram of oriented gradients (HOG) and local binary patterns in 1-dimensional space (1DLBP) can be applied for face recognition and person identification in conditions of face images with low quality and resolution, that means it partially solves small sample size problem. For future research it is aimed to improve the (14) (15) proposed algorithm for the scenario when low quality video surveillance images compare with frontal mug shot image from law enforcement or national security databases.