Today's challenges determine the need to improve the biometric authentication of personnel of critical infrastructure facilities. Common means of biometric authentication, which are usually based on the use of neural network technologies for facial image analysis, need improvement in the direction of adaptation to the conditions of recognition during the performance by personnel of their functional duties, which are characterized by the influence of interference during video recording and an increase in the probability attacks using dummies. Another area of improvement is determined by the availability of video recording tools, which provide the ability to recognize a person by the iris of the eye and the ability to recognize emotions. It is shown that the first stage of improvement of neural network means of biometric authentication is the development of a formalized description of the recognition process, which takes into account promising areas of improvement. A conceptual model containing a formalized description and criteria for evaluating the effectiveness of the recognition process is proposed. At the same time, for the first time, an approach to determining the parameters of obstacles was proposed, which involves comparing the parameters of obstacles with the location and number of key and control faces that they overlap. Recognition of attacks is proposed to be implemented based on the analysis of the dynamics of basic emotions, the dynamics of eye movement parameters and the environment. The results of this study are important in the context of the development of effective biometric authentication tools, as they provide a formalized description of the requirements for the functional capabilities of the main components of the process of recognizing the identity and emotions of personnel of critical infrastructure facilities.
In modern conditions, even minor security breaches of critical infrastructure facilities (CIF) can have a significant negative impact on the social and economic spheres of the state, as well as its defence capability and national security. One of the main requirements for a comprehensive information protection system of CIF is to ensure reliable identification and authentication of CIF personnel. For this purpose, a wide range of tools is used, including biometric authentication systems (BA) based on facial images (FI) and iris of the eye (IE) recognition [1, 26]. This is primarily explained by the widespread availability of highquality video cameras capable of accurately capturing relevant biometric parameters, enabling sufficiently precise identification and authentication of personnel. However, the results [13, 15, 16] and practical experience indicate insufficient accuracy of such authentication systems in terms of face recognition under adverse conditions. Additionally, in modern conditions, for effective performance of duties, CIF personnel must be in a satisfactory psycho-emotional state, which can be monitored both during personnel access to the object and while performing official duties through the analysis of FI and IE. This explains the relevance of research aimed at enhancing CIF security through the development and implementation of BA systems that provide both facial recognition and assessment of the emotional state of personnel.
In the process of analysis of scientific and practical works devoted to the development of means of recognizing a person and the emotional state of a person, attention was focused on the identification of promising solutions that can be used to ensure the effective functioning of the BA systems of the CIF staff.
In [30], the use of MobileSSD technology is described for the selection of human faces in a video stream in real-time. The experiments were carried out under conditions of various obstacles, including the presence of glasses on the face, and the presence of foreign objects on the face that interfere with the fixation of characteristic points of the face. Also, the effectiveness of the technology was determined in different conditions of the lighting level, and the distance from the face to the camera. It was determined that turning the face most often leads to the impossibility of its selection.
In [16], the use of computer vision technologies to solve the problem of recognition of FI, eyes and PEO in real time is defined. It is proposed to use the Blob Analysis approach to identify FI, to use a threshold function for segmentation, and to use the Circle Hough approach to highlight the IE. It is declared that the use of these solutions allows achieving high recognition results on low-quality images.
In [29], the use of deep learning methods for recognizing human emotions based on FI is considered. Conduct recognition of 7 emotions. A previously trained Haar cascade classifier was used to distinguish the face, eyes, and mouth. A convolutional neural network (CNN) was used to recognize emotions, having previously trained it on the FER-2013 database. Claimed recognition accuracy of 0.62 on the test sample.
In [18], experimental studies were conducted to reveal the dependence of a person's recognition of an actor's emotion based on his facial expression on the presence of a mask on his face. It was found that the presence of a mask worsens the recognition of facial emotions by about 20%.
In [15], in addition to the effect of the presence of a mask, the effect of the presence of sunglasses on the recognition of an actor's emotion and his identification by face is also analyzed. It was found that, unlike the presence of a mask, the presence of sunglasses did not reduce the accuracy of emotion identification and recognition.
A study of the possibilities of such FI and ROO analysis systems as FaceReader from the Noldus Information Technology company, Сaptemo from the Logic Pursuits company and BioObserver from the Herta company was also conducted.
Although the results of the analysis of scientific and practical robots and known software and hardware solutions indicate the expediency of their application to CIF, these same results testify to the complication of the identification and authentication procedure for FI and IE under the influence of various interferences. It is also possible to conclude the most promising analysis of FI and IE with the help of neural network means (NN means). Another important direction of improvement of BA tools based on FI and IE is the need to increase the effectiveness of protection against attacks with the help of dummies [31, 9, 17].
At the same time, common means of recognizing such attacks are based on the analysis of the quality of the controlled image [4], the analysis of its spatial characteristics [23, 27], the verification of the dynamics of the parameters characteristic of the FI and IE of a living person [3, 5, 26] and the execution by the controlled person of certain commands that cause changing video registration parameters [10, 12]. At the same time, the same works noted the difficulties of effective analysis of the quality of FI and IE under variable conditions of video recording and the difficulty of determining spatial characteristics during video recording with one camera.
Also, the results of the conducted analysis indicate the absence of a formalized holistic description of the BA process of CIF personnel according to FI and IE, which takes into account the presence of typical problems, the possibility of attacks on the BA system with the help of dummies, the need to identify the identity and emotional state of the staff, and so on the mechanism for determining the effectiveness of the BA process.
The main purpose of this study is to develop a conceptual model that provides a formalized holistic description of the process of recognizing a person based on the image of the face and the iris of the eye during biometric authentication of personnel of critical infrastructure facilities using neural network tools, taking into account the need to identify emotions and detect attacks using dummies.
Designing a conceptual model for face recognition incorporating both facial and iris recognition for critical infrastructure facilities involves several key components. Here's a proposed breakdown: 1. Image Acquisition. The process begins with capturing images of personnel using high-resolution cameras placed strategically at entry points or checkpoints within the facility. Pre-processing. Raw images undergo pre-processing to enhance quality and remove noise. This step may involve techniques like normalization, resizing, and filtering to standardize the images. 2. Facial Recognition. Employ algorithms like Haar cascades or deep learning-based methods to locate faces within the images. Feature Extraction. Utilize techniques like Principal Component Analysis (PCA), Local Binary Patterns (LBP), or Convolutional Neural Networks (CNNs) to extract discriminative features from the detected faces. Matching. Compare the extracted features against a database of known personnel using methods such as Euclidean distance, cosine similarity, or deep metric learning. 3. Iris Recognition. Locate and isolate the iris region within the captured face images using techniques like Houh transforms or template matching. Feature Extraction. Extract unique features from the iris pattern using methods like Gabor filters or wavelet transforms. Matching. Compare the extracted iris features against a database of enrolled personnel using algorithms like Hamming distance or phase correlation. 4. Integration. Combine the results from facial and iris recognition modules to increase the overall accuracy and reliability of the identification process. Employ fusion techniques such as score-level fusion or decision-level fusion to integrate the outputs from individual recognition modules. 5. Decision Making. Based on the fused recognition scores, decide the identity of the personnel. Apply thresholding techniques to determine acceptance or rejection based on the similarity scores obtained from the recognition process. 6. Access Control. Grant or deny access to the personnel based on the decision made during the recognition process. Interface with the facility's access control system to activate/deactivate entry mechanisms like doors, turnstiles, or gates. 7. Feedback and Iteration. Provide feedback to the system based on the outcomes of recognition decisions to improve performance over time. Employ techniques such as adaptive learning or retraining of the recognition models using new data to enhance accuracy and robustness.
Also, when developing the conceptual model, the results of [22, 24, 27] were taken into account, which led to the use of the following terms: • Neural network model - a model that describes the architecture of an artificial neural network and characterizes the neurons that are part of it. • IE is a colored ring in the front part of the pupil, consisting of muscle and connective tissue and pigment cells, which changes the size of the pupil of the eye. • FI - the image of the front part of the human head, which is bounded from above by the forehead, below by the lower edge of the chin, and from the sides by the base of the auricles. • BA - authentication based on the results of the analysis of a person's biometric data. • Attack using fakes (spoofing) – an attack based on providing a sensor to read fake biometric data. • Emotion is a mental reflection in the form of a direct, biased experience of the vital meaning of phenomena and situations, determined by the relationship of their objective properties to the needs of the subject. • Basic emotions - anger, disgust, sadness, fear, surprise, contempt, joy. • •
Key points are points on the face that are used to recognize emotions.
Control points – points on a person's head that are used to recognize a person.
This conceptual model forms the basis for implementing a comprehensive face recognition system incorporating both facial and iris recognition for securing critical infrastructure facilities.
Taking into account the specifics of the problem of developing NN means BA of CIF
personnel, in the base case the proposed model is intended to describe the processes of
neural network processing of the registered video stream to recognize the identity of CIF
personnel and the presence of an attack using dummies on the BA system by FI and IE:
〈 , 〉 → 〈 , , 〉, (
Taking [2, 28] into account, it is accepted that the elements of the set , which characterize the conditions of video registration, include: 1 – the minimum permissible level of illumination without the use of infrared illumination; 2 – the maximum permissible level of illumination; 3, 4 – viewing angles of the video camera horizontally and vertically; 5 – frame rate; 6 – color gamma format; 7 – video stream resolution; 8 – range of action of infrared illumination; 9, 10 – the maximum possible angles of changing the direction of video recording horizontally and vertically when the video camera functions in the object tracking mode; 11 – illumination range in the visible range of light (white illumination); 12 – distance to the face; 13, 14, 15 – angles between the direction of video recording and the projection of the face onto the planes Oxy, Oxz, Oyz.
The general conditions for using a video surveillance system within one location are as follows: 16 – illumination; 17 – number of video cameras; 18 – the maximum possible number of monitoring objects; 19 – the presence of obstacles.
Based on the analysis of standard solutions in the field of video surveillance, it was determined that the parameter 5can take values from 7 to 60 frames per second. The parameter 6 can be RGB, RGBA, BGR, monochrome or CMYK. The 7parameter can take the following values: VGA (640x480 - 0.3 MP), HD (1280x720 - 1 MP, 1280x960 - 1.3 MP), FullHD (1920x1080 - 2 MP), UHD (4K-3840x2160 - 8 MP, 8K - 7680x4320 - 33 MP). ).
In addition, video surveillance systems provide opportunities for: changing the spatial orientation of the video camera; changes in lighting; issuing commands to clarify the position of the face in space and eliminate obstacles. The corresponding parameters are marked as 20 − 23. In addition, video surveillance systems provide opportunities for: changing the spatial orientation of the video camera; changes in lighting; issuing commands to clarify the position of the face in space and eliminate obstacles. The corresponding parameters are marked as ( 24) and video data packet reception frequency ( 25) are also taken into account.
Consider the set of parameters characterizing the content of each of the frames of the video stream, the parameters of which are displayed in the elements of the set . Since each of the frames of the video stream is essentially a static monochrome or color image, a separate element of the set can be represented as: = | … 1,1 ,1 … 1, … … ,
… | = 1 … , where is the nth frame; s the number of frames of the video stream; – horizontal frame size; is the vertical frame size; , is the color of the pixel with coordinates( , ).
When defining the set , it is taken into account that each of its elements is interpreted as the confidence that the j-th representative of the CIF staff is recognized in the video stream. At the same time, 0 ≤ ≤ 1. Taking into account the need to determine that an illegitimate person should be recognized in the video stream and there may not be a human object at all, the number of elements is calculated as follows:
= + 2, where is the number of legitimate representatives of the CIF staff.
In the case where lies in the range from 1 to , represents the confidence that the j-th CIF personnel representative is recognized in the registered video stream. For = + 1, is the confidence that an illegitimate person is recognized in the registered video stream, and for = + 2, is the confidence that there are no people in the registered video stream.
The elements of the set describe the emotions of the CIF personnel representative, recognized based on the neural network analysis of the FI. Since in most authoritative works it is accepted that the set of basic emotions includes joy, anger, disgust, fear, sadness, surprise and neutrality, it is appropriate to describe the spectrum of basic emotions using seven parameters. Thus, each of the seven elements of ( ∈ , 0 ≤ ≤ 1) is correlated with the manifestation of a basic emotion on the face.
Thus, expressions (
To detail the basic version of the model, the decomposition of face recognition by FI and IE at the BA of CIF personnel was carried out, taking into account the need to recognize emotions and attacks using dummies.
According to [23, 27], when decomposing the procedure for recognizing a person, attention is focused on specific operations, the effectiveness of which in modern BA systems of CIF personnel can be considered insufficient. These operations should include: • • •
Selection of FI and IE in the video stream.
Detection and leveling of recognition obstacles.
Attack detection using dummies.
According to [7, 19, 25], the recognition process can be presented as a sequence of the
following operations:
(
Pre-processing of the image → .
Selection of contours FI and IE → . Determining the coordinates of control points → .
Determining the coordinates of key points → . Leveling of obstacles related to control points → .
Leveling of obstacles that concern key points → .
Neural network person recognition → .
Neural network recognition of emotions → .
Neural network recognition of additional parameters to identify attacks using γ dummy → .
Neural network recognition of attacks → .
The diagram of the decomposition of the procedure of recognition of the person according to FI and IE at the BA of the CIF staff, taking into account the need to determine emotions and detect attacks using dummies, is shown in Fig. 1.
Means of processing and recognition 〈 , 〉
Checking the parameters of
the video registration with an acceptable value
Processing of video stream parameters
Highlighting contours of the image of the face and iris
Identification and leveling of obstacles
Application of neural networks 〈 , , 〉
for face, emotion and
attack recognition
Taking into account the generally accepted technology of neural network analysis of a
video stream, the parameters of the model (̃ ) relating to the selection of FI and IE contours
should include the parameters that describe these contours in the pre-processed video
stream. By analogy with (
The presence of obstacles is intended to be described using the set . Note that the data [6, 22] indicate the absence of a generally accepted approach to determining the parameters of interference. Therefore, to describe obstacles related to the visibility of key and control points, an approach is proposed that involves comparing the parameters of obstacles with their location and the number of key and control points that they overlap. It is assumed that the obstacle may overlap one or more zones shown in Fig. 2.
Zone A corresponds to the upper part of the human head, zone B to the eyes, zone C to the nose, D to the mouth, and E to the lower part of the face. Index 1 corresponds to the left part of the face, and index 2 to the right part. The intensity of the disturbance localized in a certain area of the face is suggested to be compared with the number of key and control points that it overlaps. The intensity of interference on the IE is correlated with the area covered by this interference. Note that the proposed approach allows you to adapt the interference intensity to use in models with different numbers of key and control points. Thus
where 1, … , 10 – a parameter that determines the intensity of interference localized in the area of the face image A1, A2, B1, B2, C1, C2, D1, D2, E1, E2, respectively; 11, 12 – a parameter that determines the intensity of interference on the IE of the left and right eye.
= { 1, 2, … , 12}, =
1 =1 ∑ , 0 ≤ ≤ 1, (5) (6) • • • • • • • • where – the number of the corresponding area of the face image; – the number of control/key points in the zone ; – the degree of reduction in the visibility of the - th control/key point in the - th area of the face due to the effect of interference.
Based on the results [8, 20, 24], approaches to detect attacks on the BA system based on the use of human face and eye dummies are proposed. The first approach involves the detection of an attack based on the results of the analysis of parameters characterizing the image of the environment, and the second - the detection of an attack based on the results of the analysis of the dynamics of FI and IE parameters, which describe the basic emotions of the CIF staff representative and additional parameters describing the dynamics of eye movements. Therefore, the set , which contains additional parameters used to recognize attacks using dummies, includes:
When using the second approach, the set includes the parameters describing:
Considering (
, 〈 , 〉 ℎ
→ , 〈 , 〉 → ̃ , 〈̃ 〈̃ 〈̃ , ̃ 〈 , ̃ 〉 → 〈 , ̃ 〉 → 〈̃ , 〈̃ , 〉 → 〈̃ 〈̃ 〉 → , , 〈̃ , 〉, 〈̃ , 〉, - is the set of permissible values for ∈ ; - set of admissible values for – a tuple consisting of sets of , , hat have passed the verification of compliance of video registration parameters with permissible values; – a set of pre processed video stream parameters; ̃
, ̃ - sets of parameters of control and key points; , – sets of interference parameters; ̃ , ̃ points after leveling of obstacles;
– sets of parameters of control and key - sets of parameters of the interference.
Also, according to [11, 21, 14], the conceptual model includes a set of criteria for assessing the accuracy of the recognition process. For operations → are related to the classification of objects, the specified criteria include Accuracy, Recall, , → , → Γ , → , which Precision and F1-score. For the operations → , → segmentation, the Dice criterion was used: , →
, → , → , which are related to semantic (18) where – is the number of points describing the selected object; , is the value characteristic of the nth pixel of the selected object; , - is the value characteristic of the ith pixel of the expected output signal.
Using the proposed conceptual model of recognition (7-18) in the development of NN means, it is necessary to take into account the level of development of technologies of neural network analysis of the video stream and the criteria for evaluating the effectiveness of BA tools.
Let's look at the benefits of using a conceptual model.
The conceptual model of facial and iris recognition offers several benefits for critical infrastructure security. Enhanced Security. This approach offers a stronger layer of security compared to traditional methods like keycards or passwords. Facial and iris recognition are unique biometric identifiers that are difficult to forge or replicate. Multimodal Authentication. By combining facial and iris recognition (multimodal approach), the system adds an extra layer of verification. Even if someone manages to spoof a face, a mismatch in the iris recognition would deny access. Improved Access Control Efficiency. Facial and iris recognition systems can automate the access control process, reducing wait times and streamlining entry for authorized personnel. Reduced Reliance on Physical Credentials. Physical access cards can be lost, stolen, or copied. Facial and iris recognition eliminates the need for physical credentials, minimizing the risk of unauthorized access. Potential for Deterrence. The very presence of a sophisticated facial and iris recognition system can deter potential intruders, knowing they face a significant hurdle to gain access. Auditability. The system can maintain a record of access attempts, allowing for easier identification and investigation of suspicious activity. The conceptual model provides a framework for a secure and efficient access control system that leverages the unique identification capabilities of facial and iris recognition technology.
As a result of the analysis of scientific and practical works, it is shown that to build effective NN means of person recognition based on FI and IE of CIF personnel, it is necessary to supplement the methodological base by developing a conceptual model that will provide a formalized description of the recognition process.
It was determined that the recognition procedure includes the operations of checking the admissibility of the video registration parameters, refining the parameters of the video stream, selecting FI and IE contours, detecting and leveling interference, and applying NN means. For each of the specified operations, a list of efficiency assessment criteria adapted to the characteristics of modern means of implementation is substantiated.
For the first time, approaches to determining the parameters of obstacles for recognizing faces and emotions and recognizing attacks using dummies are proposed. The approach to determining the parameters of obstacles involves comparing the parameters of obstacles with the location and number of key and control faces that overlap. Approaches to the recognition of attacks with the help of dummies involve the detection of such attacks based on the analysis of the dynamics of basic emotions, eye movement parameters, and the environment during video recording.
Analytical expressions have been developed that provide a formalized description of each of the operations, and together, determined by the accuracy assessment criteria, form a conceptual model of the process of recognizing a person by FI and IE at the BA of CIF personnel using NN means, taking into account the need to determine emotions and detect attacks using dummies.
With the use of the developed recognition model, the prospects of improving NN means BA systems due to the use of the proposed approaches to parameter determination were determined obstacles and recognizing attacks using dummies. The development and implementation of a face recognition system integrating both facial and iris recognition technologies offer a robust solution for enhancing security at critical infrastructure facilities. By following the conceptual model outlined and the subsequent steps, organizations can Improved Security, Enhanced Efficiency Increased Reliability Adaptability and Scalability, and Continuous Improvement.
In essence, the implementation of a face recognition system incorporating facial and iris recognition technologies represents a proactive approach to security management, fostering a safe and secure environment for critical infrastructure facilities and their personnel. [5] Chandrani, S., Washef, A., Soma, M., & Debasis, M. (2015). Facial Expressions: A CrossCultural Study. In Emotion Recognition: A Pattern Analysis Approach (pp. 69-87). Wiley Publ. doi:10.1002/9781118910566. [6] Connaughton, R., Bowyer, K. W., & Flynn, P. J. (2013). Fusion of Face and Iris Biometrics.
In Handbook of Iris Recognition (pp. 219-237). Springer. [7] Dychka, I., Chernyshev, D., Tereikovskyi, I., Tereikovska, L., & Pogorelov, V. (2020).
Malware Detection Using Artificial Neural Networks. In Z. Hu, S. Petoukhov, I. Dychka, & M. He (Eds.), Advances in Computer Science for Engineering and Education II. ICCSEEA 2019. Advances in Intelligent Systems and Computing (Vol. 938, pp. 3-12).
Springer, Cham. doi:10.1007/978-3-030-16621-2_1. [8] Held, G. (2003). Securing wireless LAN's. Macon, Georgia, USA: Publisher. [9] Linartz, J.-P., & Tuyls, P. (2003). New shielding functions to enhance privacy and prevent misuse of biometric templates. In Proceedings of 4th International Conference on Audio And Video Based Biometric Person Authentication (pp. 393-402). [10] Lu, X., & Jain, A. K. (2011). Deformation modeling for robust face matching. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 30(8), 1347–1358.
[11] Ma, X., Fu, M., Zhang, X., Song, X., Becker, B., Wu, R., Xu, X., Gao, Z., Kendrick, K., & Zhao,
W. (2022). Own Race Eye-Gaze Bias for All Emotional Faces but Accuracy Bias Only for
Sad Expressions. Frontiers in Neuroscience, 16, 1-11. doi:10.3389/fnins.2022.852484.
[12] Matusugu, M., Katsuhiko, M., Yusuke, M., & Yuji, K. (2003). Subject independent facial
expression recognition with robust face detection using a convolutional neural
network. Neural Networks, 16(5-6), 555–559.
[13] Mian, A. S., Benamoun, M., & Owens, R. (2011). Keypoint detection and local feature
matching for textured face recognition. International Journal of Computer Vision, 80(
Pearson. [21] Tariq, U., Lin, K., Li, Z., Zhou, Z., Wang, Z., Le, V., Huang, T. S., Lv, X., & Han, T. X. (2012).
Emotion recognition from an ensemble of features. IEEE Transactions on Systems, Man,
and Cybernetics, Part B: Cybernetics, 42(
Research Laboratories Technical Report TR2005-097, 67. [28] Viola, P. (2005). Robust real-time face detection. International Journal of Computer
Vision, 58(