Information Security Systems

Abstract

Throughout human history, emotional manifestations have played a major role in interpersonal interaction among humans in all areas of society. In particular, information security systems for visual surveillance, based on recognizing emotional states by facial expressions, have recently become highly relevant. In this paper, we propose a method of representing geometric facial features, which aims to enhance the functioning of visual surveillance for information security systems. The method is designed to automatically reflect the facial expressions of human emotions in the form of quantitative characteristics of geometric shapes. It uses software-generated landmarks for constructing specific geometric characteristics of the face, which serve as input data for the method. Our method consists in forming seven geometric shapes based on predefined landmarks, with the subsequent quantitative expression of these shapes. The method derives quantitative features of seven forms, which are further used to identify emotional facial states. We validated the proposed method using hyperplane classification and compared its performance with analogs. As such, the classification model, which was constructed based on the proposed method, achieved a classification accuracy of 92.73% and slightly surpassed the analogs in other statistical indicators. Overall, the results of computational experiments confirmed the effectiveness of the proposed method for identifying changes in a person's emotional state by facial expressions. In addition, the use of simple mathematical calculations in our method has significantly reduced the computational complexity against analogs. Emotion recognition, emotion detection, facial feature extraction, geometric feature, face IntelITSIS'2022: 3rd International Workshop on Intelligent Information Technologies & Systems of Information Security, March 23-25,

-

orientation, information security, hyperplane classification

1. Introduction

Human emotions are crucial in interpersonal communication between people and human-machine interaction. Facial expressions have been considered the most effective and straightforward means of nonverbal interaction in systems with a human-machine interface (HMI) [ 1 ]. Methods for recognizing changes in a person’s emotional state are successfully used in alternative communication systems [ 2 ], clinical analysis [ 3 ], security systems [ 4 ], etc. Despite significant scientific and engineering advances

2022 Copyright for this paper by its authors. in emotion recognition [ 5 ], there are still several challenges in improving the performance of real-time HMI systems that might work more effectively.

Over the past few years, researchers have proposed various algorithms for detecting emotional facial expressions. There are four common characteristics for recognizing emotional facial changes: more reliable detection, convenience, cost-effectiveness, and less computational cost [ 6 ], [ 7 ]. The standard emotion classification system distinguishes six primary categories: anger, fear, joy, sadness, disgust, and surprise. Usually, investigations of novel methods for emotion recognition are conducted to detect changes in emotional manifestations in the real world based on well-known international datasets of facial expressions. Algorithms in such cases are tested and validated on a limited number of static images, working primarily offline. Nevertheless, such approaches do not identify macro facial features and are not suitable for detecting emotional expressions in real-time.

Recently, there has been significant interest among the scientific community in recognizing facial expressions by image and video sequences. These scientific studies mainly utilize the facial coding system (FACS) [ 8 ], created by Paul Ekman et al. to analyze the emotional behavior of the face. FACS is designed as a human-observer recognition system to detect minor changes in facial features. It presents the human face in fully controlled models, i.e., 46 action units (AUs), each representing socalled individual facial action units (FAUs). As of now, these FAUs are considered benchmarks for defining emotional facial states and, consequently, developing emotion recognition systems. However, numerous units in FACS impose computational complexity on facial emotion recognition, which can cause critical defects in operating information systems, especially when it comes to security and data leaks. Thus, there is an urgent need to develop a human face interpretation method for information security systems to identify changes in the emotional state by facial expressions online with little computational complexity. The presented work aims to develop a method of presenting emotional expressions of the human face through geometric facial features.

Hence, the following issues are to be addressed to achieve the aim of the work: 1. To investigate various geometric facial features that might be employed to identify emotional facial states. 2. To design a method of representing facial expressions of a human face using new geometrical features. 3. To conduct computational experiments with the proposed method and its analogs to categorize emotional facial states.

4. To validate the geometric representation of mimic expressions with a reference dataset.

2. Related work

Geometric facial features, facial features based on appearance, and a combination of these two approaches are commonly used to recognize emotional facial expressions. For instance, in study [ 9 ], researchers divided faces into a plurality of small grids and combined features from all these grids to identify facial expressions. However, a slight displacement of the face in space reduces recognition accuracy due to removing features from inappropriate places. As such, because of the linear relationship, the computational complexity (both time and memory) of facial recognition systems increases in proportion to the amount of AU. Hence, several approaches to AU analysis for facial expression have been reported in scientific studies [ 10 ] to address this issue. The number of AUs was defined within 8 to 250, yet the choice of this number was subjective. As a result, the standard number of AUs has not yet been determined [ 11 ]. The number of AUs and their locations depend on the purpose and requirements for recognizing emotions in every investigated case.

The use of AUs was analyzed in various forms, such as triangle, grid, rectangle, attention map, and so forth. Defining facial expressions based on triangles has been more successful due to less computational complexity than traditional marker placement. In study [ 12 ], researchers reported that facial expression recognition systems deteriorate by 5% each year between training and test images. In paper [ 13 ], the authors used triangular characteristics such as area and perimeter extracted from the eyes, mouth, and nose with 12 FAUs; a maximum facial recognition rate of 94% was achieved in the FG-NET aging database. In work [14], the authors developed an emotion recognition system based on a triangular approach with fuzzy rules to examine six primary emotions. A dozen FAUs were positioned on a human face to construct eight so-called formal triangles. As a result, the recognition accuracy of 87.67% was achieved. Another study [15] proposed to apply the angle and position of 52 FAUs as geometric characteristics for the facial expression recognition system. The Euclidean distance and angle between each pair of landmarks within the boundary frame were determined. These angle and distance values were subtracted from the corresponding values in the first video frame. The multi-class AdaBoost was employed with dynamic time distortion, and a support vector machine (SVM) was used on extended feature vectors. Researchers in [16] proposed to consolidate the accelerated-look model, spatial representation, and native binary pattern to strengthen recognition of mimic emotional expressions. This hybrid approach could achieve decent results in emotion recognition based on 68 different facial points.

Several approaches have been provided to implement systems for recognizing changes in an emotional state in real-time. As of now, the minimum number of AUs has still been used to determine facial expressions [17]. As an example, the Delaunay triangulation method [18] is used to associate sixty-eight FAUs to identify seven primary facial emotions defined by FACS. This method was employed to determine the spatial facial features and SVM as a classifier [ 10 ], obtaining a maximum classification level of 84%. In work [19], the authors reported applying traditional machine learning (ML) techniques (SVM, k-nearest neighbor, random forest etc.) to identify changes in four emotions (happiness, sadness, anger, and fear). The considered approaches reached the average maximum recognition accuracy of 97.47% using the method of random forest. In study [20], there was suggested a novel vectorized facial emotion recognition deep learning (DL) model based on seventy feature vectors to recognize three primary human emotions: anger, happiness, and neutral. This model could achieve an average accuracy of 94.33% on various reference datasets. Some recent studies have focused above all on spatial input data to extract various facial features using recurrent neural networks (RNNs) [21], deep convolutional neural networks (DCNNs), and multilevel ensemble DCNNs [22].

Overall, the analysis of related works revealed the most common challenges of the above approaches: poor quality of training datasets, low accuracy of classification of facial expressions, high computational complexity, and a significant amount of physical memory of the prepared models. Considering these factors, an urgent task appears to develop a new approach to describing the features of human facial expressions that will be computationally efficient and provide high recognition accuracy for real-time security systems.

The proposed method is based on the software-generated landmarks for straightforward geometric characteristics of the face. The landmarks serve as input data to identify five primary human emotional changes: anger, fear, joy, neutrality, and sadness. The method consists in forming seven geometric shapes based on the constructed landmarks, with the subsequent quantitative expression of these shapes. The quantitative expressions of the seven shapes are highlighted as the quantitative features for classifying facial expressions. Finally, the calculated quantitative traits are reflected in the corresponding expressions of emotional facial states using the method of hyperplane classification.

3. Methods and materials

The proposed method of the geometric interpretation of facial areas is designed to automatically reflect the facial expressions of human emotions in the form of quantitative characteristics of geometric shapes on the human face. It calculates the values of emotional manifestations based on landmarks that indicate specific markers in the face. 3.1.

The description of the proposed method

The scheme of the method is given in Fig. 1.

The input data of the method is image P of a group of people with highlighted regions containing human faces.

In Block 1, specific points-features of the human face are determined. For this purpose, we utilized an open-source library called MediaPipe Face Mesh [23]. This solution allows programming the face geometry through 468 3D orientations (landmarks). The example of face geometry constructed using MediaPipe Face Mesh is shown in Fig. 2.

In Fig. 2b), superimposed 486 landmarks Pu , u = 0, 467 , are illustrated by green mugs. In Block 2, quantitative characteristics of the face are calculated.

In step 2.1, feature vector X is formalized by geometric shapes  i7=1 , which ends lie at points Pu .

Input data:

An image of a group of people with highlighted regions containing faces, P

Block 1 – Determining of specific facial points:

Landmarks Pu , u = 1, 468 .

Block 2 – Obtaining quantitative characteristics of facial areas: 2.1 – Formalization of feature vector X of facial emotional states through geometric shapes  i7=1 based on points Pu , u = 1, 468 . 2.2 – Calculation of areas of geometric shapes  i7=1 .

Block 3 – Normalizing the vector:

X → X .

Output data:

Normalized feature vector X

In step 2.2, areas of geometric shapes  i7=1 are calculated. Table 1 contains the types of shapes for each face region, which were determined empirically. The distance of the segments that form shapes  i7=1 is calculated by Euclid’s distance formula. Below we provide a detailed description of shapes  i7=1 for each facial feature.

Face region “Mouth” is described by a triangle, the ends of which lie at points P17 , P37 and P267 . Quantitative characteristic 1 is the area of triangle P17P37P267 : 1 =

p1,0 ( p1,0 − P17 P37 )( p1,0 − P37 P267 )( p1,0 − P267 P17 ), where p1,0 = P17 P37 + P37 P267 + P267 P17 .

2

Face region “Corners of the lips” is described by the ratio of segments P1P61 and P61P291. Quantitative characteristic  2 is the relation:  2 = 3P1P61 .

P61P291 3 = P27P145  P33P133.

Face region “Eyes” is described by the ratio of segments P27P145 and P33P133 for a left eye. Quantitative characteristic  3 is the product:

Feature 1 ∈ {0 … 1}, 0 – closed, 1 – opened. 2 ∈ {0 … 1}, 0 – omitted, 1 – raised.

3 ∈ {0 … 1}, 0 – squinted (almost

closed), 1 – widely disclosed.

4 ∈ {0 … 1}, 0 – normal, 1 – brought together.

5 ∈ {0 … 1}, 0 – normal, 1 – raised. 6 ∈ {0 … 1}, 0 – normal, 1 – raised. 7 ∈ {0 … 1}, 0 – normal, 1 – raised.

Type of shape A triangle describing

the mouth.

A ratio of segments to the corners of the

lips.

A quadrilateral describing the left

eye.

A quadrilateral describes the nose.

A triangle that describes the upper part of the face to the

eyebrows.

A segment to the outer corner of the

eyebrows.

A segment to the inner corner of the eyebrows.

(1) (2) (3) between eyebrows. Quantitative characteristic  4 is the product: Quantitative characteristic  5 is the area of triangle P1P105P334 : , where  i represents the quantitative characteristic of the i-th feature, i = 1, 7 ,  i min stands for the minimum value of the quantitative characteristic of the i-th feature and defined empirically, imax denotes the maximum value of the quantitative characteristic of the i-th feature and determined empirically, xi represents the normalized values of the i-th feature, xi  X , xi 0;1 .

The output data of the proposed method is normalized feature vector X used to identify emotional facial states. Consequently, the method of the geometric interpretation of facial sites allows displaying a person’s face detected in an image in normalized feature vector X . 3.2.

Dataset

A reference dataset of Amsterdam Dynamic Facial Expression Set (ADFES) [24] received by the Amsterdam Interdisciplinary Center of Amsterdam University was utilized in this study to test and validate the proposed method. The ADFES dataset contains videos of humans’ emotional expressions collected from twenty-two models. The authors of this study formed a subset of the original ADFES dataset with five emotions: anger, fear, joy, neutral, and sadness. Each of the twenty-two models of ADFES depicts five different emotional states.

(a) (b) (4) (5) (6) (c) (g) Figure 3: The visual representation of the proposed geometric features: (a) – mouth, (b) – corners of the lips, (c) – eyes, (d) – nasal root, (e) – eyebrows, (f) – the outer corners of eyebrows, (g) – the inner corners of eyebrows 3.3.

Evaluation criteria and experiment setup

Let us consider the number of actual positive (P) and real negative (N) cases in the original dataset. After applying a classifier to the dataset, the targeted objects are categorized as true positive (TP), true negative (TN), false positive (FP), and false negative (FN) cases.

In this work, the proposed method was validated by several statistical indicators [25, 26] defined as

TP + TN Accuracy =

TP + TN + FP + FN Balanced accuracy = 12  TPP + TNN  , , (7) (8) Precision =

Recall = , .

(9) (10) (11) (12) 2 TP+ FP+ FN

The computational experiments were performed using Python v3.9 and the ML library called Scikitlearn. The hardware comprises an eight-core Ryzen 2700 and a single NVIDIA GeForce GTX1080 CPU with 8 GB video memory.

4. Results and discussion

This section of the work presents and discusses the results of experiments with the proposed method for identifying emotional facial states by mimic manifestations. The input data of the method serve 110 images of human faces of the ADFES data set, classified by five emotions. As a result of applying the method, the matrix of normalized values X = ( xij k ) was obtained, where i = 1,7 – facial features, j = 1,110 – objects of the training dataset, k = 1,5 – investigated emotions. Next, based on the method of hyperplane classification [23], the following linear classifier was constructed: d ( X) = 0.005565x1 + 0.002142x2 + 0.027011x3 + 0.004986x4 −

− 0.0047x5 − 0.01164x6 − 0.03891x7 + 0.028614.

Linear classifier (12) is used to classify emotional manifestations, and, consequently, identify emotional facial states by mimicking manifestations for information systems that meet security requirements.

The proposed method of the geometric interpretation of facial expressions was compared with other approaches to determine the feature changes in the emotional state, namely with the FACS encoding system [ 8 ] and with the traditional method of triangles [ 13 ]. The computational results measured by (7)-(11) are presented in Table 2.

It can be observed from Table 2 that the proposed method slightly surpasses the considered analogs in all statistical indicators, namely, accuracy – by 0.91%, precision – by 0.71%, recall – by 4.35%, balanced accuracy – by 2.17%, and 1 – in 2.75%. At the same time, the use of simple mathematical calculations within our method allowed a significant reduction of computational complexity (0.004 sec over 0.011 sec at the nearest competitor) against the analogs.

5. Conclusion

This paper proposes a method of the geometric interpretation of facial sites intended for automated reflection of mimic manifestations of human emotions like quantitative characteristics of geometric shapes on a human face. The proposed method is based on software-generated landmarks for a simple geometric characteristic of the face. The landmarks serve as input data for the method to recognize five primary changes in the emotional state: anger, fear, joy, neutral, and sadness.

The method consists of forming seven geometric shapes based on the predefined landmarks, with the subsequent quantitative expression of these shapes. The resulting data of the method is the quantitative expression of seven shapes in the form of quantitative features used herein for the classification of facial expressions. Our method has been validated with the method of hyperplane classification. The computational results show that our method demonstrates the competitive performance in classification accuracy and consequently confirms its effectiveness for identifying changes in the emotional state. In addition, the use of simple mathematical calculations in our method significantly reduces computational complexity against analogs.

Further research will be devoted to detecting human faces from low-resolution or long-distance video cameras and creating appropriate information systems for efficient security surveillance.

6. References

[14] T. Tuncer, S. Dogan, M. Abdar, M. Ehsan Basiri, and P. Pławiak, Face recognition with triangular fuzzy set-based local cross patterns in wavelet domain, Symmetry (Basel)., vol. 11, no. 6, p. 787, 2019, doi:10.3390/sym11060787. [15] D. Ghimire, J. Lee, Z.-N. Li, and S. Jeong, “Recognition of facial expressions based on salient geometric features and support vector machines,” Multimed. Tools Appl., vol. 76, no. 6, pp. 7921–7946, 2017, doi:10.1007/s11042-016-3428-9. [16] M. Iqtait, F. S. Mohamad, and M. Mamat, Feature extraction for face recognition via Active Shape Model (ASM) and Active Appearance Model (AAM), in IOP Conference Series: Materials Science and Engineering (IORA-ICOR-2017), 2018, vol. 332, p. 12032. doi:10.1088/1757-899x/332/1/012032. [17] M. Kunz, D. Meixner, and S. Lautenbacher, Facial muscle movements encoding pain—a systematic review, Pain, vol. 160, no. 3, pp. 535–549, 2019, doi:10.1097/j.pain.0000000000001424. [18] H. Valev, A. Gallucci, T. Leufkens, J. Westerink, and C. Sas, Applying Delaunay triangulation augmentation for deep learning facial expression generation and recognition, in Pattern Recognition. ICPR International Workshops and Challenges (ICPR-2021), 2021, vol. 12663, pp. 730–740. doi:10.1007/978-3-030-68796-0_53. [19] Murugappan M., Mutawa A., Facial geometric feature extraction based emotional expression classification using machine learning algorithms, PLoS One, vol. 16, no. 2, pp. 1–20, 2021, doi:10.1371/journal.pone.0247131. [20] X. Sun, S. Zheng, and H. Fu, ROI-attention vectorized CNN model for static facial expression recognition, IEEE Access, vol. 8, no. 1, pp. 7183–7194, 2020, doi:10.1109/ACCESS.2020.2964298. [21] C. Li, Z. Bao, L. Li, and Z. Zhao, Exploring temporal representations by leveraging attentionbased bidirectional LSTM-RNNs for multi-modal emotion recognition, Inf. Process. Manag., vol. 57, no. 3, p. 102185, 2020, doi:10.1016/j.ipm.2019.102185. [22] M. A. H. Akhand, S. Roy, N. Siddique, M. A. S. Kamal, and T. Shimamura, Facial emotion recognition using transfer learning in the deep CNN, Electronics, vol. 10, no. 9, p. 1036, 2021, doi:10.3390/electronics10091036. [23] C. Lugaresi et al., MediaPipe: A framework for building perception pipelines, arXiv:1906.08172, pp. 1–9, Jun. 2019, Accessed: Jan. 12, 2022. [Online]. Available: https://arxiv.org/abs/1906.08172v1. [24] J. van der Schalk, S. T. Hawk, A. H. Fischer, and B. Doosje, Moving faces, looking places: Validation of the Amsterdam dynamic facial expression set (ADFES), Emotion, vol. 11, no. 4, pp. 907–920, 2011, doi:10.1037/A0023853. [25] O. Pomorova, O. Savenko, S. Lysenko, A. Nicheporuk Metamorphic Viruses Detection Technique based on the the Modified Emulators, CEUR Workshop Proceedings 1614 (2016) 375-383. [26] G. Markowsky, O. Savenko, S. Lysenko, A. Nicheporuk The technique for metamorphic viruses' detection based on its obfuscation features, CEUR Workshop Proceedings 2104 (2018) 680–687.

References

[1] A. Trofimova , T. Wiley, M. Pagnucco , and M. Velonaki , Exploring human machine-mediated interaction for applications in social HRI , Proceedings of the 31st Australian Conference on Human-Computer-Interaction (OZCHI'-209) , 2019 , vol. 2019 , pp. 347 - 351 . doi: 10 .1145/3369457.3369492.

[2] I. G. Kryvonos , I. V Krak , O. V Barmak , and R. O. Bagriy , New tools of alternative communication for persons with verbal communication disorders , Cybern. Syst. Anal. , vol. 52 , no. 5 , pp. 665 - 673 , 2016 , doi:10.1007/s10559-016-9869-3.

[3] I. Krak , O. Barmak , and P. Radiuk , Information technology for early diagnosis of pneumonia on individual radiographs , in 3rd International Conference on Informatics & Data-Driven Medicine (IDDM-2020) , 2020 , vol. 2753 , pp. 11 - 21 . [Online]. Available: http://ceurws.org/Vol- 2753 /paper3.pdf

[4] A. Fnaiech , H. Sahli , M. Sayadi , and P. Gorce , Fear facial emotion recognition based on angular deviation , Electronics , vol. 10 , no. 3 , p. 358 , 2021 , doi:10.3390/electronics10030358.

[5] A. Saxena , A. Khanna , and D. Gupta , Emotion recognition and detection methods: A comprehensive survey , J. Artif. Intell. Syst. , vol. 2 , no. 1 , pp. 53 - 79 , 2020 , doi:10.33969/AIS. 2020 . 21005 .

[6] B. Martinez , M. F. Valstar , B. Jiang , and M. Pantic , Automatic analysis of facial actions: A survey , IEEE Trans. Affect. Comput. , vol. 10 , no. 3 , pp. 325 - 347 , 2019 , doi:10.1109/TAFFC. 2017 . 2731763 .

[7] P. Radiuk and H. Kutucu , Heuristic architecture search using network morphism for chest Xray classification , 1st International Workshop on Intelligent Information Technologies & Systems of Information Security (IntelITSIS-2020) , 2020 , vol. 2623 , pp. 107 - 121 . Accessed: May 09, 2021 . [Online]. Available: http://ceur-ws. org/ Vol- 2623 /paper11.pdf

[8] P. Ekman , W. V. Friesen , and J. C. Hager , The facial action coding system: The manual . Salt Lake City: UT Research Nexus eBook, 2002 . Accessed: Jan. 18 , 2022 . [Online]. Available: https://www.paulekman.com/product/facs-manual/

[9] S. P. Ramalingam and C. M. Paturu Venkata Subbu Sita Rama , Dimensionality reduced local directional number pattern for face recognition , J. Ambient Intell. Humaniz. Comput. , vol. 9 , no. 1 , pp. 95 - 103 , 2018 , doi:10.1007/s12652-016-0408-x.

[10] I. Dagher , E. Dahdah, and M. Al Shakik, “ Facial expression recognition using three-stage support vector machines , ” Vis. Comput. Ind. Biomed. Art , vol. 2 , no. 1 , p. 24 , 2019 , doi:10.1186/s42492-019-0034-5.

[11] E. Morales-Vargas , C. A. Reyes-García , and H. Peregrina-Barreto , On the use of Action Units and fuzzy explanatory models for facial expression recognition , PLoS One , vol. 14 , no. 10 , pp. 1 - 13 , 2019 , doi:10.1371/journal.pone. 0223563 .

[12] K. K. Kamarajugadda and T. R. Polipalli , Age-invariant face recognition using multiple descriptors along with modified dimensionality reduction approach , Multimed. Tools Appl. , vol. 78 , no. 19 , pp. 27639 - 27661 , 2019 , doi:10.1007/s11042-019-7741-y.

[13] A. Juhong and C. Pintavirooj , Face recognition based on facial landmark detection , 2017 10th Biomedical Engineering International Conference (BMEiCON-2017) , 2017 , vol. 10 , pp. 1 - 4 . doi: 10 .1109/BMEiCON. 2017 . 8229173 .