Approach to Recognizing of Visualized Human Emotions for Marketing Decision Making Systems Iryna Spivaka, Svitlana Krepycha, Oleksandr Fedorova and Serhii Spivakb a West Ukrainian National University, Lvivska str. 11, Ternopil, 46009, Ukraine b Ternopil Ivan Puluj National Technical University, Ruska str. 56, Ternopil, 46001, Ukraine Abstract The article proposes an approach to the recognition of visualized human emotions for marketing decision-making systems. The analysis of previous studies has shown the relevance and expediency of the proposed approach, as it will reduce the use of computing resources to implement the recognition process, and at the same time, increase the speed of obtaining the result. The article presents an algorithm for step-by-step identification of visualized human emotion based on the comparison of changes in the positions of key points of the selected element in accordance with changes in the characteristics of this element. Keywords 1 Recognition, Visualized Human Emotions, Pixel, Color Model, Marketing Decision. 1. Introduction Trends in any commercial activity show that making marketing decisions that will provide the greatest impact on the consumer in making decisions to maximize profits are relevant today. In recent years, in the area of marketing, the subject of intensive research has become nonverbal information, namely the study of facial expressions. It is known from psychology that all human emotions can be classified into six basic emotions, which are most used to obtain nonverbal information. The ability to automatically recognize this kind of information will simplify the interpretation of emotions on a person's face while watching advertising, product testing or using the service. The proposed approach will help to understand whether the consumer really liked the product, what color, size or smell he prefers, etc., as the survey can often get inaccurate information. The proposed approach will allow you to see the informal reaction of users, which will help to understand what necessary to focus on and what to improve. 2. Related works The development of emotion recognition in the vast majority of methods occurs in three steps [1]. In the first step, functions are defined from fixed images, and in the second step, emotions are detected with the help of already developed classifiers and in the third step is face recognition itself. The most common are Local Binary Patters (LBP) [2] – is a description of the pixels around the central pixel in binary form. The COLINS-2021: 5th International Conference on Computational Linguistics and Intelligent Systems, April 22–23, 2021, Kharkiv, Ukraine EMAIL: spivak.iruna@gmail.com (I. Spivak); msya220189@gmail.com (S. Krepych); fedorov.oleks@gmail.com (O. Fedorov); spivaksm@ukr.net (S. Spivak) ORCID: 0000-0003-4831-0780 (I. Spivak); 0000-0001-7700-8367 (S. Krepych); 0000-0002-8080-9306 (O. Fedorov); 0000-0002-7160-2151 (S. Spivak) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) LBP operator is applied to the central pixel of the image, it uses 8 pixels that are around it taking the central pixel as the main. The main disadvantage of this method is that the image needs high-quality preprocessing due to high sensitivity to noises. Figure 1: Example of an applied LBP pattern to the image All existing face recognition methods have their pros and cons, let’s have a look at the main disadvantages of each of the methods. Geometric approach to face recognition – is one of the first developed methods it consists of choosing the key points of the face such as lips, center of the eye, etc. This method does not require an expensive equipment but it affects the low reliability of the method. Disadvantages:  Low reliability;  Hight lighting requirements;  Mandatory frontal image of the person;  Does not take into account the possibility of changing facial expressions. Figure 2: Face distribution using the geometrical method Viola-Jones method and Haar features [3, 4, 13, 15] - is the most popular method of finding the facial area in images because of its relatively high speed and efficiency. Face recognition in this method is based on three basic principles:  Integral representation of the image on the basis of Haar, which allows you to calculate the necessary features;  Classifier construction method based on adaptive boosting algorithm (AdaBoost);  Method of combining classifiers into a cascade structure. Disadvantages:  At an angle of 30 * or more the probability of recognition drops rapidly;  Makes it impossible to detect a person at an arbitrary angle;  Takes a lot of training time;  Sensitive to lighting. Figure 3: Haar feature used for Viola Jones face recognition method Active appearance model (AAM) - are statistical models of images that can be adjusted to the real image by various deformations. Fitting the model to a specific image of the face is performed in the process of solving the optimization problem, the essence of which is to minimize the functionality. Figure 4: The process of realization of active appearance model However, these approaches require a lot of time and resources for training, which limits their use in a large sample of input data [7-9, 11, 12]. The article proposes an approach to recognizing the emotions of the human face by definition and changing the positions of key points of the eyes, mouth and eyebrows, which does not require large computational resources for its implementation. 3. Overview of the Research Emotion is one of the basic elements of the human psyche. A human's emotions are distinguished depending on the state of satisfaction of his needs. They can be both positive and negative, as well as neutral, when a person doesn't react in any way and remains in its original state. For example, looking the advertising - some people feel anger or disgust, while others - pleasure and interest. A person's behavior also changes depending on what emotions the person is experiencing. When making marketing decisions, the automated system must be able to identify and recognize one of the six basic human emotions:  Surprise - a short-term feeling and state of a person that occurs during a sudden and unexpected situation.  Fear is a state of anxiety and restlessness caused by the expectation of something undesirable or unpleasant.  Disgust - a feeling of disapproval towards someone or something.  Angry is a strong feeling of dissatisfaction that arises when a person's needs or expectations have not satisfied.  Happy is a feeling of satisfaction that arises when a person's needs or expectations have been met.  Sad is a feeling opposite to happy, which arises in case of loss and helplessness of a person. Figure 5: Detection of emotions on a human's face Figure 5 on the left shows the face of a man in his calm (neutral) state. To detect emotions, it is first necessary to determine the area of a person's eyes and mouth in its neutral state. The neutral state is characterized by quantitative indicators that do not belong to either positive or negative emotions. This is better seen in the example of the eyes on Figure 1. In a neutral state, a person's eyes are not as open or squinted as in other emotions. The situation is similar with other key points of the face. In the future, depending on the emotion reflected on the face, this area will increase or decrease. For example, let's take two emotions: disgust and happy, which are very similar when identifying key points. In both, the oral area increased and the eye area decreased. Once the program determines this, the search for the next key points will be narrowed only to these two emotions, and will not pass all six, which will save time to identify the required final emotion, in contrast to existing methods and approaches [5, 6, 14]. 4. Proposed approach Our performed analysis of this issue showed that to determine a human's emotions it is enough to choose the key elements on the face, namely the eyes, eyebrows, nose and mouth, and not to identify the whole face. The algorithm of proposed approach includes the following steps: Step 1. Determining a face image from a photo or video and convert it into black and white using the capabilities of the CSS-filter function, which can be implemented on any PC hardware, as it doesn't require large resources. Step 2. Selection of key elements of the face and their processing in the HSL (Hue Saturation Luminance) color model (Fig. 6), where Hue - means both color and hue; Saturation - indicates the amount of gray color; Luminance is the intensity of light projected on a given area and direction. Figure 6: The selected pixel area Step 3. Each pixel of the photo is replaced by its numerical value (the darker the hue of black, the smaller the number and vice versa) (Fig. 7). These numerical values will be used to search for key points of the selected facial elements based on the algorithm for finding the nearest neighbor, namely the hue of black. When the hue of darkness decreases by more than 15% (perhaps more, its need to check by the software) - this means that the next pixel (depending on its direction) doesn't need to be estimated. This will give us clear contour of the element we are estimating. Figure 7: Numerical matrix of pixel values of the selected area Step 4. Based on the analysis of changes in the positions of key points performing to identify human emotions. Figure 8 shows a diagram of the implementation of the proposed method, which includes three main modules: image determination, conversion and identification. Module Module "Image Conversion" "Identification" Photo Video Convert to black and Identification of key white format points Face Processing in the color Calculation and determination model classification Construction of a numerical matrix of Identification of emotion pixels Figure 8: The scheme of implementation of the proposed approach 5. Results & Discussion Based on the emotion of fear, we will show how a human's face will change, and how software will be able to detect it. For example, from the fixed image of the face the element "eye" for research is determinate. a) b) Figure 9: Images of eye expansion in the color model: a) in a neutral human state; b) in a state of fear Figure 9a) shows the image of the human eye in a neutral state, while Figure 9b) - the expansion of the eye during the action of the emotion of fear. It is known that when a person is afraid, his eyes expand and their area increases accordingly. Counting the number of pixels from the extreme left point (1) to the extreme right (2) we get 31 pixels in the neutral state and 32 pixels in the fear state. The difference of 1 pixel is not significant, because during fear the eyes cannot increase in width, only in height. Counting the number of pixels from the uppermost point (3) to the lowermost point (4) we get 17 pixels in the neutral state and 23 pixels in the fear state. Now that we have this data, we can calculate the area of the eyes, which will show us its increase during the action of the emotion of fear. The next key point is the eyebrows. a) b) Figure 10: Images of eyebrow in the color model: a) in the neutral human state; b) in a state of fear Figure 10a) shows the eyebrows and eyes of a human in a neutral state, and Figure 10b) - under the influence of fear. As we can see in a neutral emotional state, the distance between the eyebrows (from point 2 to point 1a) is 37 pixels, the distance between the eyebrows in a state of fear - 32 pixels. You should also pay attention to the distance from the eyes to the eyebrows. In the neutral state, the distance for the left eyebrow is 13 pixels and for the right 8. In a frightened person, this distance is 4 pixels for the right and left eyebrows. The last key point in recognizing the emotion of fear is the corners of the lips. a) b) Figure 11: Image of lips in the color model: a) in the neutral human state; b) in a state of fear Figure 11a) shows the area of the human mouth in a neutral state and it is 21 pixels. In the state of fear person opens his mouth and, accordingly, changes its size. In our case it is 28 pixels (Fig.11b). According to the study, we can generate table 1, which contains data on the change of distance in pixels between the extreme points of the selected elements of the face to facilitate the identification of the image of emotion. Table 1 Change of key points positions according to changes of the characteristics of the selected element Emotion Characteristic Change key points of elements Neutral 1. when a person does not 1.1. Position of a face key elements are in their usual react in any way and remains position in its original state Surprise 1. dilated eyes 1.1. The distance in pixels increases from the 2. raised eyebrows extreme points of the eye. 3. open mouth (extended) 2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye increases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 3.2. The distance from the extreme left and right points of the mouth decreases. Fear 1. dilated eyes 1.1 The distance in pixels increases from the 2. raised eyebrows extreme points of the eye. 3. wide open mouth 2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye increases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 3.2. The distance from the extreme left and right points of the mouth does not change. Disgust 1. dilated eyes 1.1. The distance in pixels increases from the 2. raised eyebrows extreme points of the eye. 3. clenched mouth 2.1. The distance in pixels from the lower extreme point of the eyebrow to the upper extreme point of the eye increases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip decreases. Angry 1. squinted eyes 1.1. The distance in pixels decreases from the 2. lowered and shifted extreme points of the eye. eyebrows 2.1. The distance in pixels from the extreme lower 3. open mouth point of the eyebrow to the extreme upper point of the eye decreases. 2.2. The distance from the extreme left point of one eyebrow to the extreme right point of the second eyebrow decreases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. Happy 1. eyes are neutral 1.1. The distance in pixels from the extreme points 2. eyebrows are neutral of the eye does not change significantly 3. slightly open mouth 2.1. The distance in pixels from the eyebrow to the eye does not change significantly 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. Sad 1. squinted eyes 1.1. The distance in pixels decreases from the 2. lowered and shifted extreme points of the eye. eyebrows 2.1. The distance in pixels from the extreme lower 3. clenched mouth point of the eyebrow to the extreme upper point of the eye decreases. 2.2. The distance from the extreme left point of one eyebrow to the extreme right point of the second eyebrow decreases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip decreases. According to the data in Table 1, the programing search for the emotion fixed in the image will be as follows. First, for example, choose one element "eyes" and check it for conformity with the characteristics of emotions when they are detected (Fig.12). Figure 12: Identification of emotions according to the characteristics of the eyes The figure shows that the further search will continue in one of three directions if eyes are dilated (Fear, Surprise or Disgust) or (Angry or Sad) if they are squinted, and for the last if eyes are unchanged (Happy or Neutral Emotion) can be applied. The next steps are to cut off unnecessary emotions by checking other characteristics. To determine the change in the position of key points of the eyes, eyebrows and mouth, the ranges of these changes should be set in the interval forms [10], which will take into account the physiological characteristics of human faces. 6. Conclusions Research in the field of psychology has shown that the emotional state of all people has common external features. This made it possible to develop a universal classifier of emotions, which can be used to determine person’s state. The article proposes an approach to the recognition of visualized human emotions using a pixel color model, which has the ability to adapt to changes in input data less time consuming compared to other existing methods, high speed and low resource usage. The proposed approach has practical value in marketing decision-making systems based on the analysis of a human's emotional state while viewing or testing a particular product or service. The article presents an algorithm for step-by-step identification of visualized human emotion based on the comparison of changes in the positions of key points of the selected element in accordance with changes in the characteristics of this element. Further research will focus on the development of automated methods and algorithms for recognizing human emotions, taking into account the physiological characteristics of the human face, gender, age and more. This consideration is necessary and appropriate, because the physiological features of the facial structure of men and women are different: the location of the eyebrows, their width, the shape of the nose, the shape of the lips and their thickness. 7. References [1] S. Minaee, M. Minaei, A. Abdolrashidi, Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network, Sensors, 2021, 21, 3046. doi:10.3390/s21093046. [2] S. Caifeng, S. Gong, P. McOwan, Facial expression recognition based on local binary patterns: A comprehensive study, Image and vision Computing 27.6 (2009): 803-816. [3] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, J. Movellan, Recognizing facial expression: machine learning and application to spontaneous behavior, in: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR ’05, San Diego, CA, USA, 2005, pp. 568-573. doi: 10.1109/CVPR.2005.297. [4] I. Spivak, S. Krepych, V. Faifura, S. Spivak, Methods and tools of face recognition for the marketing decision making, in: Proceedings of IEEE International Scientific-Practical Conference: Problems of Infocommunications Science and Technology, PICS&T ‘19, Kyiv, Ukraine, 2019, pp. 212–216. doi:10.1109/PICST47496.2019.9061229. [5] A. Kovalenko, The system of recognition of facial expressions of human emotions using a multilayer perceptron, Visn. Nat. Lviv Polytechnic University (2011) 76-81. [6] G. Yefimov, Modeling and recognition of facial expressions of emotions on a person's face, Artificial Intelligence, vol. 3 (2009) 532-542. [7] T. Baltrusaitis, P. Robinson, L. Morency, OpenFace: An open source facial behavior analysis toolkit, in: Proceedings of IEEE Winter Conference on Applications of Computer Vision, WACV ’16, Lake Placid, NY, USA, 2016, pp. 167-171. doi:10.1109/WACV.2016.7477553. [8] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR ’16, Las Vegas, NV, USA, 2016, pp. 2818–2826. doi: 10.1109/CVPR.2016.308. [9] K. He, X. Zhang, Sh. Ren, J. Sun, Deep residual learning for image recognition in: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR ’16, Las Vegas, NV, USA, 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90. [10] I. Spivak, S. Krepych, S. Budenchuk, Methods and means of expert evaluation of software systems on the basis of interval data analysis, in: Proceedings of 14th International Conference on Advanced Trends in Radioelectronics, Telecommunications and Computer Engineering, TCSET ’18, Lviv- Slavske, Ukraine, 2018, pp. 101-127. doi: 10.1109/TCSET.2018.8336178. [11] N. Mehendale, Facial emotion recognition using convolutional neural networks (FERC), SN Applied Sciences (2020). doi: 10.1007/s42452-020-2234-1. [12] V. Chirra, S. Uyyala, V. Kolli, Facial Emotion Recognition Using NLPCA and SVM, Traitement du Signal (2019) 13-22. doi: 10.18280/ts.360102. [13] Y. Kuldeep, S. Joyeeta, Facial expression recognition using modified Viola-John’s algorithm and KNN classifier, Multimedia Tools and Applications (2020). doi: 10.1007/s11042-019-08443-x. [14] M. Uddin, M. Hassan, A. Almogren, A. Alamri, M. Alrubaian, G. Fortino, Facial expression recognition utilizing local direction-based robust features and deep belief network, IEEE Access, 2017, pp. 4525-4536. doi: 10.1109/ACCESS.2017.2676238. [15] S. Adeshina, H. Ibrahim, S. Teoh, S. Hoo, Custom Face Classification Model for Classroom Using Haar-Like and LBP Features with Their Performance Comparison, Electronics 2021. doi: 10.3390/electronics10020102.