The article proposes an approach to the recognition of visualized human emotions for marketing decision-making systems. The analysis of previous studies has shown the relevance and expediency of the proposed approach, as it will reduce the use of computing resources to implement the recognition process, and at the same time, increase the speed of obtaining the result. The article presents an algorithm for step-by-step identification of visualized human emotion based on the comparison of changes in the positions of key points of the selected element in accordance with changes in the characteristics of this element.
Trends in any commercial activity show that making marketing decisions that will provide the greatest impact on the consumer in making decisions to maximize profits are relevant today. In recent years, in the area of marketing, the subject of intensive research has become nonverbal information, namely the study of facial expressions. It is known from psychology that all human emotions can be classified into six basic emotions, which are most used to obtain nonverbal information. The ability to automatically recognize this kind of information will simplify the interpretation of emotions on a person's face while watching advertising, product testing or using the service.
The proposed approach will help to understand whether the consumer really liked the product, what color, size or smell he prefers, etc., as the survey can often get inaccurate information. The proposed approach will allow you to see the informal reaction of users, which will help to understand what necessary to focus on and what to improve.
The development of emotion recognition in the vast majority of methods occurs in three steps [
All existing face recognition methods have their pros and cons, let’s have a look at the main disadvantages of each of the methods.
Geometric approach to face recognition – is one of the first developed methods it consists of choosing the key points of the face such as lips, center of the eye, etc. This method does not require an expensive equipment but it affects the low reliability of the method.
Disadvantages: Low reliability; Hight lighting requirements; Mandatory frontal image of the person; Does not take into account the possibility of changing facial expressions.
Viola-Jones method and Haar features [
Disadvantages: At an angle of 30 * or more the probability of recognition drops rapidly; Makes it impossible to detect a person at an arbitrary angle; Takes a lot of training time; Sensitive to lighting.
Active appearance model (AAM) - are statistical models of images that can be adjusted to the real image by various deformations. Fitting the model to a specific image of the face is performed in the process of solving the optimization problem, the essence of which is to minimize the functionality.
However, these approaches require a lot of time and resources for training, which limits their use in a
large sample of input data [
Emotion is one of the basic elements of the human psyche. A human's emotions are distinguished depending on the state of satisfaction of his needs. They can be both positive and negative, as well as neutral, when a person doesn't react in any way and remains in its original state. For example, looking the advertising - some people feel anger or disgust, while others - pleasure and interest. A person's behavior also changes depending on what emotions the person is experiencing.
When making marketing decisions, the automated system must be able to identify and recognize one of the six basic human emotions: Surprise - a short-term feeling and state of a person that occurs during a sudden and unexpected situation. Fear is a state of anxiety and restlessness caused by the expectation of something undesirable or unpleasant. Disgust - a feeling of disapproval towards someone or something. Angry is a strong feeling of dissatisfaction that arises when a person's needs or expectations have not satisfied. Happy is a feeling of satisfaction that arises when a person's needs or expectations have been met. Sad is a feeling opposite to happy, which arises in case of loss and helplessness of a person.
Figure 5 on the left shows the face of a man in his calm (neutral) state. To detect emotions, it is first necessary to determine the area of a person's eyes and mouth in its neutral state. The neutral state is characterized by quantitative indicators that do not belong to either positive or negative emotions. This is better seen in the example of the eyes on Figure 1. In a neutral state, a person's eyes are not as open or squinted as in other emotions. The situation is similar with other key points of the face.
In the future, depending on the emotion reflected on the face, this area will increase or decrease. For
example, let's take two emotions: disgust and happy, which are very similar when identifying key points.
In both, the oral area increased and the eye area decreased. Once the program determines this, the search
for the next key points will be narrowed only to these two emotions, and will not pass all six, which will
save time to identify the required final emotion, in contrast to existing methods and approaches [
Our performed analysis of this issue showed that to determine a human's emotions it is enough to choose the key elements on the face, namely the eyes, eyebrows, nose and mouth, and not to identify the whole face. The algorithm of proposed approach includes the following steps:
Step 1. Determining a face image from a photo or video and convert it into black and white using the capabilities of the CSS-filter function, which can be implemented on any PC hardware, as it doesn't require large resources.
Step 2. Selection of key elements of the face and their processing in the HSL (Hue Saturation Luminance) color model (Fig. 6), where Hue - means both color and hue; Saturation - indicates the amount of gray color; Luminance is the intensity of light projected on a given area and direction.
Step 3. Each pixel of the photo is replaced by its numerical value (the darker the hue of black, the smaller the number and vice versa) (Fig. 7). These numerical values will be used to search for key points of the selected facial elements based on the algorithm for finding the nearest neighbor, namely the hue of black. When the hue of darkness decreases by more than 15% (perhaps more, its need to check by the software) this means that the next pixel (depending on its direction) doesn't need to be estimated. This will give us clear contour of the element we are estimating.
Step 4. Based on the analysis of changes in the positions of key points performing to identify human emotions.
Based on the emotion of fear, we will show how a human's face will change, and how software will be able to detect it. For example, from the fixed image of the face the element "eye" for research is determinate.
Figure 9a) shows the image of the human eye in a neutral state, while Figure 9b) - the expansion of the eye during the action of the emotion of fear. It is known that when a person is afraid, his eyes expand and their area increases accordingly. Counting the number of pixels from the extreme left point (1) to the extreme right (2) we get 31 pixels in the neutral state and 32 pixels in the fear state. The difference of 1 pixel is not significant, because during fear the eyes cannot increase in width, only in height. Counting the number of pixels from the uppermost point (3) to the lowermost point (4) we get 17 pixels in the neutral state and 23 pixels in the fear state. Now that we have this data, we can calculate the area of the eyes, which will show us its increase during the action of the emotion of fear.
The next key point is the eyebrows.
Figure 11a) shows the area of the human mouth in a neutral state and it is 21 pixels. In the state of fear person opens his mouth and, accordingly, changes its size. In our case it is 28 pixels (Fig.11b).
According to the study, we can generate table 1, which contains data on the change of distance in pixels between the extreme points of the selected elements of the face to facilitate the identification of the image of emotion.
Characteristic Change key points of elements 1. when a person does not 1.1. Position of a face key elements are in their usual react in any way and remains position in its original state 1. dilated eyes 2. raised eyebrows 3. open mouth (extended) 1.1. The distance in pixels increases from the extreme points of the eye. 2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye increases. 1. dilated eyes 2. raised eyebrows 3. wide open mouth 1. dilated eyes 2. raised eyebrows 3. clenched mouth 1. squinted eyes 2. lowered and eyebrows 3. open mouth 1. eyes are neutral 2. eyebrows are neutral 3. slightly open mouth
Fear 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 3.2. The distance from the extreme left and right points of the mouth decreases. 1.1 The distance in pixels increases from the extreme points of the eye. 2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye increases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 3.2. The distance from the extreme left and right points of the mouth does not change. 1.1. The distance in pixels increases from the extreme points of the eye. 2.1. The distance in pixels from the lower extreme point of the eyebrow to the upper extreme point of the eye increases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip decreases.
1.1. The distance in pixels decreases from the shifted extreme points of the eye.
2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye decreases. 2.2. The distance from the extreme left point of one eyebrow to the extreme right point of the second eyebrow decreases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 1.1. The distance in pixels from the extreme points of the eye does not change significantly 2.1. The distance in pixels from the eyebrow to the eye does not change significantly 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip increases. 1.1. The distance in pixels decreases from the extreme points of the eye. 2.1. The distance in pixels from the extreme lower point of the eyebrow to the extreme upper point of the eye decreases.
eyebrow to the extreme right point of the second eyebrow decreases. 3.1. The distance from the extreme point of the lower lip to the middle point of the upper lip decreases.
According to the data in Table 1, the programing search for the emotion fixed in the image will be as follows. First, for example, choose one element "eyes" and check it for conformity with the characteristics of emotions when they are detected (Fig.12).
The figure shows that the further search will continue in one of three directions if eyes are dilated (Fear,
Surprise or Disgust) or (Angry or Sad) if they are squinted, and for the last if eyes are unchanged (Happy
or Neutral Emotion) can be applied. The next steps are to cut off unnecessary emotions by checking other
characteristics. To determine the change in the position of key points of the eyes, eyebrows and mouth, the
ranges of these changes should be set in the interval forms [
Research in the field of psychology has shown that the emotional state of all people has common external features. This made it possible to develop a universal classifier of emotions, which can be used to determine person’s state. The article proposes an approach to the recognition of visualized human emotions using a pixel color model, which has the ability to adapt to changes in input data less time consuming compared to other existing methods, high speed and low resource usage.
The proposed approach has practical value in marketing decision-making systems based on the analysis of a human's emotional state while viewing or testing a particular product or service. The article presents an algorithm for step-by-step identification of visualized human emotion based on the comparison of changes in the positions of key points of the selected element in accordance with changes in the characteristics of this element.
Further research will focus on the development of automated methods and algorithms for recognizing human emotions, taking into account the physiological characteristics of the human face, gender, age and more. This consideration is necessary and appropriate, because the physiological features of the facial structure of men and women are different: the location of the eyebrows, their width, the shape of the nose, the shape of the lips and their thickness.