There are many approaches to determining the emotional states of a person in an image or video. Recognizing emotions on a human face is characterized by high complexity compared to voice analysis or measurement of other indicators. There are many factors on a person's face that describe one or another emotional state of a person. Convolutional Neural Networks (CNNs) are widely used for image recognition and classification, and are well suited to the task of classifying emotions on human faces. However, there is no universal approach that perfectly classifies all types of images. Therefore, it is necessary to conduct a comparative analysis of different CNN architectures to determine the best of them for the given task. This study compares AlexNet, LeNet, and MobileNet architectures for the classification of emotional states in human face images. A structure of a convolutional neural network was also developed, which showed better results in comparison with analogues. The results of the analysis will make it possible to draw conclusions about the choice of optimal CNN architectures and develop improved proprietary architectures to improve the quality of emotion classification. Analysis of existing architectures allows to develop a new one, with better results according to the criterion of accuracy.
Emotions are an integral part of human life, which allows to improve interaction between people and to understand the attitude of a specific person to a certain situation. Being able to recognize emotions such as happiness, sadness, anger, fear, surprise, and disgustmakes it possible to better understand a person or group of people. Nowadays, artificial intelligence plays a big role in people's lives, allowing to improve the quality of life in various areas. Therefore, the use of artificial intelligence, in particular convolutional neural networks (CNN), to solve the problems of assessing a person's emotional state based on the analysis of facial images is an urgent task. Convolutional neural networks are using to recognize graphic images, which have proven to be the best approach for image classification compared to other neural network approaches. There are a large number of architectures of convolutional neural networks, but they are characterized by high classification quality only for a specific type of images. Unfortunately, there is no universal approach to qualitative classification of all types of images. In addition, such approaches will avoid a subjective vision and assessment of a person's condition. In addition, determining a person's emotional state will allow to automatically identify fake states, which is relevant in connection with the significant development of technologies that allow generating fake information. An important aspect of the development of a qualitative classification system is the sample on the basis of which research is conducted.
We used the CK+48 dataset [
The analysis of architectures of convolutional neural networks made it possible to develop a new architecture for the specific task of recognizing human emotions, which showed better results according to the criterion of accuracy.
The object of research is graphic images of human faces in various emotional states The subject of research is the architecture of convolutional neural networks for solving classification problems.
The purpose of the work is to develop a new convolutional neural network architecture for recognizing human emotions in images based on the analysis of existing architectures.
Significant attention of scientists is paid to the approaches of classification and recognition of
human facial emotions Several popular datasets have been developed for these tasks, and one of
them is CK+48 [
The article [
in [
A generalized analysis of approaches to recognition of human emotions and classification
using neural networks is given in the articles [
In the article [
In the article [
The analysis of approaches to the application of elements of artificial intelligence and computer vision systems in the tasks of processing graphic images is considered in articles [1214]. In particular, the analysis of approaches that allow to speed up image processing processes was carried out and new architectures of convolutional neural networks were proposed for the classification of specific types of images.
In the work, [15] authors consider an approach to searching for similar images.
To conduct the research, the following tasks should be implemented: - Analyze existing approaches to image classification; - Conduct research on the classification of architectures of convolutional neural networks based on the CK+48 dataset; - Develop a new architecture for the task of image emotion recognition - Analyze the results of the obtained studies.
There are several datasets that are used to conduct research on the tasks of classifying and
identifying human emotions. CK+48, which includes up to 1000 images divided into 7 classes:
(angry, happy, fear, neutral, disgust, sad, surprise) [
This stage includes both the use of existing architectures and those developed in this work.
The generalized emotion classification algorithm consists of two main components: dataset formation and direct neural network training. The quality and time of training will depend on the quality of the dataset distribution and the correct selection of the convolutional neural network architecture. 6. Analysis of architectures of convolutional neural networks
Convolutional neural networks are actively used for the classification of various types of images. The use of images as input data without the need for additional conversion into other formats is the main advantage of their use. CNN consist of the following elements: - Convolutional layers are one of the main elements of the neural network, which performs the operation of convolution of the input image with a mask. As a result, smaller images are created.
- Pooling layer – allows ones to reduce the spatial size of the collapsed function. This step also makes it possible to highlight features in the image more precisely.
- Fully Connected Layers – this layer is used at the final stage of neural network operation, which allows the model to interact with all functions of the input data.
The number of filters is a crucial parameter when working with a neural network as it directly impacts the network's quality, specifically the allocation of parameters. However, including too many parameters can lead to overtraining and worsen the results.
The size of the filters affects what patterns or features in the images are detected by the network. Larger filters are able to detect broader patterns, while smaller filters are able to detect finer details.
Strides - A larger stride will result in a smaller spatial dimension in the next layer. Activation function - the following activation functions are the most used - ReLU, tanh, sigmoid.
The depth of the network is the number of layers in it. Greater depth allows the model to learn more complex features, but may lead to overtraining.
This convolutional neural network architecture was one of the first and proved to be effective for classifying small images. A generalized view of the convolutional neural network architecture is shown in Figure 2.
This architecture is actively used for recognizing road signs or classifying human faces.
The process of training and operation of neural networks is characterized by the need for significant computing resources and memory.
The MobileNets architecture is designed specifically to work on mobile devices with low latency.
This architecture is designed to work with real-time systems. In many cases, this architecture shows better results compared to VGG16 and others.
A graphical representation of the blocks of the proposed convolutional network structure is shown in Figure 3. This architecture consists of convolutional layers, max-pooling layers, and Dense layers.
Increasing the number of convolutional and pooling layers increases the training time, however, allows for higher research accuracy, which is a higher priority task.
This architecture was selected based on an experimental approach.
Experiments were conducted on the same sample using different architectures with the same ratio of the training sample to the test sample.
AlexNet The results of using the alexnet architecture are shown in Figure 4
This graph shows the indicators of ROC curves for 7 classes. Taking into account the given data, it can be concluded that the classification accuracy is high.
Developed architecture The matrix of classification results is shown in Figure 5.
Layer parameters for the developed architecture for images from the CK+48 dataset: Layer (type) Output Shape Param # ================================================================= conv2d_73 (Conv2D) (None, 48, 48, 16) 448 _________________________________________________________________ max_pooling2d_73 (MaxPooling (None, 24, 24, 16) 0 _________________________________________________________________ conv2d_74 (Conv2D) (None, 24, 24, 16) 2320 _________________________________________________________________ activation_37 (Activation) (None, 24, 24, 16) 0 _________________________________________________________________ max_pooling2d_74 (MaxPooling (None, 12, 12, 16) 0 _________________________________________________________________ conv2d_75 (Conv2D) (None, 12, 12, 32) 12832 _________________________________________________________________ activation_38 (Activation) (None, 12, 12, 32) 0 _________________________________________________________________ max_pooling2d_75 (MaxPooling (None, 6, 6, 32) 0 _________________________________________________________________ conv2d_76 (Conv2D) (None, 4, 4, 64) 18496 _________________________________________________________________ max_pooling2d_76 (MaxPooling (None, 2, 2, 64) 0 _________________________________________________________________ conv2d_77 (Conv2D) (None, 2, 2, 64) 102464 _________________________________________________________________ activation_39 (Activation) (None, 2, 2, 64) 0 _________________________________________________________________ max_pooling2d_77 (MaxPooling (None, 1, 1, 64) 0 _________________________________________________________________ flatten_19 (Flatten) (None, 64) 0 _________________________________________________________________ dense_37 (Dense) (None, 128) 8320 _________________________________________________________________ dropout_19 (Dropout) (None, 128) 0 _________________________________________________________________ dense_38 (Dense) (None, 7) 903 ================================================================= Total params: 145,783 Trainable params: 145,783 Non-trainable params: 0
This image shows training and validation metrics over training epochs for a given model. It can be observed that both losses start relatively high but gradually decrease as the training progresses. The validation loss closely follows the training loss, indicating that the model is not overfitting significantly.
The ROC curve for the developed CNN architecture is shown in Figure 6.
Analyzing the above results, it can be concluded that the developed network showed better results compared to ALexNet and LeNet
The classification results using 4 neural network architectures are shown in Figure 7.
e p y T
AlexNet 0 10 20 30 40 50 60 70
80
Accuracy
As the number of classes increases, the complexity of developing a universal convolutional neural network architecture increases. Therefore, the process of designing a neural network architecture that would show the best results is an important and difficult task. The results of the comparison of the developed architecture with well-known ones show better results, which allows to define the developed architecture as possible to be used for the classification of emotions on the human face.
In this work, existing architectures of convolutional neural networks were analyzed and our own was proposed, which showed better results.
As a result of the research conducted on the basis of the CK+48 dataset, it can be concluded that convolutional network architectures show approximately similar results, however, the LeNet architecture showed better results according to the accuracy criterion - 72%, unlike ALexNet or MobileNet - 64 and 60%, respectively. Instead, the developed architecture showed even better results, namely - 75%. The number of training parameters is 145,783.
Further research needs to broaden the network of architectures for research and not to be limited only to the best known.
[13] Berezsky, O., Liashchynskyi, P., Pitsun, O., Liashchynskyi, P., Berezkyy, M. Comparison of Deep Neural Network Learning Algorithms for Biomedical Image Processing. CEUR Workshop Proceeding, 2022, 3302, pp. 135–145. https://ceur-ws.org/Vol-3302/paper7.pdf [14] Pitsun, O. MLOps Approach for Automatic Segmentation of Biomedical Images / Oleh Berezsky , Oleh Pitsun , Grygoriy Melnyk , Yuriy Batko , Petro Liashchynskyi , Mykola Berezkyi // CEUR Workshop Proceeding, 2023, pp. 241–248. https://ceur-ws.org/Vol-3302/paper7.pdf [15] O. Veres, B. Rusyn, A. Sachenko and I. Rishnyak, "Choosing the method of finding similar images in the reverse search system", CEUR Workshop Proceedings, vol. 2136, pp. 99-107, 2018.